# Verifiable evaluations of machine learning models using zkSNARKS

Tobin South  Alexander Camuto  Shrey Jain  Shayla Nguyen  Robert Mahari
Christian Paquin  Jason Morton  Alex ‘Sandy’ Pentland

###### Abstract

In a world of increasing closed-source commercial machine learning models,
model evaluations from developers must be taken at face value. These benchmark
results—whether over task accuracy, bias evaluations, or safety checks—are
traditionally impossible to verify by a model end-user without the costly or
impossible process of re-performing the benchmark on black-box model outputs.
This work presents a method of verifiable model evaluation using model
inference through zkSNARKs. The resulting zero-knowledge computational proofs
of model outputs over datasets can be packaged into verifiable evaluation
attestations showing that models with fixed private weights achieve stated
performance or fairness metrics over public inputs. These verifiable
attestations can be performed on any standard neural network model with
varying compute requirements. For the first time, we demonstrate this across a
sample of real-world models and highlight key challenges and design solutions.
This presents a new transparency paradigm in the verifiable evaluation of
private models.

zkSNARKs, secure ML, evals, benchmarking, model evaluations, verifiable
attestations, zkml

##  1 Introduction

Model transparency, bias checking, and result reproducibility are at odds with
the creation and use of closed-sourced machine learning (ML) models. It is
common practice for researchers to release model weights and architectures to
provide experimental reproducibility, foster innovation, and iteration, and
facilitate model auditing of biases. However, the drive towards
commercialization of models by industry (and, in the case of extremely large
language models, the concern over the safety of open-source models) has led to
the increasing practice of keeping model weights private (Karpur et al., 2023;
Brown et al., 2020).

Keeping model weights private (regardless of whether the model architecture is
public) limits the ability of external observers to examine the model and its
performance properties. This presents two key concerns. Firstly, it can be
hard to verify any claims that are made about a model’s performance, either in
the form of scientific evaluation results or commercial marketing. Secondly,
performance characteristics that were not intended or tested for are hard to
examine. Instances of racial and gendered bias have been found post-hoc in
production ML models (Buolamwini & Gebru, 2018). These unwanted model
performance characteristics, such as bias, come in many forms and are often
not initially tested for, leaving it to the public and interested parties to
determine the characteristics of a model.

Tools such as algorithmic audits (Buolamwini & Gebru, 2018; Raji & Buolamwini,
2019b) can partially allow auditing via API, but come at a significant expense
and are not possible when models are not publicly facing. Many high-risk and
customer-impacting models exist without public APIs, such as resume screeners
(Bogen & Rieke, 2018) or models used internally by law enforcement or
governments. Even cases where models are public can benefit from verifiable
evaluations given the technically challenging and expensive process of running
these models.

While a model creator or provider can always state claims about a specific
model’s performance on an evaluation task or set of characteristics (often in
the forms of documentation or model cards such as Mitchell et al. (2019)),
this is insufficient to verify a model being used has a performance
characteristic. To verify the claimed capability, an end user must be able to
confirm that the model in use was run on the benchmark in question, that it
performs at a desired level, and that the same model is being used at a later
date. This problem is not purely academic, as some scholars have raised
concerns that OpenAI’s models have degraded over time (Chen et al., 2023).

##### The problem.

Users of AI systems, have a vested interest in knowing about the performance
capabilities of these systems. At the same time, high-performance AI systems
may be more expensive to provide, or systems that perform well on certain
benchmarks may underperform elsewhere. This creates incentives for model
providers to provide users with a different model than advertised. Users
without technical sophistication or computational resources struggle to
confirm that the models they are using meet advertised benchmarks. While the
EU AI Act and the US Executive Order on AI include transparency obligations
for model providers to share technical details about the model (The White
House, 2023; European Commission, 2021), enforcing compliance with this
requirement is extremely challenging in practice (especially as the market for
AI becomes increasingly fragmented). We sketch a technical safeguard against
this practical problem by allowing AI system users to verify the properties of
the model they are using without requiring them to run their own tests or
trust model providers.

##### Contributions.

To address these challenges, this work presents the first end-to-end
demonstration of using zero-knowledge machine learning to generate evaluation
proofs on arbitrary ML models. In doing so, this work contributes:

  * •

A framework for general-purpose verifiable attestation of neural networks and
other models across a variety of datasets and tasks using fully succinct non-
interactive proofs.

  * •

A challenge-based model for checking model inference matches performance-
attested model weights.

  * •

Analysis of computational costs and complexity of model evaluation to help
present realistic incentive models.

  * •

A demonstration of evaluation across traditional non-neural ML models, multi-
layer perceptions, convolutional neural networks, long short-term memory
networks, and small transformers.

  * •

A working implementation for public use with arbitrary models using the ezkl
toolkit.

Figure 1: System diagram of verifiable ML evaluation using the zkSNARK ezkl
toolkit. A model can be compiled into a proving key ($pk$) and verification
key ($vk$) which can be used to generate repeated inference proofs over a
dataset ($\pi$), which can then be aggregated into a verifiable evaluation
attestation. Using the same proving and verification keys, any future
inference of a model can be checked to confirm a model with the same model
weight hash, $H(W)$, was used to generate the output. Inference data can be
arbitrary.

##  2 Background and Related Works

The field of secure inference of ML models has grown rapidly from specialized
interactive proof protocols (Ghodsi et al., 2017) to more general purpose
approaches, such as through multi-party computation (Knott et al., 2021;
Mishra et al., 2019) and ML inference using homomorphic encryption (Lou &
Jiang, 2021; Juvekar et al., 2018). These approaches, while preserving privacy
of inputs, fail to provide publicly verifiable proof that ML inference
occurred correctly.

Instead, much attention has turned to the inference of ML models inside
zkSNARKs (Zero-Knowledge Succinct Non-Interactive Argument of Knowledge).
Early work in zero-knowledge machine learning (zkML) (Feng et al., 2021) drew
on older slower proving systems (Groth, 2016), or using model architecture
specific optimizations (Liu et al., 2021; Lee, 2020) (such as tailoring for
convolution neural networks). These limitations were addressed by Kang et al.
(2022), which followed ezkl by leveraging Plonkish arithmetization through the
Halo2 (zcash, 2022) proving system.

This work builds upon and heavily leverages the
ezkl111https://github.com/zkonduit/ezkl toolkit, which itself is underpinned
by the Halo2 proving system. This toolkit is continually under development,
targeting the deployment of verifiable execution of ML models on-chain in Web3
contexts.

Although there is a related growing body of literature on verifiable training
(Sun & Zhang, 2023; Garg et al., 2023; Jia et al., 2021), we do not use or
address the issue of training here.

The idea of using zkML as a tool to create verifiable accuracy claims emerges
naturally as an extension from previous zkML work. Previous works mention
verifiable accuracy claims as applications of their zkML proof approaches,
with Kang et al. (2022) adding an additional inventive model around the idea
of a variable claim check on zkML over a constrained set of models. Weng et
al. (2022) takes this idea and extends it in a constrained multi-stakeholder
environment context for CNNs using collaborative inference and homomorphic
encryption, and recent work has shown similar trustless fairness benchmarking
approaches using logistic regressions (Tang et al., 2023). This work
differentiates from the above as the first to provide a generalizable
framework for neural network benchmarking, going much further in identifying
constraints to provide a practical framework for deployment, as well as
discussing choices of evaluation data across context and model type.

This work builds on the burgeoning fields of ML reproducibility (Beam et al.,
2020a; Semmelrock et al., 2023; Albertoni et al., 2023; Belz et al., 2023;
Pineau et al., 2021; Beam et al., 2020b; Olszewski et al., 2023; Gundersen et
al., 2022) and AI fairness (Muthukumar et al., 2018; Mehrabi et al., 2019;
Corbett-Davies & Goel, 2018; May et al., 2019; Raji & Buolamwini, 2019a;
Kärkkäinen & Joo, 2021; Liu et al., 2023; Parraga et al., 2023). While prior
work seeks to ensure unbiased, reproducible, and fair ML models it does not
address the practical risks that model providers will create unbiased and
reproducible models, and then later choose to deploy lower quality models due
to cost, compute, or performance incentives. We seek to pragmatically address
this issue to give users the ability to verify that the model they are using
actually meets relevant benchmarks of bias and fairness.

##  3 Problem Definition

A model provider has a private model, $f(\cdot,\cdot)$, with weights $W$. The
provider wants to make a claim, $C$, about the model’s performance over a
labeled dataset $\\{(x_{i},y_{i})|i\in{0,\ldots,N}\\}$. Previously, the
provider would publish the model outputs
$\\{(x_{i},f(x_{i}))|i\in{0,\ldots,N}\\}$ or an aggregation of those outputs
(e.g., $\frac{1}{N}\sum_{0}^{N}\left|y_{i}-f(x_{i})\right|$).

Here, we want the ability to verify these published facts without needing the
full weights. To do so, we publish the results and an attached proof of
inference $\pi$ containing the model weights hash $H(W)$. The output then
becomes $\\{(x_{i},f(x_{i}),\pi,H(W))|i\in{0,\ldots,N}\\}$ or an aggregated
claim such as $(\Pi,H(W),\frac{1}{N}\sum_{0}^{N}\left|y_{i}-f(x_{i})\right|)$
where $\Pi$ is a meta-proof that verifies and attests to all included
inferences being valid and the aggregation step being correctly computed. It
must be possible to verify that the inferences are valid and that the model
weight hash matches those inferences.

This is sufficient for scientific result confirmation but insufficient for
models being deployed in the real world. To confirm the future use of a model
is done by the same model with matching performance characteristics, a proof
of inference must be done. This can be achieved by sharing with an end user
the result of the inference on their input $x^{*}$ as well as a proof of
inference at the model weights hash $(x^{*},f(x^{*}),\pi,H(W))$. This allows
the end user to confirm the model they access matches the model of the
performance claim by verifying that the proof is valid and the corresponding
model weight hash matches the original claim.

##  4 System Design

To generate and audit verifiable evaluation attestations, the system follows a
broad four-step process shown in Figure 1 leveraging the ezkl toolkit.

First, a model of interest that has been trained and prepared for deployment
is set up for inference. The model is converted into a standard model format
(ONNX) and a circuit corresponding to the internal operations of the inference
is created. This circuit generation process can be calibrated according to
accuracy vs. resource tradeoffs and generates a large proving key, $pk$, and a
verification key, $vk$, for the proof setup. This proof setup takes the form
$\text{Setup}(1^{\lambda},W,f)\rightarrow(pk,vk)$ using a standard security
parameter $\lambda$, the model weights, and architecture to produce a
commitment to the execution of a circuit. This circuit corresponds to the
sequence of operations used in the inference of the model. This setup step
composes many different proof arguments to provide an extremely flexible
approach to model setup that works for the vast majority of ML model
architectures, with technical details of how this was achieved in Appendix
A.1.

Second, an evaluation dataset is selected to confirm performance or check for
bias. Inference over the dataset is performed using standard practices to
produce a set of input files containing input-output pairs $(x,y)$.

Third, for each data pair, a witness file is generated (including some
quantization) for the $(x,y)$ pair, and a computational inference proof is
created as a zkSNARK. This proof takes $W$ as a secret input value to the
circuit and $x$ as a public input value and uses the previous commitment $pk$
to generate the proof $\pi$. This
$\text{Prove}(pk,W,x,y)\rightarrow\pi\supset\left\\{H(W),y\right\\}$ is a
computationally expensive step, with more details in Appendix A.3.

These proof files can be aggregated into a verifiable evaluation attestation,
including artifacts showing model performance and consistent model weight
hashes. This can either be done by naively bundling the inference proofs or
through an additional aggregation circuit that checks if all the proofs are
valid and combines all the outputs into one. This additional setup comes with
important additional privacy considerations for the model results and is
elaborated in Appendix A.4. This claim can be publicly published to the world
without the risk of leaking model weights.

At a later time, a model user or auditor who sees the result of a model
inference $(x^{*},y^{*})$ can challenge the model provider to confirm which
model was used to generate this pair. At an increased cost over standard
inference (levied on either the auditor or provider), the provider holding the
private model weights can be asked to prove using a new zkSNARK that such an
input-output pair can be generated within accuracy bounds from a model with a
matching circuit and model weight hash. By performing
$\text{Prove}(pk,W,x^{*},y^{*})\rightarrow\pi^{*}$ using the same $W$ and
$pk$, the provider can prove that they can generate $y^{*}$ using the model
with $H(W)$, and hence the model with the same benchmark properties from the
verifiable evaluation attestation. If a proof cannot be generated with
matching hashes, the challenge has been failed. More discussion of this
challenge based approach can be found in subsection A.5.

###  4.1 Trust Model & Example Uses

To identify the need for this system, we outline two simple real-world threat
models where verifiable evaluations are needed.

Firstly, consider the case of a new model architecture with private weights
being published in an academic context where the authors make a claim of the
model’s performance on a benchmark. As the model weights are private, model
users and auditors cannot verify that the model performs at the described
levels. The public needs to verify only that a model _exists_ such that its
execution over a dataset produces the benchmark result. The verifiable
evaluation performance attestation demonstrates that the authors have a set of
model weights that can achieve such performance on an evaluation. Even though
weights are kept private, the general architecture of the model (e.g., whether
it is a CNN or Transformer) will be implicitly leaked in the current proof
system (although future work could obfuscate some elements of this such as the
number of layers in a CNN at the cost of proof size).

Secondly, consider the case of any model being used in production, via a
publicly facing API or for internal use only. As above, a model consumer wants
to know that there exists a model that performs well and has the described
characteristics (such as not exhibiting racial bias on test sets). However,
the model consumer also wants to know that this well-performant model is the
one being used during inference. This motivates the model weight hash
verification during inference time. Performing zero-knowledge inference is a
computationally expensive process compared to standard ML inference.

The model consumer, for a known cost, could challenge the model provider to
present a proof of inference and the model weight hash for an input of
concern. In doing so, this provides only a probabilistic guarantee of model
use but minimizes costs. This also extends to more sophisticated attacks where
the attested model is sometimes used but is occasionally swapped with a
different model (such as to minimize costs during high server load or replace
safeguarded models with more performant ones on difficult queries).

Alternatively, in cases of high concern, an inference proof could be generated
and checked for every run. To do so, a “predict, then settle” strategy can be
used, in which the model inference is provided immediately, but the proof
(possibly computed in parallel) is provided at a delay after the proof process
completes. This delay could be seconds, minutes, or more and still provide
useful finality that doesn’t slow down the delivery of the result. This is
especially relevant in legal and regulatory regimes where model performance
needs to remain auditable over time, including contexts such as using FDA-
approved models in medical settings or model vendors used in high-risk
settings.

Note that conditional on randomness in inference and reasonable accuracy
bounds due to quantization, an audit can be performed post hoc on an inference
output pair so long as a verifiable evaluation attestation exists. A longer
discussion of challenge-based model auditing can be found in Appendix A.5.

##  5 Experimental Results

To demonstrate the flexibility and utility of the verifiable model
attestations, we perform two groups of experiments. First, we show an
evaluation of example models from a range of modalities, ranging from bias
checking of facial detection via standard convolutional neural networks to
limited inference of a language model.

Second, we perform inference on a series of network architectures at varying
model sizes to estimate total memory and compute time requirements, so as to
estimate total costs of benchmarking. Code to rerun experiments and use the
system can be found online222https://github.com/tobinsouth/VerifiableEvals.

###  5.1 Example Verifiable Model Attestations

While general in design, the specifics of generating a verifiable evaluation
attestation vary across model types. Any model that can be expressed as a
computational graph in ONNX can be evaluated, and a sample of example models
is shown in Table 1.

The simplest such example of an evaluation would be a multi-layer perceptron
(MLP) benchmarking on a small dataset such as MNIST (Deng, 2012).

An inference proof is generated on each flattened image from the held-out test
split to show its mapping from the input to a number. Bundling these inference
proofs with the model weight hash and the ground truth values allows an
inspector to evaluate the model accuracy by simply summing across the witness
outputs and the ground truths in the attestation. Given the openness of the
dataset, there is no need to privatize the ground truths. However, the naive
bundling of the inference proofs for an attestation allows inspectors to see
exact model outputs. Instead, a simple zero-knowledge proof (ZKP) could be
made to verify each inference with the verification key, confirm $H(W)$
remains constant, and perform the simple
$\tilde{y}_{i}\mathrel{\overset{\scriptscriptstyle?}{\scriptstyle=}}y_{i}$
comparison across all the test data to produce a verifiable evaluation
attestation without releasing individual data. In this case, the information
in the naive bundle, including the ground truths are provided as witnesses to
the proof with model outputs as private inputs and ground truths as public
inputs. The full attestation with all test data proofs and a verification key
is less than 100MB.

Other small models can behave exactly the same. A benchmark over a CNN applied
to MNIST can be done exactly the same (with different performance
characteristics as explored below).

Models such as linear regression, SVM, or random forest can all be benchmarked
in this approach. While these are not natively convertible to ONNX, there are
a variety of tools to convert standard sklearn models into tensor-based models
for use on ONNX such as hummingbird (Nakandala et al., 2019) and sk2torch. We
use a small dataset (Bennett & Mangasarian, 1992) to demonstrate their use due
to their small size.

Models can be more than classifiers. Consider a variational autoencoder (VAE)
(Kingma & Welling, 2014) trained on the CelebA faces dataset (Liu et al.,
2015). While full inference of VAE can be proved, the internal
reparameterization can create minor challenges for proofs due to the random
generation. Instead we prove a partial execution of the VAE showing decoding
from the latent space. Here the witness inputs are points in the latent space
to be passed into the decoder and a naive bundle will contain these points
with proven image outputs from the latent space. This provides an interesting
approach to allowing for verifiable statements about the latent space. So long
as the space has been sufficiently sampled (such as via a grid across the
space centered around the mean) an inspector could see if any undesirable
outputs are generated or inspect to see the degree to which certain properties
of the latent space (such as race in facial images) are clustered.

This generative approach can be applied to small language models as well.
Autoregressive models such as an LSTM (Sak et al., 2014) or a decoder-only
transformer (Radford et al., 2019) require a proof for each inference (e.g.,
each token they generate). This makes the computational cost of generating a
long sequence of tokens quite high. It’s worth noting, however, that these
proofs can be parallelized. Since the witness can be generated preemptively
before proving, the full sequence of tokens can be generated from normal
inference to create a series of witnesses that autoregressively show the next
tokens being generated, which can then each be proven in parallel. We
demonstrate very small language models
(nanoGPT333https://github.com/karpathy/nanoGPT, designed to replicate GPT-2)
running on a sample of HumanEval (Chen et al., 2021). These language models
can be pretrained as the verifiable benchmarking attestation only requires
inference.

We assume a public tokenizer and pass pre-tokenized sequences of tokens from
the prompt into the model to generate the next token. For simple benchmarking
datasets with a single token answer from multiple choice reasoning (such as
MMLU (Hendrycks et al., 2021)) this can be straightforward to benchmark and
package. However, the space of LLMs has opened more rich and complex
benchmarks such as HumanEval (Chen et al., 2021) which requires generating
longer strings as code and then evaluating the execution that code. Creating a
naive bundle as a verifiable evaluation attestation is simple, requiring only
that all token generation steps be put together such that an inspector can
confirm that the code was generated by the model with the hash $H(W)$.
Generating a zero-knowledge proof to privatize the generated code while
attesting to its performance is much harder. This would require not only
verifying each inference using the verification key, but also doing so in the
correct order ensuring coherence of the previous tokens in the witness, and
verifying the correct execution of the code and its output, which is beyond
the scope of this work.

The above approaches can be extended to a variety of models, such as other
autogressive models or new language model architecture, diffusion models which
behave much like VAE decoders, or sequence to sequence models such as Wav2Vec
(Baevski et al., 2020) or Whisper (Fedus et al., 2022) which apply the public
tokenizer assumption to audio processing and generate a sequence output as
witness. Larger models, such as a mixture of experts, are also possible, with
model size being the only constraint. As we see below, proof time and resource
requirements grow dramatically with large models.

| Model
---
| Model
---
Params
| Model
---
Flops
| Model
---
Constraints
| Prove
---
Time (s)
| Verify
---
Time (s)
| Proof
---
Size
| PK
---
Size
| VK
---
Size
Linear Regression | 30 | – | 62 | 0.1 | 0.01 | 13K | 715K | 1.7K
SVM | 30 | – | 626 | 0.3 | 0.02 | 23K | 16M | 2.5K
Random Forest | 80 | – | 3627 | 2.9 | 0.02 | 26K | 276M | 2.7K
MLP | 3610 | 3500 | 1920 | 0.3 | 0.02 | 21K | 14M | 2.3K
Small CNN | 19870 | 68600 | 35844 | 3.1 | 0.03 | 15K | 390M | 1.8K
VAE (decoder) | 1065747 | 12582912 | 2016632 | 142 | 0.42 | 1.9M | 16G | 2.5K
LSTM | 29184 | 950272 | 495712 | 35 | 0.10 | 41K | 4.1G | 2.5K
nanoGPT | 250624 | 51396608 | 9398936 | 2781 | 2.69 | 0.7M | 219G | 4.2K
Table 1: A collection of example models, their parameter sizes, and the total
resource and time requirements for generating an inference proof. Larger
models take longer and have larger proving key sizes, but proof size,
verification key sizes, and verification time all remain low.

###  5.2 Increasing Costs from Increasing Size

Figure 2: Time and storage requirements for model inference proofs for
increasing model sizes across multi-layered perceptions (MLP), convolutional
neural networks (CNN), and attention-based transformers (Attn). Model
requirements scale linearly with the number of constraints in the proof, which
is driven by the number of operations used in a model inference. Model
parameters heterogeneously impact the number of operations in model inference.
Inferences run on a commodity compute cluster slower than in Table 1.

As has been seen widely in ML, increasing numbers of model parameters can lead
to dramatically increased performance (Kaplan et al., 2020). However, these
larger models also require more computation and greater resource requirements
for inference. The same is true when performing inference of models inside
zkSNARKs.

Increasing the number of operations a model performs—often measured in
multiply–accumulate operations (MACs) or floating-point operations
(FLOPs)—increases linearly the number of constraints that need to be proved.
While the total number of parameters in a model helps determine the number of
operations, this is not a direct relationship. Different model architectures
use parameters differently, and hence some models (such as CNNs) have a far
greater number of operations per parameter than others (such as linear layers
in deep neural networks). This relationship can be seen in Figure 2. The CNNs
have a lower scaling ratio as optimizations have been made to remove
irrelevant operations (such as kernels operating on zero-values) which are
included in pytorch operation counts but not in the number of constraints.

To generate an inference proof, the prover must make a KZG commitment (Kate et
al., 2010) to all the operations needed for inference. This requires a
structured reference string (SRS) from a powers of tau ceremony (Nikolaenko et
al., 2022) that has at least as many rows as there are constraints. These SRS
files and the resulting proving keys they enable can become very large on the
order of multiple gigabytes for reasonably sized models to hundreds of
gigabytes for extremely large models. In general, this large proving key size
and the memory requirements it puts on a proving system are the limitations on
model size.

While these keys can be quite large, the burden is on the prover. The
resulting proof sizes and verification keys are extremely small. These too
scale linearly with model operations but at a far slower rate, allowing for a
large number of proof files and their verification key to be shared without a
large file size burden. In turn, the verification time is also extremely
quick.

##  6 Discussion

While both flexible and powerful, this system is still extremely slow. As a
result verifiable evaluations on datasets is expensive. This motivates key
classes of questions for discussion. The first centers around how to estimate
current evaluations costs and what future research may provide to reduce them.
Second, given limited resources important choices must be made over what
benchmarks to run. Third, a discussion of how speed improvements could be
achieved. Finally, we turn attention to private evaluations.

###  6.1 Size and Cost Analysis

For a given model and dataset, the total time it would take to perform
inference proofs over the dataset and aggregate it can be roughly
approximated. Each specific model architecture has a different relationship
between its parameters and the number of operations it performs during
inference. However, once a model is profiled for the number of operations (and
hence the number of constraints), which can be done extremely efficiently,
estimating the total cost to benchmark the model on a given dataset is easy.

The proof time complexity for a given model is determined by the number of
constraints needed to prove. For a given number of constraints, $n_{con}$, the
proof rounds up to the nearest SRS file row count, which provides a proof with
time complexity of $O\left(2^{\lceil{\log_{2}(n_{con})}\rceil}\right)$ which
for practical purposes is approximately linear $O(n_{con})$. While the
relationship between parameters, operations (MACs), and constraints is usually
linear, stating the complexity as $O(n_{parameters})$ masks the varied scaling
rate between parameters and constraints across different model types.

During experimentation, we found proving key size ($pk$) to be the largest
limiting factor, even though it grows $O(n_{con})$, as large models require
larger memory space to hold the proving key, increasing the size and cost of
the compute cluster used.

To estimate the total compute cost needed after determining the number of
constraints for a model, the estimated proving time can be multiplied by the
size of datasets to get total runtime, which can be cost calculated for
whatever compute cluster has sufficient RAM to hold the proving key. While
this can take some experimenting to determine proving speed for a given
cluster size, it can provide robust cost estimates for large datasets.

Beyond the construction of the verifiable evaluation attestation, these cost
estimates are useful for designing costs and incentive structures for model
challenges. If challenging a model is costly, either the challenger or
provider (or perhaps the loser of the challenger) will need to bear the cost.
Having a publicly known cost for model inference makes this process fairer.

###  6.2 Dataset Selection Choice

A key motivation for this system is the desire to bring transparency,
accountability, and robustness to model benchmarking and its importance for
fairness and bias checking. This fundamentally relies on our ability to have
robust benchmarks to assess model qualities.

However, benchmarks are often incomplete evaluation measures for models (Liao,
2021) and can struggle with identifying problems such as intersectional bias
(Lalor et al., 2022).

While recent work has proposed new more comprehensive benchmarks (especially
in the field of NLP such as Liang et al. (2023)), further work will be needed
for more robust and complete benchmarks; and addressing the challenges of bias
in models is more complicated, requiring changes to values and norms around
how AI is built and deployed (Schwartz et al., 2022; Suresh & Guttag, 2019).

This is future underscored by the push towards prioritizing AI safety (The
White House, 2023), which includes comprehensive model evaluations, essential
for discerning a model’s capabilities in both desired and potentially harmful
tasks. Tasks such as red-teaming model performance or other alignment checks
could also be included in a verifiable evaluation attestation.

###  6.3 Future Speed Improvements

One of the main constraints on the application of this system (especially with
regard to large foundation models) is the speed with which proofs can be
generated. This work builds on the ezkl toolkit, which is constantly
undergoing speed improvements through optimizations to circuits for inference
operations and engineering improvements. Future work will improve these speeds
through proof splitting and parallelization (Camuto et al., 2023c), GPU
acceleration (Camuto et al., 2023b; Derei, 2023), or using alternative
underlying proof systems (Kothapalli et al., 2022; Boneh et al., 2020; Setty,
2020). Other approaches such as cqlin from Eagen & Gabizon (2023) show promise
for ML, while other unpublished work has performed further optimizations.
(Kang, 2023).

Beyond changes to the proving system and hardware acceleration, changes to
models can have nontrivial effects on model proof runtimes. It is possible to
optimize the design of deep neural networks for more efficient inference in
private settings (Jha & Reagen, 2023). Similarly, choices can be made during
benchmarking design, such as running model inference through batches, which
could have some small speed improvements at the cost of larger proof
requirements.

###  6.4 Private Benchmarks

In the system design used here, the _choice_ of benchmark has been made
public. While the exact outputs of the model over a benchmark can be hidden
through aggregation (see Appendix A.4 for more details), the choice of the
dataset is fundamental to knowing what the model performance attests to.
However, one could imagine a scenario where a model provider wants to prove
their performance over a private benchmark dataset. This is immediately
possible by making model inputs and outputs private after aggregation of
inference proofs, but it requires additional work to justify, as these results
are information-free in a vacuum.

Two models could be compared on private benchmarks, where a hash of a
benchmark dataset is shared without ever sharing the model. This could allow a
model provider to show improvements in performance over time with new models
without ever sharing the benchmark that they use. Similarly, we could have
benchmarks for general foundation model performance that are not released to
the internet to avoid training on the test, which could position such a system
run by a company or regulator as a gold standard of performance measures.

Alternatively, you may want to have a benchmark on a private dataset but with
additional abilities to prove facts about the dataset. For example, a
verifiable evaluation attestation could be made, including a dataset hash, and
various facts about the dataset and its properties could be shared via
separate zkSNARKs that include the dataset hash.

##  7 Conclusion

We present a novel method for verifiable performance and bias benchmarking of
ML models using zkSNARKs. This approach addresses the critical issue of
verifying model performance claims in environments where model weights are
kept private, which is increasingly common in commercial ML applications. This
helps ensure transparency and accountability in model evaluations,
particularly in high-stakes scenarios where model reliability and fairness are
paramount. The system packages together repeated model inference proofs to
demonstrate the accuracy of models either through simple bundling of small
proofs and verification files or through meta-proofs of performance over model
inference proofs. The system’s flexibility was demonstrated across a range of
ML models ranging from small perceptrons and regression models to medium-sized
transformers.

The system faces challenges, primarily related to the computational cost and
time required for generating proofs, especially for large-scale models. The
trade-off between accuracy and resource demands during the proof generation
process remains a critical consideration. Future improvements in the speed and
efficiency of the zkSNARKs, possibly through hardware acceleration or proving
system changes, could significantly improve the cost and applicability of this
approach. Moreover, the discussion around private benchmark datasets opens new
avenues for maintaining confidentiality while ensuring model integrity. In
conclusion, this work lays the groundwork for a more transparent and
verifiable approach to ML model evaluations.

## Broader Impact

Extensive work in machine-learning seeks to ensure that models are fair,
unbiased, reproducible, and transparent. However, this work seldom confronts
the reality that model developers operate within economic incentives that may
cause them not to give users access to models that achieve high performance.
AI regulation, like the EU AI Act, provides additional transparency
obligations but these may be challenging to enforce against the growing number
of closed-source model providers. The goal of this work is to demonstrate a
technical approach that allows users without computational resources or
technical sophistication confirm that they are using the correct models for
their application, without having to trust the model providers.

##### Ethics and Societal Consequences

Ethics in machine learning is gaining prominence both research and applied
communities. Frequently, however, prescriptive or aspirational approaches fail
to pragmatically consider market incentives that may cause model providers to
misrepresent the models they offer. In our view, it would be a mistake to
ignore this practical dimension. While we take a somewhat pessimistic
view—assuming that at least some providers of AI systems are untrustworthy—we
do not believe that our approach undermines trust or furthers mistrust;
instead our aim is to give all users the ability to verify claims made by
model providers. The goal of our work is to empower research done in
responsible ML by holding model developers accountable and shifting the costs
of verification away from model users. Our hope is that this will further
responsible ML research and practice.

## References

  * Albertoni et al. (2023) Albertoni, R., Colantonio, S., Skrzypczy’nski, P., and Stefanowski, J.  Reproducibility of machine learning: Terminology, recommendations and open issues.  _ArXiv_ , abs/2302.12691, 2023.
  * Baevski et al. (2020) Baevski, A., Zhou, Y., Mohamed, A., and Auli, M.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  _Advances in neural information processing systems_ , 33:12449–12460, 2020.
  * Beam et al. (2020a) Beam, A., Manrai, A. K., and Ghassemi, M.  Challenges to the reproducibility of machine learning models in health care.  _JAMA_ , 2020a.
  * Beam et al. (2020b) Beam, A. L., Manrai, A. K., and Ghassemi, M.  Challenges to the Reproducibility of Machine Learning Models in Health Care.  _JAMA_ , 323(4):305–306, 01 2020b.  ISSN 0098-7484.
  * Belz et al. (2023) Belz, A., Thomson, C., Reiter, E., Abercrombie, G., Alonso-Moral, J. M., Arvan, M., Cheung, J. C. K., Cieliebak, M., Clark, E., van Deemter, K., Dinkar, T., Dusek, O., Eger, S., Fang, Q., Gatt, A., Gkatzia, D., Gonz’alez-Corbelle, J., Hovy, D., Hurlimann, M., Ito, T., Kelleher, J. D., Klubicka, F., Lai, H., van der Lee, C., van Miltenburg, E., Li, Y., Mahamood, S., Mieskes, M., Nissim, M., Parde, N., Pl’atek, O., Rieser, V., Romero, P., Tetreault, J., Toral, A., Wan, X.-Y., Wanner, L., Watson, L. J., and Yang, D.  Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in nlp.  _ArXiv_ , abs/2305.01633, 2023.
  * Bennett & Mangasarian (1992) Bennett, K. P. and Mangasarian, O. L.  Robust linear programming discrimination of two linearly inseparable sets.  _Optimization Methods and Software_ , 1:23–34, 1992.
  * Bogen & Rieke (2018) Bogen, M. and Rieke, A.  Help wanted: An examination of hiring algorithms, equity, and bias.  2018\.
  * Boneh et al. (2020) Boneh, D., Drake, J., Fisch, B., and Gabizon, A.  Halo infinite: Recursive zk-snarks from any additive polynomial commitment scheme.  _Cryptology ePrint Archive_ , 2020.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Buolamwini & Gebru (2018) Buolamwini, J. and Gebru, T.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _Conference on fairness, accountability and transparency_ , pp. 77–91. PMLR, 2018.
  * Camuto et al. (2023a) Camuto, A., Gănescu, B., Passerat-Palmbach, J., and Morton, J.  Honey I snarked the GPT.  EZKL Blog, Oct 2023a.
  * Camuto et al. (2023b) Camuto, A., Wawrzyniak, S., and Morton, J.  Steps in hardware, leaps in performance.  EZKL Blog, Nov 2023b.
  * Camuto et al. (2023c) Camuto, A., Wawrzyniak, S., and Morton, J.  Splitting and parallelizing proofs.  EZKL Blog, Oct 2023c.
  * Camuto et al. (2023d) Camuto, A., Wawrzyniak, S., and Morton, J.  Removing additional commitment cost.  EZKL Blog, Oct 2023d.
  * Chen et al. (2023) Chen, L., Zaharia, M., and Zou, J.  How is chatgpt’s behavior changing over time?  _arXiv preprint arXiv:2307.09009_ , 2023.
  * Chen et al. (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W.  Evaluating large language models trained on code, 2021.
  * Corbett-Davies & Goel (2018) Corbett-Davies, S. and Goel, S.  The measure and mismeasure of fairness: A critical review of fair machine learning.  _ArXiv_ , abs/1808.00023, 2018.
  * Deng (2012) Deng, L.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * Derei (2023) Derei, T.  _Accelerating the PlonK zkSNARK Proving System using GPU Architectures_.  PhD thesis, Lehigh University, 2023.
  * Eagen & Gabizon (2023) Eagen, L. and Gabizon, A.  cqlin: Efficient linear operations on kzg commitments with cached quotients.  Cryptology ePrint Archive, Paper 2023/393, 2023.
  * European Commission (2021) European Commission.  Artificial intelligence act, 4 2021.
  * Fedus et al. (2022) Fedus, W., Zoph, B., and Shazeer, N.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.  _The Journal of Machine Learning Research_ , 23(1):5232–5270, 2022.
  * Feng et al. (2021) Feng, B., Qin, L., Zhang, Z., Ding, Y., and Chu, S.  Zen: An optimizing compiler for verifiable, zero-knowledge neural network inferences.  2021\.
  * Garg et al. (2023) Garg, S., Goel, A., Jha, S., Mahloujifar, S., Mahmoody, M., Policharla, G.-V., and Wang, M.  Experimenting with zero-knowledge proofs of training.  In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_ , pp. 1880–1894, 2023.
  * Ghodsi et al. (2017) Ghodsi, Z., Gu, T., and Garg, S.  Safetynets: Verifiable execution of deep neural networks on an untrusted cloud.  In _Neural Information Processing Systems_ , 2017.
  * Groth (2016) Groth, J.  On the size of pairing-based non-interactive arguments.  _IACR Cryptol. ePrint Arch._ , 2016:260, 2016.
  * Gundersen et al. (2022) Gundersen, O. E., Shamsaliei, S., and Isdahl, R. J.  Do machine learning platforms provide out-of-the-box reproducibility?  _Future Generation Computer Systems_ , 126:34–47, 2022.
  * Haböck (2022) Haböck, U.  Multivariate lookups based on logarithmic derivatives.  _Cryptology ePrint Archive_ , 2022.
  * Hendrycks et al. (2021) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Jha & Reagen (2023) Jha, N. K. and Reagen, B.  Deepreshape: Redesigning neural networks for efficient private inference.  _ArXiv_ , abs/2304.10593, 2023.
  * Jia et al. (2021) Jia, H., Yaghini, M., Choquette-Choo, C. A., Dullerud, N., Thudi, A., Chandrasekaran, V., and Papernot, N.  Proof-of-learning: Definitions and practice.  _2021 IEEE Symposium on Security and Privacy (SP)_ , pp. 1039–1056, 2021.
  * Juvekar et al. (2018) Juvekar, C., Vaikuntanathan, V., and Chandrakasan, A.  Gazelle: A low latency framework for secure neural network inference.  In _27th USENIX Security Symposium (USENIX Security 18)_ , pp. 1651–1669, 2018.
  * Kang et al. (2022) Kang, D., Hashimoto, T., Stoica, I., and Sun, Y.  Scaling up trustless dnn inference with zero-knowledge proofs.  _arXiv preprint arXiv:2210.08674_ , 2022.
  * Kang (2023) Kang, D. D.  Tensorplonk: A “gpu”’ for zkml, delivering 1,000x speedups.  Medium, 2023.
  * Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T. J., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.  Scaling laws for neural language models.  _ArXiv_ , abs/2001.08361, 2020.
  * Kärkkäinen & Joo (2021) Kärkkäinen, K. and Joo, J.  Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation.  _2021 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , pp. 1547–1557, 2021.
  * Karpur et al. (2023) Karpur, A., Lahav, D., Matheny, J., Alstott, J., and Nevo, S.  Securing artificial intelligence model weights: Interim report.  2023\.
  * Kate et al. (2010) Kate, A., Zaverucha, G. M., and Goldberg, I.  Constant-size commitments to polynomials and their applications.  In _Advances in Cryptology-ASIACRYPT 2010: 16th International Conference on the Theory and Application of Cryptology and Information Security, Singapore, December 5-9, 2010. Proceedings 16_ , pp. 177–194. Springer, 2010.
  * Kingma & Welling (2014) Kingma, D. P. and Welling, M.  Auto-encoding variational bayes.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2014.
  * Knott et al. (2021) Knott, B., Venkataraman, S., Hannun, A. Y., Sengupta, S., Ibrahim, M., and van der Maaten, L.  Crypten: Secure multi-party computation meets machine learning.  _ArXiv_ , abs/2109.00984, 2021.
  * Kothapalli et al. (2022) Kothapalli, A., Setty, S., and Tzialla, I.  Nova: Recursive zero-knowledge arguments from folding schemes.  In _Annual International Cryptology Conference_ , pp. 359–388. Springer, 2022.
  * Lalor et al. (2022) Lalor, J. P., Yang, Y., Smith, K., Forsgren, N., and Abbasi, A.  Benchmarking intersectional biases in nlp.  In _North American Chapter of the Association for Computational Linguistics_ , 2022.
  * Lee (2020) Lee, S.  vcnn: Verifiable convolutional neural network based on zk-snarks.  2020\.
  * Liang et al. (2023) Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., Newman, B., Yuan, B., Yan, B., Zhang, C., Cosgrove, C., Manning, C. D., R’e, C., Acosta-Navas, D., Hudson, D. A., Zelikman, E., Durmus, E., Ladhak, F., Rong, F., Ren, H., Yao, H., Wang, J., Santhanam, K., Orr, L. J., Zheng, L., Yuksekgonul, M., Suzgun, M., Kim, N. S., Guha, N., Chatterji, N. S., Khattab, O., Henderson, P., Huang, Q., Chi, R., Xie, S. M., Santurkar, S., Ganguli, S., Hashimoto, T., Icard, T. F., Zhang, T., Chaudhary, V., Wang, W., Li, X., Mai, Y., Zhang, Y., and Koreeda, Y.  Holistic evaluation of language models.  _Annals of the New York Academy of Sciences_ , 1525:140 – 146, 2023.
  * Liao (2021) Liao, T.  Are we learning yet? a meta review of evaluation failures across machine learning.  In _NeurIPS Datasets and Benchmarks_ , 2021.
  * Liu et al. (2023) Liu, J., Chen, H., Shen, J., and Choo, K.-K. R.  Faircompass: Operationalising fairness in machine learning.  _ArXiv_ , abs/2312.16726, 2023.
  * Liu et al. (2021) Liu, T., Xie, X., and Zhang, Y.  zkcnn: Zero knowledge proofs for convolutional neural network predictions and accuracy.  _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_ , 2021.
  * Liu et al. (2015) Liu, Z., Luo, P., Wang, X., and Tang, X.  Deep learning face attributes in the wild.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 3730–3738, 2015.
  * Lou & Jiang (2021) Lou, Q. and Jiang, L.  Hemet: A homomorphic-encryption-friendly privacy-preserving mobile neural network architecture.  In _International Conference on Machine Learning_ , 2021.
  * May et al. (2019) May, C., Wang, A., Bordia, S., Bowman, S. R., and Rudinger, R.  On measuring social biases in sentence encoders.  _ArXiv_ , abs/1903.10561, 2019.
  * Mehrabi et al. (2019) Mehrabi, N., Morstatter, F., Saxena, N. A., Lerman, K., and Galstyan, A. G.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54:1 – 35, 2019.
  * Mishra et al. (2019) Mishra, P., Lehmkuhl, R. T., Srinivasan, A., Zheng, W., and Popa, R. A.  D elphi : A cryptographic inference service for neural networks.  2019\.
  * Mitchell et al. (2019) Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., and Gebru, T.  Model cards for model reporting.  In _Proceedings of the conference on fairness, accountability, and transparency_ , pp. 220–229, 2019.
  * Muthukumar et al. (2018) Muthukumar, V., Pedapati, T., Ratha, N. K., Sattigeri, P., Wu, C.-W., Kingsbury, B., Kumar, A., Thomas, S., Mojsilovic, A., and Varshney, K. R.  Understanding unequal gender classification accuracy from face images.  _ArXiv_ , abs/1812.00099, 2018.
  * Nakandala et al. (2019) Nakandala, S., Yu, G.-I., Weimer, M., and Interlandi, M.  Compiling classical ml pipelines into tensor computations for one-size-fits-all prediction serving.  In _System for ML Workshop_. NeurIPS, 2019.
  * Nikolaenko et al. (2022) Nikolaenko, V., Ragsdale, S., Bonneau, J., and Boneh, D.  Powers-of-tau to the people: Decentralizing setup ceremonies.  _Cryptology ePrint Archive_ , 2022.
  * Olszewski et al. (2023) Olszewski, D., Lu, A., Stillman, C., Warren, K., Kitroser, C., Pascual, A., Ukirde, D., Butler, K., and Traynor, P.  ”get in researchers; we’re measuring reproducibility”: A reproducibility study of machine learning papers in tier 1 security conferences.  In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_ , CCS ’23, pp. 3433–3459, New York, NY, USA, 2023. Association for Computing Machinery.  ISBN 9798400700507.  doi: 10.1145/3576915.3623130.
  * Parraga et al. (2023) Parraga, O., Móre, M. D., de Oliveira, C. M., Gavenski, N. S., Kupssinskü, L. S., Medronha, A., Moura, L. V., Simões, G. S., and Barros, R. C.  Fairness in deep learning: A survey on vision and language research.  _ACM Computing Surveys_ , 2023.
  * Pineau et al. (2021) Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d’Alché Buc, F., Fox, E., and Larochelle, H.  Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program).  _J. Mach. Learn. Res._ , 22(1), jan 2021.  ISSN 1532-4435.
  * Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 2019.
  * Raji & Buolamwini (2019a) Raji, I. D. and Buolamwini, J.  Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products.  _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , 2019a.
  * Raji & Buolamwini (2019b) Raji, I. D. and Buolamwini, J.  Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products.  In _Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society_ , pp. 429–435, 2019b.
  * Sak et al. (2014) Sak, H., Senior, A. W., and Beaufays, F.  Long short-term memory recurrent neural network architectures for large scale acoustic modeling.  2014\.
  * Schwartz et al. (2022) Schwartz, R., Vassilev, A. T., Greene, K., Perine, L. A., Burt, A., and Hall, P.  Towards a standard for identifying and managing bias in artificial intelligence.  2022\.
  * Semmelrock et al. (2023) Semmelrock, H., Kopeinik, S., Theiler, D., Ross-Hellauer, T., and Kowald, D.  Reproducibility in machine learning-driven research.  _arXiv preprint arXiv:2307.10320_ , 2023.
  * Setty (2020) Setty, S.  Spartan: Efficient and general-purpose zksnarks without trusted setup.  In _Annual International Cryptology Conference_ , pp. 704–737. Springer, 2020.
  * Sun & Zhang (2023) Sun, H.-L. and Zhang, H.  zkdl: Efficient zero-knowledge proofs of deep learning training.  _ArXiv_ , abs/2307.16273, 2023.
  * Suresh & Guttag (2019) Suresh, H. and Guttag, J. V.  A framework for understanding sources of harm throughout the machine learning life cycle.  _Equity and Access in Algorithms, Mechanisms, and Optimization_ , 2019.
  * Tang et al. (2023) Tang, G., Tan, W., and Cai, M.  Privacy-preserving and trustless verifiable fairness audit of machine learning models.  _International Journal of Advanced Computer Science and Applications_ , 2023.
  * The White House (2023) The White House.  Executive order on the safe, secure, and trustworthy development and use of artificial intelligence, 10 2023.
  * Weng et al. (2022) Weng, J., Weng, J., Tang, G., Yang, A., Li, M., and Liu, J.-N.  pvcnn: Privacy-preserving and verifiable convolutional neural network testing.  _IEEE Transactions on Information Forensics and Security_ , 18:2218–2233, 2022.
  * zcash (2022) zcash.  The halo2 book, 2022.  URL zcash.github.io/halo2/.

##  Appendix A System Details

###  A.1 Flexible Model Setup

One of the key features is the flexible nature of this proof system. The ezkl
toolkit, which underpins the system, takes an ONNX file as input, an open
standard for saving ML models that can be accessed from standard ML libraries.

Each operation in the model’s computational graph is represented by one of the
100 ONNX operation types, and each operation will be converted into a proof
constraint. While simple in concept, implementation details are critical to
achieve efficiency. For example, most operations in the model can be reduced
to Einstein summations to minimize the number of constraint implementations
needed (Camuto et al., 2023a).

These constraints are constructed into a large proving table for Halo2 (zcash,
2022). Many further optimizations occur here to achieve speed improvements
which are detailed elsewhere. Notably, the ability to create parallel region
layouts using cycles, the use of fixed-column lookups to represent nonlinear
functions, and the choice to perform operations using fixed-point arithmetic
for speed (Camuto et al., 2023a).

While generating this circuit for proving there are many calibration choices
to be made around quantization and scale. These can be summarized as tradeoffs
between accuracy and the quantity of resources needed for the proof. In this
work, calibration for resources is preferred for large model sizes.

Beyond the proof circuit for inference of the model operations, we introduce
an additional set of actions in the circuit to calculate a proof of the hash
of the model weights (via ZKG hashing per Camuto et al. (2023d)). This results
in a final proof that contains both the input-output values (which are made
public in the proof circuit) and the model weight hash.

The outputs of this step are the compiled circuit, a large proving key, and a
small verification key.

###  A.2 Proof System and Arguments

The flexibility of the ezkl proof system stems from the library’s reduction of
any zk-equivalent of an ML operation into some combination of four arguments:

  1. 1.

A cumulative dot product argument.

  2. 2.

A cumulative sum/product argument.

  3. 3.

An elementwise addition/multiplication/subtraction argument.

  4. 4.

A lookup argument, used to represent and constrain non-linear functions within
the zero-knowledge-circuit.

These “arguments” are used to enforce and guarantee that the result of a
computation is indeed from a particular agreed-upon computation graph (such as
a neural network).

####  A.2.1 Cumulative Arguments

Arguments 1. and 2. are constructed in a similar fashion. In the case of the
cumulative sum/product consider a vector $\mathbf{x}$ of length $N$. To create
a set of constraints within our zk-circuit we create a new vector $\mathbf{m}$
of length $N-1$ and set the following elementwise constraints:

$x_{i}\circ m_{i}=m_{i+1}\ \forall i\in 1..N,\ \text{where}\ m_{0}=0.$ (1)

$\circ$ is the addition operator in the case of the cumulative sum and the
multiplication operator in the case of the cumulative product. The final
element of the vector $\mathbf{m}$ then represents the cumulative sum or
product, and the constraints above are enforced within the circuit using Halo2
selectors.

The dot product argument is constructed in a very similar fashion. We now have
a second input vector $\mathbf{y}$, also of length $N$ and constrain the
following:

$x_{i}\circ y_{i}+m_{i}=m_{i+1}\ \forall i\in 1..N,\ \text{where}\ m_{0}=0.$
(2)

As before, the final element of the vector $\mathbf{m}$ then represents the
dot product.

####  A.2.2 Elementwise Arguments

Argument 3. is constructed without leveraging the intermediate calculations of
A.2.1. Consider two vectors $\mathbf{x}$ and $\mathbf{y}$, both of length $N$.
We constrain the resulting output $\mathbf{m}$ via the following:

$x_{i}\circ y_{i}=m_{i}\ \forall i\in 1..N,$ (3)

####  A.2.3 Lookup Arguments

The ezkl system leverages Halo2 as a proving backend with a few modifications.
Most significantly, it changes the original lookup argument, typically used to
constraint non-linear functions like $\mathbf{ReLU}$, to the more efficient
logUP lookup argument (Haböck, 2022).

####  A.2.4 More Complex Arguments

More complex arguments such as those for the $\mathbf{min}$ and $\mathbf{max}$
functions, can be constructed as combinations of the above arguments. For
instance the $\mathbf{max}$ argument can be constructed as follows:

  1. 1.

Calculate the claimed

$m=\text{max}(x),$

and instantiate a lookup table $\mathbf{a}$ which corresponds to the
$\mathbf{ReLU}$ element-wise operation.

  2. 2.

Constrain $\mathbf{w}=\mathbf{x}-(m-1)$.

  3. 3.

Use lookup $\mathbf{a}$ on $\mathbf{w}$, this is equivalent to clipping
negative values: $\mathbf{y}=\mathbf{ReLU}(w).$

  4. 4.

Constrain the values $\mathbf{y}$ to be equal to 0 or 1, i.e. use a selector
that enforces that

$y_{i}*(y_{i}-1)=0,\qquad\forall i\in 1\dots N.$

Any non-clipped values should be equal to 1 as at most we are subtracting the
max. This demonstrates that the there is no value of $\mathbf{x}$ that is
larger than the claimed maximum of $\mathbf{x}$.

  5. 5.

We have now constrained $m$ to be larger than any value of $\mathbf{x}$, we
must now demonstrate that at least one value of $\mathbf{x}$ is equal to $m$,
i.e that $m$ is an element of $\mathbf{x}$. We do this by constructing the
argument

$\mathbf{x}=\mathbf{ReLU}(1-\sum_{i}y_{i})=0.$

Note that

$\sum_{i}y_{i}=0\iff z=1$

and thus no values of the witness are equal to

$\text{max}(x).$

Conversely, if

$\sum_{i}y_{i}>=1\iff z=0$

and thus least one value is equal to 1.

###  A.3 Dataset Inference

The most computationally expensive and decision-intensive step is the
inference of the ML model over the benchmarking dataset to generate the
zkSNARKs. One extremely key choice is the selection of a relevant dataset, an
issue we leave for subsection 6.2. Instead, we focus here on the mechanics of
how the inferences can be done.

While zero-knowledge ML is often described in terms of the inference occurring
inside the proof generation, it can more aptly be described as the proof
verifying that an inference was performed; a distinction that has important
practical implications.

Firstly, the model can be run over the dataset inputs as it would in any other
inference context to generate input-output pairs, $(x_{i},y_{i})$. These pairs
are then quantized per the setup choice above to produce the witness inputs,
$(\tilde{x}_{i},\tilde{y}_{i})$. When accuracy is sacrificed, the quantized
input-output pair may be different from the original values by a few percent
(this can be calibrated). Importantly, as far as the test set is concerned,
this accuracy trade-off can be examined _before_ performing the expensive
proof step. The proof step will verify that the witness could be authentically
generated using the model weights (which are treated as private inputs for the
proof). Hence, iterative calibration on the benchmarking test set to ensure
witness data has the performance characteristics is important.

For each of these witness files, a proof is generated using the proving key
and the circuit. This is the slowest part of the stack and scales linearly
with the size of the test set. For each witness file,
$(\tilde{x}_{i},\tilde{y}_{i},H(W))$, a proof file will be generated
$\Pi_{i}$. These proof files are each very small (on the order of kilobytes)
and can be individually verified using the verification key or aggregated.

###  A.4 Aggregation Choices

Once a collection of inference proofs has been generated, there is a question
of how to package them for publication and sharing. The simplest naive
solution is to zip up the proofs, which include the witness outputs, and a
verification key, allowing any future examiner to calculate performance
measures. This is computationally efficient and allows for flexible measures
of performance to be made. However, this naive bundle exposes the test data
outputs to unwanted use (such as training a copycat model based on these
outputs). Even though this is a reduced risk on test benchmarks, some model
providers would prefer not to release inference.

Having the full test data inference results available also affords the
possibility of a post-hoc audit on other properties contained in that data
(e.g., a test set might only be intended for showing performance, but a
careful audit of response performance across classes could reveal biases in
the performance across classes). One such example might be sharing overall
facial recognition accuracy, of which a finer grain analysis would show that
the performance is much larger for white and male faces (Buolamwini & Gebru,
2018). Since proof files and verification keys are very small, these naive
bundles of inference proofs remain small and fast to verify even for large
dataset sizes, with the vast majority of the work in proof generation.

A secondary approach would be to perform a vanilla aggregation circuit which
checks if all the proofs are valid and combines all the outputs into one. This
has similar downsides to compressing the collection but now comes with the
additional cost of the aggregation circuit, which can become quite large. This
zero-knowledge aggregation step makes sense when posting model inferences to a
blockchain, but less so when sharing a performance attestation.

A third approach uses such an aggregation circuit but explicitly creates a
custom Halo2 circuit to aggregate just the result we want (e.g. the accuracy
score, confusion matrix). This allows for a separate privacy regime to be
applied, where data outputs are kept private, and only a verifiable accuracy
metric is shared. This additional step adds complexity both in the form of
additional computational requirements and the need to audit accuracy
calculation code, but adds a significant layer of privacy. This additional
zero-knowledge proof (ZKP) approach would need to be updated for different
model types and outputs (e.g., models with varying input output formats,
models with different accuracy measures, or benchmarks requiring complex
generation of evaluation like the HumanEval example in subsection 5.1.)

####  A.4.1 Posting Performance Claims

Other work has discussed the use of specific billboards for sharing
attestations (Tang et al., 2023), but the portable nature of the verifiable
evaluation attestations is that they can be shared or hosted anywhere and
copied repeatedly without additional risk. As such, posting performance claims
to Github, Papers with Code, Huggingface, IPFS, another third-party public-
facing hosting service, or even one’s own website is sufficient. Ideally, this
posting would then be mirrored elsewhere for accountability.

###  A.5 Challenge-based Production Model Audits

Any inference of a model that produces a visible input output pair
$(x^{*},y^{*})$ can be challenged. This includes regularly scheduled audits
where an inference step is accompanied by an inference proof challenge, as
well as post-hoc audits, where an inference that seems suspicious can be
checked against an existing verifiable evaluation attestation. The latter
approach limits the ability of the model provider to anticipate a challenge,
creating incentives to always use the claimed model. A model user could choose
to challenge a provider randomly, creating a predictable likelihood of bad
actors getting caught, or selectively in cases where a user is dissatisfied
with performance.

The key caveat is that each verifiable inference is vastly more expensive than
standard inference. In cases where audits are legally mandated (such as in
high risk contexts), a model provider could bear this cost. However, in
production deployment models, the provider has an incentive to perform as few
verifiable inferences as possible. As such, the model user could pay an
additional fee for auditing to account for the cost. If an audit passes, the
user is satisfied. However, to make auditing worthwhile in cases of concern, a
reward system could be put in place, where a user is rewarded if an audit
fails. This could be done using escrow or smart contracts, or via simple terms
of service in API usage deals. In the case of a random audit with probability
$p$ at cost $c$ per verifiable inference, a reward of $\frac{c}{p}$ would
cover the costs in expectation for the user with an untrusted model provider.

Further, even a new verifiable evaluation attestation can be created post-hoc.
If concerns are raised about an aspect of model performance (such as a
revealed bias on a subclass of data (Buolamwini & Gebru, 2018)), a new
verifiable evaluation attestation can be created to examine the issue. This
process is extremely costly, matching the cost levels of pre-emptive
evaluation. An approach where a user challenges the performance of the model
is possible, where a user provides a new test set to run on (as suggested by
Kang et al. (2022)) requiring both a larger upfront cost and subsequent reward
incentives. Such larger audits might be requested by civil liberty
organizations or occur due to new standards (such as new bias requirements or
evaluations in medical ML settings).