# opML: Optimistic Machine Learning on Blockchain

KD Conway  Hyper OracleChina<EMAIL_ADDRESS>,  Cathie So  Hyper
OracleChina<EMAIL_ADDRESS>,  Xiaohang Yu  Hyper OracleChina
<EMAIL_ADDRESS>and  Kartin Wong  Hyper OracleUSA<EMAIL_ADDRESS>
(2018)

###### Abstract.

The integration of machine learning with blockchain technology has witnessed
increasing interest, driven by the vision of decentralized, secure, and
transparent AI services. In this context, we introduce opML (Optimistic
Machine Learning on chain), an innovative approach that empowers blockchain
systems to conduct AI model inference. opML lies a interactive fraud proof
protocol, reminiscent of the optimistic rollup systems. This mechanism ensures
decentralized and verifiable consensus for ML services, enhancing trust and
transparency. Unlike zkML (Zero-Knowledge Machine Learning), opML offers cost-
efficient and highly efficient ML services, with minimal participation
requirements. Remarkably, opML enables the execution of extensive language
models, such as 7B-LLaMA, on standard PCs without GPUs, significantly
expanding accessibility. By combining the capabilities of blockchain and AI
through opML, we embark on a transformative journey toward accessible, secure,
and efficient on-chain machine learning.

Blockchain, Machine Learning, Fraud Proof, Rollup, Layer 2

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

In the rapidly evolving digital landscape, technological innovations are
continually reshaping the way we interact with and harness the power of
information. Among these innovations, the convergence of two remarkable
forces, Artificial Intelligence (AI) and blockchain technology, stands out as
a pivotal development. AI, with its capacity for advanced data analysis and
decision-making, and blockchain, a decentralized ledger known for its security
and transparency, have joined forces to explore new frontiers in the digital
realm (Salah et al., 2019; Dinh and Thai, 2018; Yang et al., 2022). As two
pioneering forces, each with its distinct capabilities, AI and blockchain, are
now merging to redefine the boundaries of what’s possible in the digital
world. This synergy has given rise to the concept of ”Onchain AI” (Salah et
al., 2019; Dinh and Thai, 2018; Yang et al., 2022), a paradigm that holds the
promise of delivering decentralized, secure, and efficient AI services
directly within the blockchain network.

However, a prevalent challenge in the current landscape of ”Onchain AI” is the
infeasibility of conducting AI computations directly on chain (Li et al.,
2022). For example, a simple task of naïve matrix multiplication of
$1000\times 1000$ integers would cost over 3 billion gas (Zheng et al., 2021),
which far exceeds the current Ethereum’s block gas limit (Wood et al., 2014).
Consequently, most of these applications resort to off-chain computations on
centralized servers, only uploading the results onto the blockchain. While
this strategy may yield functional results, it inherently sacrifices
decentralization. Such a trade-off not only poses significant security
challenges but also diminishes the core principles of trust and transparency
that blockchain technology aims to uphold.

One alternative approach is to leverage Zero-Knowledge Machine Learning (zkML)
(Zhang et al., 2020; Weng et al., 2021). zkML represents a novel paradigm in
the integration of machine learning and blockchain. zkML’s reliance on zk-
SNARKs (Zero-Knowledge Succinct Non-Interactive Arguments of Knowledge)
(Goldreich and Oren, 1994; Ben-Sasson et al., 2019, 2014; Bootle et al., 2016)
has been pivotal in safeguarding confidential model parameters and user data
during the training and inference processes. This not only mitigates privacy
concerns but also reduces the computational burden on the blockchain network,
making zkML a promising candidate for decentralized ML applications.

While zkML undeniably presents a range of advantages in enhancing privacy and
security within machine learning on the blockchain, it is crucial to
acknowledge its inherent limitations. One of the most prominent challenges is
the high cost associated with proof generation in zkML. The process demands
considerable computational resources, resulting in extended generation times
and substantial memory consumption. Consequently, zkML is most suitable for
relatively small models, as its inefficiency becomes apparent when handling
larger and more complex models. zkML would take more than 1000 times the
memory and computation consumption for the ZK proof generation (Labs, [n.
d.]). As a result, zkML may prove impractical for extensive AI applications
that require the processing of substantial datasets and intricate model
parameters.

Faced with the limitations of zkML, we turn to investigate using fraud proof
to prove the correctness of the ML results on chain instead of using the zero-
knowledge proof (ZKP, also known as validity proof). Fraud proofs are commonly
used in blockchain systems, including rollup systems, which are part of a
broader category known as optimistic systems. Notable examples of rollup
systems employing fraud proofs include Optimism (Optimism, [n. d.]) and
Arbitrum (Kalodner et al., 2018). In the system using fraud proof, we
optimistically assume that every proposed result is valid by default. In the
case of mitigating potentially invalid results, the system using fraud proof
will introduce a challenge period during which participants may challenge the
submitter. The fraud proof is generated via interactive pinpoint protocol
proving that the provided result is incorrect. The arbitration process is
designed to validate a fraud proof by executing only a few minor computation
steps, making the on-chain cost extremely low.

Building upon the foundation of optimistic system design, we introduce opML:
Optimistic Machine Learning on the blockchain111opML is open sourced.
https://github.com/hyperoracle/opml. Diverging from the approach of zkML,
which relies on zero-knowledge proofs, opML adopts a fraud-proof system to
guarantee the correctness of ML results. In the opML framework, submitters can
execute ML computations within a native environment and subsequently provide
the results directly on the blockchain. This approach maintains an optimistic
assumption that each proposed result is inherently valid. During the challenge
process, validators will check the correctness of these submitted result. If
the result is invalid, he will start the dispute game (bisection protocol)
with the submitter and tries to disprove the claim by pinpointing one concrete
erroneous step. Finally, arbitration about a single step will be conducted on
smart contract.

| opML | zkML
---|---|---
Model size | any size | small/limited
Proof | fraud proof | validity proof
Requirement | Any PC with CPU/GPU | Large memory for proof generation
Finality | Delay for challenge period | No delays
Service cost | low | extremely high
Security | crypto-economic security (AnyTrust) | cryptographic security
Table 1. Comparison between opML and zkML

#### opML v.s. zkML.

The comparison of opML and zkML is shown in Table 1. Generally speaking, opML
and zkML are different from the following perspectives:

  * •

Proof System: opML adopts the fraud proof, while zkML adopts the zero-
knowledge proof to prove the correctness of the ML results.

  * •

Performance: The performance is considered in the two perspectives: The proof
generation time, and the memory consumption.

    * –

The proof time of zkML is long, and significantly grows with increasing model
parameters222An example of a significantly long proving time of zkML is
running a nanoGPT model inference with 1 million parameters inside zkML
framework takes around 80 minutes of proving time (ezkl, [n. d.]).. But for
opML, computing the fraud proof can be conducted in the nearly-native
environment in a short time.

    * –

The memory consumption of zkML is extremely large compared to the native
execution. But opML can be conducted in the nearly-native environment with the
nearly-native memory consumption. Specifically, opML can handle a 7B-LLaMA
model (the model size is around 26GB) within 32GB memory. But the memory
consumption for the circuit in zkML for 7B-LLaMA model may be on the order of
TB or even PB levels, making it totally impractical.

Limited by the long proof generation time and large memory consumption, zkML
can only handle some small AI models with an extremely high service cost.
While opML can provide proof in the native execution, and thus it is more
cost-efficient and practical for some large models.

  * •

Security: zkML can make ML onchain with the highest level of security
possible, but is limited by proof generation time and other constraints. opML
can make ML onchain with flexibility and performance, but may not be as secure
as zkML because it is secured by economics, not by mathematics and
cryptography like ZK. The security model of opML is similar to the current
optimistic rollup systems. With the fraud proof system, opML can provide an
any-trust guarantee: any one honest validator can force opML to behave
correctly. Under the AnyTrust assumption, opML can maintain correctness and
liveness.

  * •

Finality: We can define the finalized point of zkML and opML as follows:

    * –

zkML: zk proof of ML inference is generated (and verified)

    * –

opML: challenge period of ML inference has ended

Though opML needs to wait for the challenge period, zkML also needs to wait
for the generation of ZKP. When the proving time of zkML is longer than the
challenge period of opML (it happens when ML model is large), opML can reach
finality even faster than zkML.

The contribution of this work is summarized as follows:

  * •

We propose opML: optimistic machine learning on blockchain. Compared to zkML,
opML can provide ML service with low cost and high efficiency.

  * •

We propose a novel multi-phase fraud proof protocol with semi-native execution
and lazy loading design, which can greatly improve the efficiency of the fraud
proof protocol and overcome the memory limitation in the fraud proof VM.

The organization of this paper goes as follows. In Section 2, we introduce the
architecture and the workflow of opML. In Section 3, we introduce the fraud
proof virtual machine (FPVM) in opML. In Section 4, we design a high-efficient
ML engine for both native execution and fruad proof. In Section 5, we
demonstrate the dispute game in opML. In Section 6, we introduce the multi-
phase dispute game in opML, which can help to overcome the limitations of the
current fraud proof system. In Section 7, we show that opML can achieve safety
and liveness under the AnyTrust assumption. In Section 8, we investigate the
incentive mechanism in opML to resolve the verifier dilemma. In Section 9, we
discuss the potential optimization and applications on opML. Section 10
reviews the related work. Section 11 concludes this paper with a final remark.

##  2\. Overview

Figure 1. OPML Overview

###  2.1. Design Principles

The design principles of opML are listed as follows:

  * •

Deterministic ML Execution: The native ML execution may result in different
execution results due to the randomness and computation on floating-point
numbers. opML adopts the fixed-point arithmetic and the software-based
floating point libraries to guarantee the consistency and determinism of the
ML execution. Therefore, we can use a deterministic state transition function
to represent the ML execution process.

  * •

Separate Execution from Proving: opML takes the same source code and compiles
it twice. One is compiled for native execution, which is optimized for speed.
And we allow using multi-thread CPU and GPU to speed up the native execution.
The same code is also compiled to fraud proof VM instruction333In the current
implementation, we adopts MIPS as the fraud proof VM. We are also exploring
other efficient fraud proof VM such as WASM, RISC-V, etc., which will be used
in the fraud proof protocol. This dual-target approach assures that execution
is fast, while proving is based on the machine-independent code.

  * •

Optimistic Machine Learning with Interactive Fraud Proofs: opML adopts the
interactive fraud proof, which will bisect a dispute down to a single
instruction, and resolve the base-case using the on-chain fraud proof VM.

  * •

Optimizing ML fraud proof with Multi-Phase Protocol: The existing fraud proof
systems that are widely adopted in the optimistic rollup systems need to
cross-compile the whole computation into fraud proof VM instructions, which
will result in inefficient execution and huge memory consumption. opML
proposes a novel multi-phase protocol, which allows semi-native execution and
lazy loading, which greatly speeds up the fraud proof process.

###  2.2. Architecture

opML relies on a fraud-proof mechanism to ensure the accuracy of machine
learning results on the chain. The fraud-proof in opML comprises three
integral components:

  * •

Fraud Proof Virtual Machine (FPVM). Given a stateless program and its inputs,
FPVM can trace any instruction step and prove it on layer 1 blockchain (L1).

  * •

Machine Learning Engine. The highly efficient machine learning engine is
designed to cater to both native execution and fraud-proof scenarios. This
engine not only ensures a swift and accurate execution of machine learning
tasks but also guarantees the consistency and determinism of the results.

  * •

Interactive Dispute Game. The dispute game will bisect a dispute down to a
single instruction, and resolve the disputed one-step instruction using on-
chain FPVM.

###  2.3. Workflow

In opML, the requester first initiates a task, and upon the server’s
completion, results are committed on-chain. Verifiers (Challengers) check
these results, and if a dispute arises, the bisection protocol is initiated.
This dispute resolution process aims to pinpoint and address any identified
erroneous steps. Finally, smart contract arbitration concludes the workflow by
decisively settling any disputes related to a single computation step. The
opML workflow unfolds as follows:

  1. (1)

The requester first initiates an ML service task.

  2. (2)

The submitter then performs the ML service task and commits the result on
chain.

  3. (3)

The verifier (challenger) will validate the results. If a verifier deems the
results incorrect, he will initiate the dispute game (bisection protocol) with
the server. The goal is to disprove the claim by pinpointing a specific
erroneous step.

  4. (4)

The smart contract conducts arbitration concerning the disputed step,
providing a conclusive resolution to the dispute.

  5. (5)

Finally, after a defined “challenge period”, the results will be confirmed.

Both the server (submitter) and the verifier (challenger) need to stake in the
system, and providing a incorrect result will lead to loss of the stake. Thus,
if all parties follow their incentives, only the valid result will be
committed. Therefore, in the common case, the server (submitter) will always
provide correct results, and the verifiers (challengers) will always find the
results valid, and the dispute game would not happen. After a defined
“challenge period”, the results will be confirmed.

##  3\. Fraud Proof Virtual Machine

Figure 2. Fraud Proof VM Memory Layout

We build a fraud proof virtual machine (FPVM) for offchain execution and
onchain arbitration. We guarantee the equivalence of the offchain VM and the
onchain VM implemented on a smart contract. Operationally, the FPVM is a state
transition function. This state transition is referred to as a Step, that
executes a single instruction in the FPVM. Given an input state $S_{pre}$
steps on a single instruction encoded in the state, the FPVM, as a state
transition function (STF) $VM(\cdot)$, will produce a new state $S_{post}$,
i.e., $VM(S_{pre})\rightarrow S_{post}$. Thus, the trace of a program executed
by the FPVM is an ordered set of VM states, which highlights the effects of
running a machine learning program on the FPVM. The execution trace $T$ is a
sequence $(S_{0},S_{1},S_{2},\cdots,S_{n})$, where each $S_{i}$ is a VM state,
and $S_{i}=VM(S_{i-1}),\forall i\in[1,n]$. Each execution trace has an unique
initial state $S_{0}$, which is determined by the ML model and input.

The VM state is managed with a Merkle tree (Szydlo, 2004), only the Merkle
root will be uploaded to the onchain smart contract. The Merkle root
represents for the VM state. As shown in Figure 2, the memory in FPVM is
divided into several areas, namely, “program code”, “input”, “output”, “oracle
key”, “oracle value” and “model”. The machine learning program is placed in
the “program” field; and the input for the ML model is placed in the “input”
field; the ML model is placed in the “model” field; the results of opML will
be placed in the “output” field at the end of the execution of the ML program.
There is a key-value oracle in fraud proof system, which is used by FPVM to
read external data during its state transition. The oracle key and value will
be placed in the “oracle key” field and “oracle value” field correspondingly.
Besides, the memory in FPVM is represented as a merkle tree. The tree has a
fixed-depth of 27 levels, with leaf values of 32 bytes each. This spans the
full 32-bit address space, where each leaf contains the memory at that part of
the tree. The merkle root of the tree reflects the effects of memory writes in
FPVM. With this merkle tree structure, each field in the FPVM memory can be
represented as a merkle subtree root.

##  4\. Machine Learning Engine

In opML, a highly efficient machine learning engine is designed to cater to
both native execution and fraud-proof scenarios. This engine not only ensures
a swift and accurate execution of machine learning tasks but also guarantees
the consistency and determinism of the results. This becomes particularly
crucial in the dispute resolution process, where the machine learning engine
can statelessly provide a valid output and verify the dispute, reinforcing the
reliability of the overall system.

###  4.1. Separate Execution from Proving

Following the design principle “Separate Execution from Proving”, we design a
high efficient machine learning engine for opML. We provide two kinds of
implementation of the machein learning engine. One is compiled for native
execution, which is optimized for high speed. And we use multi-thread CPU and
GPU to speed up the native execution. Another is compiled to a fraud proof
program for FPVM. This dual-target approach assures that the execution is
fast, while proving is based on the machine-independent code.

Consider the matrix multiplication in the machine learning engine, illustrated
in Figure 3. Native execution employs GPU calculation (CUDA calculator
(Sanders and Kandrot, 2010)) for acceleration. As shown in Figure 4, for the
proving phase, the machine learning engine is compiled into machine-
independent FPVM instructions, as CUDA calculator compatibility is lacking.
The two implementations ensure consistent execution results.

⬇

1// a,b,c are GPU device pointers to matrix

2void gpu_matrix_mult(int *a,int *b, int *c, int m, int n, int k) {

3 int row = blockIdx.y * blockDim.y + threadIdx.y;

4 int col = blockIdx.x * blockDim.x + threadIdx.x;

5 int sum = 0;

6 for(int i = 0; i < n; i++) {

7 sum += a[row * n + i] * b[i * k + col];

8 }

9 c[row * k + col] = sum;

10}

Figure 3. Matrix multiplication in ML engine using GPU for native execution

⬇

1void fpvm_matrix_mult(int *a, int *b, int *c, int m, int n, int k) {

2 for (int i = 0; i < m; ++i) {

3 for (int j = 0; j < k; ++j) {

4 int smu = 0;

5 for (int h = 0; h < n; ++h) {

6 smu += a[i * n + h] * b[h * k + j];

7 }

8 c[i * k + j] = smu;

9 }

10 }

11}

Figure 4. Matrix multiplication in ML engine to machine-independent code for
FPVM

In the current implementation, to ensure the efficiency of AI model
computation in the FPVM, we have implemented a lightweight machine learning
engine specifically designed for this purpose instead of relying on popular ML
frameworks like Tensorflow (Abadi et al., 2016) or PyTorch (Paszke et al.,
2019). Additionally, a script that can convert Tensorflow and PyTorch models
to this lightweight library is provided. With the provided script, we can
train a model on popular ML frameworks like Tensorflow or PyTorch, and then
convert it to opML model.

###  4.2. Consistency and Determinism

The inconsistency in ML results can be attributed to two factors: randomness
and variability in floating-point computations. To address randomness, fixing
the random seed in the random number generator is a common practice since
computer-generated randomness is essentially pseudo-randomness. Regarding the
inconsistency in floating-point computations, during the native execution of
DNN computations, especially across diverse hardware platforms, differences in
execution results may arise due to the nature of floating-point numbers. For
instance, parallel computations involving floating-point numbers, such as
$(a+b)+c$ versus $a+(b+c)$, often yield non-identical outcomes due to rounding
errors (Goldberg, 1991). Additionally, factors such as programming language,
compiler version, and operating system can influence the computed results of
floating-point numbers, leading to further inconsistency in ML results
(Goldberg, 1991).

To tackle these challenges and guarantee the consistency of opML, we employ
two key approaches:

  * •

Fixed-point arithmetic, also known as quantization technology (Yang et al.,
2019), is adopted. This technique enables us to represent and perform
computations using fixed precision rather than floating-point numbers. By
doing so, we mitigate the effects of floating-point rounding errors, leading
to more reliable and consistent results. It’s important to note that using
fixed-point arithmetic may result in a minimal loss of accuracy in DNN models.
This tradeoff between execution performance and model accuracy is a key
consideration when adopting such precision techniques.

  * •

We leverage software-based floating-point libraries (softfloat) (Hauser, [n.
d.]) that are designed to function consistently across different platforms.
These libraries ensure cross-platform consistency and determinism of the ML
results, regardless of the underlying hardware or software configurations.

By combining fixed-point arithmetic (Yang et al., 2019) and software-based
floating-point libraries (softfloat) (Hauser, [n. d.]), we establish a robust
foundation for achieving consistent and reliable ML results within the opML
framework.

##  5\. Interactive Dispute Game

In the interactive dispute game, two or more parties (with at least one honest
party) are assumed to execute the same program. Then, the parties can
challenge each other with a pinpoint style to locate the disputed step. The
step is sent to the smart contract on blockchain for arbitration.

###  5.1. Locating the Disputed Point

Figure 5. Dispute Game in opML

At the beginning, the submitter and the verifier agree on the initial state
$S_{0}$, but disagree with the final state, i.e., $S_{n}\neq$
$S^{\prime}_{n}$. The objective of this protocol, as illustrated in Figure 5,
is to identify a specific VMk in a sequence of VM instructions executed within
the context of a VM state $S_{0}$ (with $n$ such instructions in total),
wherein $S_{k-1}=$ $S^{\prime}_{k-1}$ but $S_{k}\neq$ $S^{\prime}_{k}$.

The dispute game happens in a sequence of rounds. At the beginning of each
round, the submitter and the verifier agree on a start VM state $S_{i}$ and
they disagree on the end state $S_{i+j}$, for some $j>1$. Initially, $i=0$ and
$j=n$. Then the challenger is now required to claim what state $S_{m}$ is at
the midpoint VM state, where $m=i+\lfloor\frac{j}{2}\rfloor$. Next, the
submitter must say whether he agrees or disagrees with the submitter’s claimed
midpoint state $S_{m}$. Now there are two cases:

  * •

If the submitter agrees with the challenger’s midpoint state, the protocol has
identified a smaller dispute: the challenger and the submitter agree on
$S_{m}$, where $m=i+\lfloor\frac{j}{2}\rfloor$, but disagree with $S_{i+j}$.
In the next round, we will let $i\leftarrow i+\lfloor\frac{j}{2}\rfloor$ and
$j\leftarrow N-\lfloor\frac{j}{2}\rfloor$.

  * •

If the submitter disagrees with the challenger’s midpoint state, the protocol
also has identified a smaller dispute: the challenger and the submitter agree
on $S_{i}$ but disagree with $S_{m}$, where $m=i+\lfloor\frac{j}{2}\rfloor$.
In the next round, we will let $i\leftarrow i$ and
$j\leftarrow\lfloor\frac{j}{2}\rfloor$.

In both cases, the length of the dispute will be cut in half. The same
procedure is repeated, cutting repeatly in half, until $j=1$.

The dispute game exhibits remarkable efficiency. In terms of time complexity,
both the verifier and the submitter require only $\lceil\log_{2}n\rceil$
rounds of challenge-response to converge on the same value of $k$, wherein
$S_{k-1}=$ $S^{\prime}_{k-1}$ but $S_{k}\neq$ $S^{\prime}_{k}$. This
synchronization between the submitter and the verifier is facilitated through
a challenge-response mechanism featuring a timeout penalty (Kalodner et al.,
2018; Teutsch and Reitwießner, 2019). The state $S_{k-1}$, along with
auxiliary data, will be subsequently forwarded to a smart contract for
arbitration.

###  5.2. Onchain Arbitration

For on-chain arbitration, the submitter and the verifier will send (VMIk,
Sk-1, Sk) to contract for arbitration. The onchain VM will take Sk-1 as the
input and conduct one-step execution to output the correct Sk. Since the
onchain VM only execute one instruction, the size of the data (witness) that
needs to access is only $O(1)$. For onchain arbitration, the witness consists
of a partial expansion of the Merkle tree representing the before state
$S_{k-1}$. The onchain VM uses the partially expended state tree to read the
next instruction, emulate the instruction execution, and then compute the
Merkle root hash of the resulting state. Note that the one-step onchain VM
execution always takes only $O(1)$ computation and memory consumption, so that
it is always possible to conduct the onchain arbitration using a feasible
amount of Ethereum gas.

If the challenger produces a valid one-step proof in the onchain arbitration,
he wins the challenge. Otherwise, the submitter wins the challenge.

##  6\. Multi-Phase Dispute Game

###  6.1. Limitations of One-Phase Dispute Game

With the design principle of “Separate Execution from Proving”, we can achieve
a high performance in the common optimistic case, where the submitter will
always provide a correct result. However, to prevent the malicious behavior in
the pessimistic case, we still need to generate a fraud proof for opML. In the
fraud proof protocol of opML, we cross-compile the ML computation into fraud
proof VM instructions and then start the dispute game to find the dispute
step. However, cross-compiling the whole ML computation into fraud proof VM
instructions has significant limitations:

  * •

Low Execution Efficiency: The one-phase dispute game has a critical drawback:
for proving, all computations must be executed within the FPVM, preventing us
from leveraging the full potential of GPU/TPU acceleration or parallel
processing. Consequently, this restriction severely hampers the efficiency of
providing fraud proof for large model inference, which also aligns with the
current limitation of the referred delegation of the computation (RDoC)
protocol (Canetti et al., 2011).

  * •

Limited Memory in Fraud Proof VM: The fraud proof VM has limited memory, for
example, the MIPS VM can only support at most 4 GB memory. Due to the limited
memory size, we can not load a large model into the fraud proof VM directly.
For example, the size of a 7B-llama model in float64 is around 26GB, which can
not be stored in the memory of the MIPS FPVM.

###  6.2. Overview of Multi-Phase Protocol

To address the constraints imposed by the one-phase protocol and ensure that
opML can generate fraud proof in a performance level comparable to the native
environment, we propose a multi-phase protocol. The multi-phase dispute game
has the following properties that can well resolve the limitations above:

  * •

Semi-Native Execution: With the multi-phase design, we only need to conduct
the computation in the VM only in the final phase, resembling the single-phase
protocol. For other phases, we have the flexibility to perform computations
that lead to state transitions in the native environment, leveraging the
capabilities of parallel processing in CPU, GPU, or even TPU. By reducing the
reliance on the VM, we significantly minimize overhead, resulting in a
remarkable enhancement in the execution performance of opML, almost akin to
that of the native environment.

  * •

Lazy Loading Design: To optimize the memory usage and performance of the fraud
proof VM, we implement a lazy loading technique. This means that we do not
load all the data into the VM memory at once, but only the keys that identify
each data item. When the VM needs to access a specific data item, it uses the
key to fetch it from the external source and load it into the memory. Once the
data item is no longer needed, it is swapped out of the memory to free up
space for other data items. This way, we can handle large amounts of data
without exceeding the memory capacity or compromising the efficiency of the
VM.

Figure 6. Multi-Phase Dispute Game

Figure 6 demonstrates a verification game consisting of two phases (k = 2). In
Phase-2, the process resembles that of a single-phase verification game, where
each state transition corresponds to a single VM micro-instruction that
changes the VM state. In Phase-1, the state transition corresponds to a ”Large
Instruction” encompassing multiple micro-instructions that change the
computation context. The submitter and challenger will first start the dispute
game on Phase-1 using a bisection protocol to locate the dispute step on a
”large instruction”. This step will be send to the next phase, Phase-2.
Phase-2 works like the single-phase dispute game. The bisection protocol in
Phase-2 will help to locate the dispute step on a VM micro-instruction. This
step will be sent to the arbitration contract on the blockchain.

###  6.3. State Transition to Next Phase

To ensure the integrity and security of the transition to the next phase, we
rely on the Merkle tree. As shown in Figure 7, this operation involves
extracting a Merkle sub-tree reconstruction, thus guaranteeing the seamless
continuation of the dispute game process. Take the state transition in two-
phase protocol as an example. On the phase-1, assume that the submitter and
the challenger have already located the dispute step on a ”large instruction”,
i.e., $S^{k}_{i-1}=S^{\prime k}_{i-1}\land S^{k}_{i}\neq S^{\prime k}_{i}$.
Then we will build a Merkle tree on the state data $S^{k}_{i-1}$ to get the
Merkle tree root $root(S^{k}_{i-1})$. We will construct the initial state in
the next phase $S^{k+1}_{0}$ using the Merkle root. Specially, we need an
empty VM image first, initialize it with the running program in the “program
code” memory field in the FPVM, and then we will set the “input” memory field
in FPVM as $root(S^{k}_{i-1})$, which will serves as a key for lazy loading.
After completing initialization and filling in data, we can get the initial
state in the next phase $S^{k}_{i-1}\Rightarrow S^{k+1}_{0}$. Due to the
complexity of construction $S^{k+1}_{0}$ and the gas limitation of Ethereum,
we build a zero-knowledge circuit to correctness of the construction of
$S^{k+1}_{0}$ and verify the zero-knowledge proof on chain. Similarly, at the
end of the execution in phase 2, we need to check the consistency of the
submitter’s state $S^{k}_{i}\Leftrightarrow S^{k+1}_{n}$ and the challenger’s
state $S^{\prime k}_{i}\Leftrightarrow S^{\prime k+1}_{n}$. Specially, the
state $S^{k}_{i}$ is stored as the “output” field of the FPVM memory, and we
can prove it by providing a Merkle proof.

Figure 7. State Transition in Multi-Phase Dispute Game

###  6.4. DNN Computation in Multi-Phase opML

In this demonstration, we present a DNN computation in a two-phase opML
approach:

  * •

The computation process of Machine Learning, specifically Deep Neural Networks
(DNN), can be represented as a computation graph denoted as $G$. This graph
consists of various computation nodes, capable of storing intermediate
computation results.

  * •

DNN model inference is essentially a computation process on the aforementioned
computation graph. The entire graph can be considered as the inference state
(computation context in Phase-1). As each node is computed, the results are
stored within that node, thereby advancing the computation graph to its next
state.

  * •

Therefore, we can first conduct the dispute game on the computation graph (at
Phase-1). On the phase-1 dispute game, the computation on nodes of the graph
can be conducted in native environment using multi-thread CPU or GPU. The
bisection protocol will help to locate the dispute node, and the computation
of this node will be sent to the next phase (Phase-2) bisection protocol.

  * •

In Phase-2 bisection, we transform the computation of a single node into
Virtual Machine (VM) instructions, similar to what is done in the single-phase
protocol.

It is worth noting that we anticipate introducing a multi-phase opML approach
(comprising more than two phases) when the computation on a single node within
the computation graph remains computationally complex. This extension will
further enhance the overall efficiency and effectiveness of the fraud poof
protocol.

###  6.5. Performance Improvement

In this section, we present a brief discussion and analysis of our proposed
multi-phase fraud proof framework.

Suppose there are $n$ nodes in the DNN computation graph, and each node needs
to take $m$ VM micro-instructions to complete the calculation in VM. Assuming
that the speedup ratio on the calculation on each node using GPU or parallel
computing is $\alpha$. This ratio represents the acceleration achieved through
GPU or parallel computing and can reach significant values, often ranging from
tens to even hundreds of times faster than the single-thread VM execution.
Based on these considerations, we draw the following conclusions:

  * •

Performance: Two-phase opML outperforms single-phase opML, achieving a
computation speedup of $\alpha$ times. The utilization of multi-phase
verification enables us to take advantage of the accelerated computation
capabilities offered by GPU or parallel processing, leading to substantial
gains in overall performance.

  * •

Space Complexity: Two-phase opML reduces space complexity of the Merkle tree.
When comparing the space complexity of the Merkle trees, we find that in two-
phase opML, the size is $O(m+n)$, whereas in single-phase opML, the space
complexity is significantly larger at $O(mn)$. The reduction in space
complexity of the Merkle tree further highlights the efficiency and
scalability of the multi-phase design.

In summary, the multi-phase fraud proof framework presents a remarkable
performance improvement, ensuring more efficient and expedited computations,
particularly when leveraging the speedup capabilities of GPU or parallel
processing. Additionally, the reduced Merkle tree size adds to the system’s
effectiveness and scalability, making multi-phase opML a compelling choice for
various applications.

##  7\. Security Analysis

In this section, we provide the security analysis of our system. For
simplicity, we use the AnyTrust assumption from Arbitrum (Kalodner et al.,
2018).

The AnyTrust assumption. AnyTrust, rather than “majority trust”, assumes that,
for every claim, there is at least one honest node. Namely, either the
submitter is honest, or at least one verifier is honest and will challenge
within the pre-defined period. Suppose there are $m$ verifiers, $m-1$ of whom
collude to stay silent about a submitter’s wrong claim. Under AnyTrust, the
only honest verifier will still succeed in disproving the wrong claim in front
of the smart contract, and the wrong claim is thus rejected. Like the previous
work (Kalodner et al., 2018; Teutsch and Reitwießner, 2019; Zheng et al.,
2021), we also assume data availability and anti-censorship (i.e., every
verifier can always interact with the contract). They are solved by orthogonal
countermeasures (Benet, 2014).

Safety and Liveness. We can show that opML can maintain safety and liveness
under the AnyTrust assumption.

  * •

Safety: _The safety of opML under the AnyTrust assumption means that any one
honest validator can force opML to behave correctly._ Suppose there exists
only one honest node, and all other nodes are malicious. When a malicious node
provides a incorrect result on chain, the honest node will always check the
result and find it incorrect. At that time, the honest node can challenge the
malicious one. The dispute game can force the honest node and the malicious
node to pinpoint one concrete erroneous step. This step will be sent to the
arbitration contract on the blockchain, and will find out the result provided
by the malicious node is incorrect. Finally, the malicious node will be
penalized and the honest node can get a corresponding reward.

  * •

Liveness: _The Liveness of opML under the AnyTrust assumption means that any
proposed result will be either accepted or rejected by the contract within a
maximum period time._ Firstly, the instruction set and the execution traces of
the opML program is finite, and can be done within $T_{e}$. Therefore, in the
dispute game, the number of interactions is also finite. Therefore, the
dispute game will end within a maximum period time $T_{d}$. If the provided
result is correct, even in the worst case where $m-1$ malicious verifiers
start the dispute game, the result will be accepted within $T_{e}+(m-1)\times
T_{d}$. If the result is incorrect provided by a malicious verifier, the
honest verifier will challenge it, even in the worst case where $m-2$
verifiers will start the dispute game to delay the actions of honest verifier,
the result will be rejected within $T_{e}+(m-1)\times T_{d}$.

Compared with “AnyTrust” and “Majority Trust” In the following, we will
conduct a brief security discussion between the system with “AnyTrust” and
“Majority Trust”. As discussed above, opML is an “AnyTrust” system where any
one honest validator can force opML to behave correctly. In contrast, systems
like Layer 1 blockchains (e.g., Bitcoin (Nakamoto, 2008) and Ethereum (Wood et
al., 2014)) and Oracles (e.g., Chainlink (Breidenbach et al., 2021)) fall into
the category of ”Majority Trust” systems. These ”Majority Trust” systems
become susceptible to attacks when the number of malicious validators
surpasses the threshold of $\lceil fm\rceil$, where $m$ represents the total
number of validators, and $f$ denotes the Byzantine fault tolerance ratio. The
specific value of $f$ varies with different consensus algorithms; for
instance, it is 0.5 for Proof of Work (PoW) blockchains like Bitcoin. Consider
a system with $m$ validators, and let $p$ denote the probability of a
validator being malicious. The probability that a “Majority Trust” system is
secure is

$P_{\text{majority-trust}}=\sum_{i=0}^{\lceil fm\rceil}{m\choose
i}p^{i}(1-p)^{m-i}=(m-k){m\choose k}\int_{0}^{1-p}t^{n-k-1}(1-t)^{k}dt.$

The probability that a system with “AnyTrust” is secure is

$P_{\text{any-trust}}=\sum_{i=0}^{m-1}{m\choose i}p^{i}(1-p)^{m-i}=1-p^{m}.$

It is evident that the ”AnyTrust” system exhibits superior security compared
to the ”Majority Trust” system, specifically denoted as $P_{\text{any-
trust}}>P_{\text{majority-trust}}$. Additionally, as illustrated in Figure
LABEL:fig:secure-probability, the ”AnyTrust” system attains a high level of
security even with a limited number of nodes. For instance, even with a
relatively high probability of 0.5 for a validator to be malicious (indicating
that half of the validators can potentially be malicious), the ”AnyTrust”
system maintains a 99.9% security probability with only 10 validators. In
contrast, the ”Majority Trust” system struggles to provide such a robust
security guarantee under similar conditions.

##  8\. Incentive Mechanism

To achieve the safety and liveness of opML, we need to guarantee that at least
one honest validator will check the results. Therefore, it is important to
design an incentive-compatible mechanism where the rational validators will
always check the results and the rational submitter will never cheat.

###  8.1. Verifier Dilemma

Like the optimistic rollup systems, opML can also suffer from the Verifier’s
Dilemma (Luu et al., 2015). Here we consider a simple verification game
$\mathbb{G}$. Assume that all the validators (including the submitter) need to
stake $S$ on chain, and the computation cost for opML execution is $C$. If the
submitter provides an incorrect and nobody challenges him, he can benefit $B$
from cheating, but all the validator will suffer from a loss $L$. If the
submitter provides an incorrect and it is successfully challenged by one
validator, then the submitter will lose all his stake, and the validator can
get the corresponding reward $R$. The action for the submitter is to cheat or
not. And the action for the validator is to check or not. The payoff matrix is
shown in Table 2, and the Nash equilibrium strategy (Roughgarden, 2010) in
this verification game is shown in the following theorem.

validator/submitter | cheat | No cheat
---|---|---
validate | $(R-C,-S)$ | $(-C,-C)$
No validate | $(-L,B)$ | $(0,-C)$
Table 2. Payoff matrix in the verification game

###### Theorem 8.1.

The mixed strategy Nash equilibrium of this verification game is that the
submitter will randomly cheat with probability $p_{c}=\frac{C}{R+L}$, and the
validator will randomly check with probability $p_{v}=\frac{B+C}{B+S}$.

###### Proof.

According to the payoff matrix in Table 2, we can find that there is no
dominant strategy for both validator and submitter. Therefore, there is no
pure strategy Nash equilibrium in the verification game. We now turn to
consider the mixed strategy one. Assume that the submitter will randomly cheat
with probability $p_{c}$, and the validator will randomly check with
probability $p_{v}$. Then for the mixed strategy Nash equilibrium, we have

$\begin{cases}p_{c}(R-C)+(1-p_{c})(-C)=-p_{c}L&\\\ p_{v}(-S)+(1-p_{v})B=-C&\\\
\end{cases}$

By solving the equations above, we have that $p_{c}=\frac{C}{R+L}$ and
$p_{v}=\frac{B+C}{B+S}$. The proof is thus completed. ∎

There is no pure strategy Nash equilibrium in this verification game. This
means that a rational submitter can randomly cheat, while the rational
validator will also randomly check. This is not what we want. What’s worse is
if the submitter cheat randomly, with probability of cheating less than
$c/(R+L)$, then a rational validator will never check, so the submitter will
never get caught cheating. This is indeed the verifier dilemma, where there’s
no point in checking somebody’s work if you know they’re not going to cheat;
but if you’re not going to check then they have an incentive to cheat.

###  8.2. Attention Challenge

To solve the verifier dillema, we introduce the Attention Challenge
(Mamageishvili and Felten, 2023) into opML.

In opML, we denote $x$ as the user’s input to opML (corresponding to the
initial state $S_{0}$), $f(\cdot)$ as the deterministic computation function,
and $f(x)$ is the opML result (corresponding to the final state $S_{n}$).
Instead of revealing $f(x)$ on chain, the submitter will first reveal
$H(A_{s},f(x))$ on chain, where $A_{s}$ is the address of the submitter and
$H(\cdot)$ is the hash function.

There exists a window of time, where the validator whose hash value
$H(A_{v},f(x))<T$ should respond on chain, where $A_{v}$ is the address of the
validator, and $T$ is a chosen threshold. After that, the submitter can post
the opML result $f(x)$ on chain, which can be challenged as normal if any
validator disagrees with it. The submitter can accuse any validator who
responded incorrectly or did not respond. This accusation will be checked on
chain and the validator who responded incorrectly or did not respond will be
penalized by $G$. Note that the accusation will be valid only after the
submitter’s claim $f(x)$ is eventually accepted as correct. If any validator
is penalized, the submitter will get half of the penalty $\frac{1}{2}G$, and
the other half will be burned.

We denote $p_{t}=\Pr(H(A_{v},f(x))<T)$ as the probability that the validator
needs to respond, which can be adjusted by chosen a proper $T$. Then if the
validator checks the results, then his utility is $p_{c}R-C$, where $p_{c}$ is
the probability that the submitter checks and $C$ is the computation cost. If
the validator does not check the results, he may be punished by no-responding
when he needs to respond. In this case, the validator’s utility is
$-p_{c}\cdot L-p_{t}\cdot G$. In the following theorem, we can show that when
$p_{t}\cdot G>C$, the rational validator will always check, and the rational
submitter will never cheat.

###### Theorem 8.2.

When $p_{t}\cdot G>C$, the only Nash equilibrium of the verification game with
attention challenge mechanism is that validator will always check and the
submitter will never cheat.

###### Proof.

The utility for the validator to check the results is
$U(\text{check})=p_{c}R-C$, and the utility for the validator to be lazy and
not check the results is $U(\text{lazy})=-p_{c}\cdot L-p_{t}\cdot G$. When
$p_{t}\cdot G>C$, we have that

$\displaystyle
U(\text{check})-U(\text{lazy})=p_{c}(R+C)+p_{t}G-C>0,\quad\forall
p_{c}\in[0,1]$

Therefore, no matter what probability the submitter cheats, the dominant
strategy for the validator is to always check. Given that the validator will
always check, the dominant strategy for the submitter is to always provide a
correct result. The proof is thus completed. ∎

Besides, the cost of introducing the attention challenge mechanism in opML is
very low. Assume that the interest rate of locking up the stake $G$ is $r$,
and the gas fee for the validator to respond is $t$. Then the extra cost for
the attention challenge mechanism in opML is $rG+tp_{t}$. The minimal cost is
$rG+tp_{t}\geq 2\sqrt{rtC}$ when $G=\sqrt{\frac{tC}{r}}$ and
$p_{t}=\sqrt{\frac{rC}{t}}$.

Specifically, as shown in (Horowitz, [n. d.]), A GPT-3.5 inference takes
approximately 1 second on an A100 with the raw computation cost $0.001 for
1,000 tokens (compared to OpenAI’s pricing of $0.002/1000 tokens (OpenAI, [n.
d.])), i.e., $C=\$0.001$. With the interest rate $r=0.001$ and the gas fee
$t=\$1$, we can set the probability $p_{t}=0.1\%$ and the validator needs to
deposit $G=1$. The total cost for the attention challenge per opML request is
only $\$0.002$.

##  9\. Discussion

###  9.1. Optimization with Multi-Sect Dispute Points and Multi-Step zkVM

Furthermore, we can utilize multiple checkpoints and multi-step ZK verifier to
provide a zk-fraud proof to speed up the finality of opML. Assuming that there
are $N$ steps in the opML fraud proof VM. Then the number of interactions
would be $O(\log_{2}(N))$. One optimization approach to reduce to the
interactions is to use a multi-step on-chain arbitrator, instead of only one-
step on-chain arbitrator.

This can be done by using a zkVM, e.g., zkWASM (Gao et al., 2022), RICS Zero
(Zero, [n. d.]), Cairo VM (Goldberg et al., 2021), which can provide zk-proof
for the multi-step execution. Suppose the zkVM can generate a zk-proof for $m$
steps, then the number of interactions can be reduce from $O(\log_{2}(N))$ to
$O(\log_{2}(N/m))$. Besides, instead of using bisection protocol with only 1
checkpoint, we can provide multiple checkpoints (for example $k$ ($k>1$)
checkpoints) in the dispute game (verification game). At this time, we can
further reduce the interactions number from $O(\log_{2}(N/m))$ to
$O(\log_{k+1}(N/m))$. With these optimizations, we can greatly speed up the
finality of opML, and it is possible to achieve almost instant confirmation!

###  9.2. Training/Fine-tuning in opML

In opML, the use of a fraud-proof VM plays a pivotal role in ensuring the
correctness of ML results. This specialized VM has the capability to support
general computations, encompassing both inference and training tasks. The
training and fine-tuning process can also be conceptualized as a series of
state transitions when model parameters undergo deterministic updates.

Specifically, the inference phase of a DNN model involves straightforward
forward computation on the DNN computation graph, denoted as $G$. On the other
hand, the training process encompasses both forward computation and backward
update (backpropagation) on the same DNN computation graph $G$. While the
tasks differ in their objectives, the computation processes for forward
computation and backward update are similar, enabling a unified approach.

Through the integration of a multi-phase opML approach, we can extend support
to the training process efficiently as well. Here’s how it works: The dispute
game initiates during each iteration of the training process, to locate a
dispute within a specific iteration. Subsequently, the process advances to the
next phase, where the submitter and challenger engage in a dispute protocol on
the computation graph for both forward and backward processes. This allows for
the pinpointing of the dispute node, and the computation related to this node
is then forwarded to the subsequent phase, where the fraud-proof VM arbitrates
the issue.

opML’s extension to the training process has significant implications for
verifying ML model generation on the blockchain. By utilizing on-chain data to
train and update the ML model, opML ensures that the process is auditable and
transparent. Moreover, opML’s integration with the training process provides
an effective means to validate ML model updates, safeguarding against
potential backdoors (Hong et al., 2022) and ensuring the model’s integrity and
security.

###  9.3. Combining zkML and opML for Enhanced Privacy

In contrast to zkML, opML lacks intrinsic privacy features. However, a
compelling approach to augment privacy is by integrating the strengths of both
opML and zkML. One feasible strategy involves leveraging zkML for input
processing in the early layers of the model and subsequently employing opML
for the remaining layers. This amalgamation allows us to attain a nuanced form
of privacy—while not achieving full zk-level privacy, it is possible to ensure
a secure and non-reversible level of obfuscation. A crucial consideration in
this integration is determining the optimal number of layers to implement
through zkML, striking a delicate balance between computational efficiency and
heightened security.

##  10\. Related Work

AI computation on blockchain. The concept of AI computation on the blockchain
has garnered attention from both academia and industry. In the academic realm,
Das et al. have suggested the potential for ”Computationally Intensive Smart
Contracts” that encompass machine learning tasks (Das et al., 2018). Teutsch
et al. have explored the application of ”Autonomous machine learning” in the
context of TrueBit (Teutsch and Reitwießner, 2019), while Wüst et al. have
recognized the current limitations of smart contracts for machine learning
applications (Wüst et al., 2020). Within the industry, Microsoft has put forth
the concept of ”Decentralized and Collaborative AI” on blockchain platforms
(Harris and Waggoner, 2019). Several startups, including Cortex (Chen et al.,
2018), SingularityNET (Goertzel et al., 2017), Algorithmia (Kurtulmus and
Daniel, 2018) have also embraced this vision. However, the existing proposals
typically demonstrate only limited AI computations, achieving consensus on a
small number of nodes (Harris and Waggoner, 2019; Kurtulmus and Daniel, 2018;
Goertzel et al., 2017; Chen et al., 2018) or provide insufficient technical
details (Das et al., 2018; Teutsch and Reitwießner, 2019; Wüst et al., 2020).
These approaches lack scalability, rendering them impractical for real-world
deep neural network computations within Ethereum smart contracts.

Zero-Knowledge Machine Learning (zkML). In (Liu et al., 2021), the authors
provide zero-knowledge proof for an CNN model. In (Zhang et al., 2020), the
authors develop approaches to efficiently turn decision tree predictions and
accuracy into statements of zero knowledge proofs. In (Wang and Hoang, 2022),
the authors propose ezDPS: an efficient and zero-knowledge machine learning
inference pipeline. In (Xing et al., 2023), the authors propose a Zero-
Knowledge Proof-based Federated Learning (ZKP-FL) scheme on blockchain. In
(Weng et al., 2021), the authors propose Mystique, an efficient conversions
for zero-knowledge proofs with applications to machine learning, which can
improve the execution time by a factor by 7$\times$. In (Garg et al., 2023),
the authors propose zero-knowledge proof of training (zkPoT). However, due to
its huge cost, zkPoT only works for the logistic regression, and can not be
applied to some larger models.

Our work also resembles and references the work on Agatha (Zheng et al.,
2021). Agatha system is developed by Microsoft Research, which demonstrates
the practical contract verification for DNN computation on public Ethereum.
Agatha adopts the fraud proof to verify the computation results. Instead of
using fraud proof VM, Agatha adopts the graph-based pinpoint protocol (GPP)
which enables the pinpoint protocol on computational graphs. Since Agatha is
not open-sourced, we can not compare the performance of Agatha and opML
directly. Generally, our work distinguishes from Agatha (Zheng et al., 2021)
and the existing works in the following perspectives:

  * •

Agatha does not support Turing-complete computation. Agatha enables the native
DNN computation by the graph-based pinpoint protocol (GPP) instead of VM-based
pinpoint protocol. opML is designed to enable Turing-complete computation.

  * •

With our Turing-complete fraud-proof VM, opML can handle various aspects of
ML, including model inputs and outputs process, and can support various ML
models. This versatility allows opML to support ML training and fine-tuning, a
feature that Agatha lacks. For applications like LLaMA that require data
processing, multiple inferences, and intermediate result storage, going beyond
typical DNN model calculation, where a Turing-complete VM is essential. opML
excels in this context, outperforming Agatha.

  * •

With our Turing-complete fraud-proof VM, opML extends its capabilities beyond
deep neural network (DNN) computation. It accommodates various machine
learning algorithms, including K-nearest neighbors (KNN) (Soucy and Mineau,
2001), decision trees (Song and Ying, 2015), random forests (Cutler et al.,
2012), and more, enhancing its versatility and potential for a broader range
of AI applications.

##  11\. Conclusions

In conclusion, the optimistic approach to on-chain AI and machine learning
brings substantial advantages over existing methods, enabling the fusion of AI
power with the integrity and security of the blockchain. opML excels by
delivering a cost-effective and efficient ML service compared to zkML. This
positions opML as a key player in reshaping the future of decentralized,
secure, and transparent AI services. As the landscape of on-chain AI evolves,
opML stands as a pivotal solution, unlocking the full potential of these
technologies and ensuring a transformative journey toward accessible, secure,
and efficient on-chain machine learning.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  Tensorflow: a system for large-scale machine learning.. In _Osdi_ , Vol. 16. Savannah, GA, USA, 265–283.
  * Ben-Sasson et al. (2019) Eli Ben-Sasson, Iddo Bentov, Yinon Horesh, and Michael Riabzev. 2019.  Scalable zero knowledge with no trusted setup. In _Advances in Cryptology–CRYPTO 2019: 39th Annual International Cryptology Conference, Santa Barbara, CA, USA, August 18–22, 2019, Proceedings, Part III 39_. Springer, 701–732.
  * Ben-Sasson et al. (2014) Eli Ben-Sasson, Alessandro Chiesa, Eran Tromer, and Madars Virza. 2014.  Succinct $\\{$Non-Interactive$\\}$ zero knowledge for a von neumann architecture. In _23rd USENIX Security Symposium (USENIX Security 14)_. 781–796.
  * Benet (2014) Juan Benet. 2014.  Ipfs-content addressed, versioned, p2p file system.  _arXiv preprint arXiv:1407.3561_ (2014).
  * Bootle et al. (2016) Jonathan Bootle, Andrea Cerulli, Pyrros Chaidos, Jens Groth, and Christophe Petit. 2016.  Efficient zero-knowledge arguments for arithmetic circuits in the discrete log setting. In _Advances in Cryptology–EUROCRYPT 2016: 35th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Vienna, Austria, May 8-12, 2016, Proceedings, Part II 35_. Springer, 327–357.
  * Breidenbach et al. (2021) Lorenz Breidenbach, Christian Cachin, Benedict Chan, Alex Coventry, Steve Ellis, Ari Juels, Farinaz Koushanfar, Andrew Miller, Brendan Magauran, Daniel Moroz, et al. 2021\.  Chainlink 2.0: Next steps in the evolution of decentralized oracle networks.  _Chainlink Labs_ 1 (2021), 1–136.
  * Canetti et al. (2011) Ran Canetti, Ben Riva, and Guy N Rothblum. 2011.  Practical delegation of computation using multiple servers. In _Proceedings of the 18th ACM conference on Computer and communications security_. 445–454.
  * Chen et al. (2018) Z Chen, W Wang, X Yan, and J Tian. 2018.  Cortex-AI on blockchain: The decentralized AI autonomous system.  _Cortex White Paper_ (2018).
  * Cutler et al. (2012) Adele Cutler, D Richard Cutler, and John R Stevens. 2012.  Random forests.  _Ensemble machine learning: Methods and applications_ (2012), 157–175.
  * Das et al. (2018) Sourav Das, Vinay Joseph Ribeiro, and Abhijeet Anand. 2018.  Yoda: Enabling computationally intensive contracts on blockchains with byzantine and selfish nodes.  _arXiv preprint arXiv:1811.03265_ (2018).
  * Dinh and Thai (2018) Thang N Dinh and My T Thai. 2018.  AI and blockchain: A disruptive integration.  _Computer_ 51, 9 (2018), 48–53.
  * ezkl ([n. d.]) ezkl. [n. d.].  zkSnark nanoGPT.  https://hackmd.io/mGwARMgvSeq2nGvQWLL2Ww##Honey-I-SNARKED-the-GPT.
  * Gao et al. (2022) Sinka Gao, Guoqiang Li, Hongfei Fu, Heng Zhang, and Junyu Zhang. 2022.  ZAWA: A ZKSNARK WASM Emulator.  _Proc. ACM Program. Lang_ 1, 1 (2022).
  * Garg et al. (2023) Sanjam Garg, Aarushi Goel, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, Guru-Vamsi Policharla, and Mingyuan Wang. 2023.  Experimenting with Zero-Knowledge Proofs of Training.  _Cryptology ePrint Archive_ (2023).
  * Goertzel et al. (2017) Ben Goertzel, Simone Giacomelli, David Hanson, Cassio Pennachin, and Marco Argentieri. 2017.  SingularityNET: A decentralized, open market and inter-network for AIs.  _Thoughts, Theories Stud. Artif. Intell. Res._ (2017).
  * Goldberg (1991) David Goldberg. 1991.  What every computer scientist should know about floating-point arithmetic.  _ACM computing surveys (CSUR)_ 23, 1 (1991), 5–48.
  * Goldberg et al. (2021) Lior Goldberg, Shahar Papini, and Michael Riabzev. 2021.  Cairo–a Turing-complete STARK-friendly CPU architecture.  _Cryptology ePrint Archive_ (2021).
  * Goldreich and Oren (1994) Oded Goldreich and Yair Oren. 1994.  Definitions and properties of zero-knowledge proof systems.  _Journal of Cryptology_ 7, 1 (1994), 1–32.
  * Harris and Waggoner (2019) Justin D Harris and Bo Waggoner. 2019.  Decentralized and collaborative AI on blockchain. In _2019 IEEE international conference on blockchain (Blockchain)_. IEEE, 368–375.
  * Hauser ([n. d.]) John R. Hauser. [n. d.].  Berkeley SoftFloat.  http://www.jhauser.us/arithmetic/SoftFloat.html.
  * Hong et al. (2022) Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. 2022.  Handcrafted backdoors in deep neural networks.  _Advances in Neural Information Processing Systems_ 35 (2022), 8068–8080.
  * Horowitz ([n. d.]) Andreessen Horowitz. [n. d.].  Navigating the High Cost of AI Compute.  https://a16z.com/navigating-the-high-cost-of-ai-compute/.
  * Kalodner et al. (2018) Harry Kalodner, Steven Goldfeder, Xiaoqi Chen, S Matthew Weinberg, and Edward W Felten. 2018.  Arbitrum: Scalable, private smart contracts. In _27th USENIX Security Symposium (USENIX Security 18)_. 1353–1370.
  * Kurtulmus and Daniel (2018) A Besir Kurtulmus and Kenny Daniel. 2018.  Trustless machine learning contracts; evaluating and exchanging machine learning models on the ethereum blockchain.  _arXiv preprint arXiv:1802.10185_ (2018).
  * Labs ([n. d.]) Modulus Labs. [n. d.].  The Cost of Intelligence.  https://medium.com/@ModulusLabs/chapter-5-the-cost-of-intelligence-da26dbf93307/.
  * Li et al. (2022) Tao Li, Yaozheng Fang, Ye Lu, Jinni Yang, Zhaolong Jian, Zhiguo Wan, and Yusen Li. 2022.  SmartVM: A Smart Contract Virtual Machine for Fast On-Chain DNN Computations.  _IEEE Transactions on Parallel and Distributed Systems_ 33, 12 (2022), 4100–4116.
  * Liu et al. (2021) Tianyi Liu, Xiang Xie, and Yupeng Zhang. 2021.  ZkCNN: Zero knowledge proofs for convolutional neural network predictions and accuracy. In _Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security_. 2968–2985.
  * Luu et al. (2015) Loi Luu, Jason Teutsch, Raghav Kulkarni, and Prateek Saxena. 2015.  Demystifying incentives in the consensus computer. In _Proceedings of the 22Nd acm sigsac conference on computer and communications security_. 706–719.
  * Mamageishvili and Felten (2023) Akaki Mamageishvili and Edward W Felten. 2023.  Incentive Schemes for Rollup Validators.  _arXiv preprint arXiv:2308.02880_ (2023).
  * Nakamoto (2008) Satoshi Nakamoto. 2008.  Bitcoin: A peer-to-peer electronic cash system.  _Decentralized business review_ (2008).
  * OpenAI ([n. d.]) OpenAI. [n. d.].  OpenAI’s Pricing.  https://openai.com/pricing.
  * Optimism ([n. d.]) Optimism. [n. d.].  Ethereum, scaled. OP Mainnet is a low-cost and lightning-fast Ethereum L2 blockchain powered by Optimism.  https://www.optimism.io/.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ 32 (2019).
  * Roughgarden (2010) Tim Roughgarden. 2010.  Algorithmic game theory.  _Commun. ACM_ 53, 7 (2010), 78–86.
  * Salah et al. (2019) Khaled Salah, M Habib Ur Rehman, Nishara Nizamuddin, and Ala Al-Fuqaha. 2019.  Blockchain for AI: Review and open research challenges.  _IEEE Access_ 7 (2019), 10127–10149.
  * Sanders and Kandrot (2010) Jason Sanders and Edward Kandrot. 2010.  _CUDA by example: an introduction to general-purpose GPU programming_.  Addison-Wesley Professional.
  * Song and Ying (2015) Yan-Yan Song and LU Ying. 2015.  Decision tree methods: applications for classification and prediction.  _Shanghai archives of psychiatry_ 27, 2 (2015), 130.
  * Soucy and Mineau (2001) Pascal Soucy and Guy W Mineau. 2001.  A simple KNN algorithm for text categorization. In _Proceedings 2001 IEEE international conference on data mining_. IEEE, 647–648.
  * Szydlo (2004) Michael Szydlo. 2004.  Merkle tree traversal in log space and time. In _Advances in Cryptology-EUROCRYPT 2004: International Conference on the Theory and Applications of Cryptographic Techniques, Interlaken, Switzerland, May 2-6, 2004. Proceedings 23_. Springer, 541–554.
  * Teutsch and Reitwießner (2019) Jason Teutsch and Christian Reitwießner. 2019.  A scalable verification solution for blockchains.  _arXiv preprint arXiv:1908.04756_ (2019).
  * Wang and Hoang (2022) Haodi Wang and Thang Hoang. 2022.  ezDPS: An Efficient and Zero-Knowledge Machine Learning Inference Pipeline.  _arXiv preprint arXiv:2212.05428_ (2022).
  * Weng et al. (2021) Chenkai Weng, Kang Yang, Xiang Xie, Jonathan Katz, and Xiao Wang. 2021.  Mystique: Efficient conversions for $\\{$Zero-Knowledge$\\}$ proofs with applications to machine learning. In _30th USENIX Security Symposium (USENIX Security 21)_. 501–518.
  * Wood et al. (2014) Gavin Wood et al. 2014\.  Ethereum: A secure decentralised generalised transaction ledger.  _Ethereum project yellow paper_ 151, 2014 (2014), 1–32.
  * Wüst et al. (2020) Karl Wüst, Sinisa Matetic, Silvan Egli, Kari Kostiainen, and Srdjan Capkun. 2020.  ACE: Asynchronous and concurrent execution of complex smart contracts. In _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_. 587–600.
  * Xing et al. (2023) Zhibo Xing, Zijian Zhang, Meng Li, Jiamou Liu, Liehuang Zhu, Giovanni Russello, and Muhammad Rizwan Asghar. 2023.  Zero-Knowledge Proof-based Practical Federated Learning on Blockchain.  _arXiv preprint arXiv:2304.05590_ (2023).
  * Yang et al. (2019) Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. 2019.  Quantization networks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7308–7316.
  * Yang et al. (2022) Qinglin Yang, Yetong Zhao, Huawei Huang, Zehui Xiong, Jiawen Kang, and Zibin Zheng. 2022.  Fusing blockchain and AI with metaverse: A survey.  _IEEE Open Journal of the Computer Society_ 3 (2022), 122–136.
  * Zero ([n. d.]) RISC Zero. [n. d.].  RISC Zero : General-Purpose Verifiable Computing.  https://www.risczero.com/.
  * Zhang et al. (2020) Jiaheng Zhang, Zhiyong Fang, Yupeng Zhang, and Dawn Song. 2020.  Zero knowledge proofs for decision tree predictions and accuracy. In _Proceedings of the 2020 ACM SIGSAC Conference on Computer and Communications Security_. 2039–2053.
  * Zheng et al. (2021) Zihan Zheng, Peichen Xie, Xian Zhang, Shuo Chen, Yang Chen, Xiaobing Guo, Guangzhong Sun, Guangyu Sun, and Lidong Zhou. 2021.  Agatha: Smart contract for DNN computation.  _arXiv preprint arXiv:2105.04919_ (2021).


# Application of ASV for Voice Identification after VC and Duration Predictor
Improvement in TTS Models

###### Abstract

One of the most crucial components in the field of biometric security is the
automatic speaker verification system, which is based on the speaker’s voice.
It is possible to utilise ASVs in isolation or in conjunction with other AI
models. In the contemporary era, the quality and quantity of neural networks
are increasing exponentially. Concurrently, there is a growing number of
systems that aim to manipulate data through the use of voice conversion and
text-to-speech models. The field of voice biometrics forgery is aided by a
number of challenges, including SSTC, ASVSpoof, and SingFake.

This paper presents a system for automatic speaker verification. The primary
objective of our model is the extraction of embeddings from the target
speaker’s audio in order to obtain information about important characteristics
of his voice, such as pitch, energy, and the duration of phonemes. This
information is used in our multivoice TTS pipeline, which is currently under
development. However, this model was employed within the SSTC challenge to
verify users whose voice had undergone voice conversion, where it demonstrated
an EER of 20.669.

Index Terms—  ASV, voice conversion, TTS, counter spoofing.

##  1 Introduction

Voice conversion is a technique employed to alter the voice of one individual
to that of another. VC models are employed in the enhancement of TTS systems,
the alteration of the voice of call centre operators with the objective of
increasing sales, and the transformation of the human voice into that of
fictional characters. However, these models can also be employed for
unscrupulous purposes, such as deceiving bank employees, perpetrating fraud by
impersonating relatives, and committing identity forgery.

The process of text-to-speech conversion involves the transformation of text
into audio that is intended to resemble human speech. A high-quality TTS model
is capable of producing a voice that is genuinely real-life in its likeness,
and thus indistinguishable from the real thing.Such systems are employed in a
variety of fields, enabling the automation of processes that were previously
infeasible. Examples include the voice-acting of books and advertisements,
websites and applications for visually impaired individuals, and so forth.
However, as with VC models, such architectures can also be employed for the
purpose of deception.

The advent of modern technology has led to the rapid development of systems
capable of detecting modified and synthetic voice. Such models are referred to
as ’antispoofing’ models. The community ASVspoof([1], [2], [3], [4]), which
regularly organises challenges and collects data corpuses to study the
detection of voice biometric forgery, plays an active role in the development
of such systems. It is also important to note the contribution of the SingFake
competition[5], which highlighted the significant shortcomings of SOTA models
on the ASVSpoof benchmark. In particular, the researchers identified two key
issues with the models in question. Firstly, they noted that the models lack
generalisability in the context of music. Secondly, they observed that the
models are not as effective when applied to different languages. This
illustrates the need for further research and development of models that can
effectively detect spoofing in voice biometrics.

The detection of voice spoofing is of great import, as effective detection can
prevent a multitude of malicious activities involving the use of a spoofed
voice. However, the problem of identifying the attacker is not insignificant.
Thus, if the original voice can be identified in a fake voice that may have
been obtained using a voice conversion system, a scammer can be identified.

The current state of research in this area is insufficient. In this context,
the SSTC challenge is of particular importance, as it provided researchers
with the opportunity to explore in depth methods for identifying the original
voice in fake audio recordings. The database facilitates research in this
field, thereby facilitating the emergence of novel methods for detecting the
original voice of an attacker, which in turn enhances the overall level of
security in the field of voice biometrics.

In order to construct a TTS system, it was deemed necessary to obtain a number
of crucial speech characteristics of the speaker. In order to achieve this, a
system for automatic speaker verification was employed, with the use of
special encoders to obtain as much useful speaker information as possible.
This paper describes a model that discriminates speakers acceptably and its
embeddings contain sufficient information to predict various voice
characteristics. Furthermore, the impact of the model’s embeddings on
predicting the duration of each phoneme is also described. In addition, we
participated in an SSTC challenge in which we were required to verify voices
that had undergone voice conversion. The outcomes of this Challenge, along
with a comprehensive overview of the system, are presented in this paper.

##  2 Model encoders

A series of encoders, each configured to accept a different audio frontend,
were employed to extract high-level feature maps. Sound representations were
employed in this study, including the Constant-Q Transform (CQT), Mel
spectrogram, and Pitch spectrogram. This section provides a detailed
description of the implementation of models for the primary features from the
previously mentioned audio transformations.

###  2.1 CQT Encoder

The Non-stationary Gabor Transform algorithm [6], with the following
parameters, was employed to obtain the CQTs:

  * •

Number of octaves = 8

  * •

Number of bins per octave = 64

  * •

Block length = 1 second

The obtained sound representation was encoded using a 10-block Specblock
encoder. Each Specblock comprises a complex convolution layer, a complex ELU
activation function and a complex dropout.

Fixed configuration of complex convolutional layers:

  * •

kernel size = 3

  * •

stride = 2

  * •

padding = 1

Complex ELU is an activation function for complex numbers that is applied by
separately applying ELU to the real and imaginary parts. The resulting
components are then recombined into a single complex number. The complex
dropout operates in a manner analogous to that of a regular dropout: a random
selection process, with a probability of 0.4, results in the nullification of
some parameters.

Following the encoding process, the real and imaginary parts are concatenated
along the first dimension. The configuration of the increase in the number of
channels in the convolutional layers is presented in Table. 1.

block number | input channels | output channels
---|---|---
1 | 1 | 32
2 | 32 | 32
3 | 32 | 32
4 | 32 | 32
5 | 32 | 64
6 | 64 | 64
7 | 64 | 128
8 | 128 | 128
9 | 128 | 128
10 | 128 | 128
Table 1: Channel configuration

###  2.2 Mel-spectrogram Encoder

To obtain the mel spectrogram, we employed the following parameters for
conversion:

  * •

Size of Fast Fourie Transform, as well as the window size = 1201

  * •

Hop length = 600

  * •

Number of mel filterbanks = 128

  * •

Type of window - Hann

In order to obtain high-level feature maps from mel-spectrograms, a Vision
Transform(ViT) [7] with the following configuration was employed:

  * •

Embedding dimensionality = 256

  * •

Hidden dimensionality = 512

  * •

Number of attention heads = 32

  * •

Number of layers = 3

  * •

Output dimensionality = 512

  * •

Patch size = 16

  * •

Number of patches = 320

###  2.3 Pitch Encoder

The pitch spectrogram was obtained by first utilising the distributed inline
filtering with overlap (DIO) algorithm to generate a series of continuous
pitch contours. Subsequently, the Contious Wavelet Transform(CWT)([8], [9],
[10]) was applied. The measurements were conducted in scale factors within the
range [1, 37]. The resulting pitch spectrogram was run through the VIT with
the following parameters:

  * •

Embedding dimensionality = 256

  * •

Hidden dimensionality = 512

  * •

Number of attention heads = 32

  * •

Number of layers = 3

  * •

Output dimensionality = 512

  * •

Patch size = 9

  * •

Number of patches = 356

##  3 model and loss function

The input data for the model is a 4-second audio segment. To ensure the
uniformity of input data, the data were resampled to 24,000 before being fed
into the model. The segment of length 4 was selected at random from the
original audio recording.

The prepared audio is fed to the Pitch encoder, the CQT encoder and the mel-
spectrogram encoder. The resultant vectors are then concatenated along first
dimension following their passage through the encoders. They are then passed
through a fully-connected layer with an output feature count of 2048. The
results obtained are passed through the ELU activation function. This process
involves the extraction of primary embeddings.

The model was trained using the Additive Margin Softmax (AM-Softmax) [11] loss
function with a parameter scale of 30 and a margin of 0.4. The loss function
is presented in eq. 1

$\mathcal{L}=-\frac{1}{n}\sum^{n}_{i=1}\log\frac{e^{30(W^{T}_{y_{i}}f_{i}-0.4)}}{e^{30(W^{T}_{y_{i}}f_{i}-0.4)}+\sum^{c}_{j=1,j\neq
y_{i}}e^{30W^{T}_{j}f_{i}}}$ (1)

##  4 EXPERIMENTS AND RESULTS OF THE MODEL AS PART OF TTS

###  4.1 Datasets description

To test the model, we used an open
dataset111https://www.kaggle.com/datasets/kongaevans/speaker-recognition-
dataset containing the speeches of the following five speakers: Benjamin
Netanyahu, Jens Stolnberg, Julia Gillard, Margaret Tarcher and Nelson Mandela.
There are 1,500 speech samples for each speaker. The dataset was collected
using open data from American
Rhetoric222https://www.americanrhetoric.com/speechbank.htm  (online speech
bank).

Despite the large number of samples used, using only 5 to test ASV is not
sufficient. In addition to the above dataset, we also used the CMU ARCTIC[12],
which consists of 18 speakers, each of which has 500-1000 speech samples
collected.

We have used LibriTTS-R [13] for the training of the model. From the train
subset of this dataset, 800 speakers were randomly selected, with 200
utterances selected for each speaker.

###  4.2 Metrics description

Since the model was trained to ensure that the embeddings of one person were
close to each other in the multidimensional space and those of different
people were far apart, we needed to use a metric that would estimate the ratio
of cosine similarities between identical pairs of voices and between different
pairs of voices at a certain threshold.

We chose TPR @ FPR. The idea behind this metric is that, given a fixed rate of
false positive pairs, we count how many true positive pairs we have before
this last allowed false positive pair, and then divide the resulting number by
the total number of true positive pairs. The algorithm used to calculate this
metric:

  1. 1.

Fixing the threshold value TPR @ FPR = N.

  2. 2.

Calculate the cosine similarity between all positive pairs.

  3. 3.

Calculate the cosine similarity between all negative pairs.

  4. 4.

Calculate how many elements make up N times the number of false positive
pairs. This number is k.

  5. 5.

Sort all cosine similarity values of false pairs in descending order. The K-th
number is the threshold distance.

  6. 6.

Count the number of positive pairs whose cosine distance is less than the
threshold distance.

  7. 7.

Divide the number obtained by the total number of positive pairs.

###  4.3 Loss function selection

The choice was between ArcFace [14] with scale=30, margin=0.5 and AM-Softmax
3. The model was trained and tested on non-overlapping subsets of the test
dataset for 5 epochs. The result is shown in Table 2

metric/loss | ArcFace | AM-Softmax
---|---|---
<EMAIL_ADDRESS>| 0.7134 | 0.906
<EMAIL_ADDRESS>| 0.3785 | 0.6846
<EMAIL_ADDRESS>| 0.2429 | 0.5101
<EMAIL_ADDRESS>| 0.1560 | 0.3960
<EMAIL_ADDRESS>| 0.04291 | 0.2148
Table 2: Different loss function training results

Based on these results, we decided to use AM-Softmax.

###  4.4 Training results

We trained the model for 30 epochs, for a total training time of 30 hours. The
training was carried out using x1 Tesla A100 40GB. The checkpoint model at the
epoch when it exhibited the highest<EMAIL_ADDRESS>on the validation subsample
was selected for evaluation. The outcomes of the model on the test set are
presented in Table 3.

metric | value
---|---
<EMAIL_ADDRESS>| 0.9869
<EMAIL_ADDRESS>| 0.9469
<EMAIL_ADDRESS>| 0.9044
<EMAIL_ADDRESS>| 0.8354
<EMAIL_ADDRESS>| 0.5920
Table 3: Results on testing subsample

Furthermore, we proceeded to test our model on in-the-wild data. Four distinct
voices were recorded from open sources, ensuring that they were not included
in any of the datasets used in the experiments. The voices were divided into
two categories, male and female, in a one-to-one ratio. A two-minute audio
segment was divided into a four-second fragment with no overlap. The
distribution in the embedding space obtained with t-SNE is illustrated in
Figure 1.

Fig. 1: Embeddings distribution

###  4.5 Impact of ASV embeddings on the duration predictor

The duration predictor is one of the key components of our TTS Pipeline, which
is currently under development. The objective of the duration predictor is to
predict the duration of pronunciation of phonemes by a specific speaker. As
this work does not extend to the deepening of TTS, for the sake of
convenience, we refer to the duration predictor architecture as a black box.
The black box architecture is immutable and can be trained.

The target values were obtained from the annotation generated by the Montreal
Forced Aligner [15]. For each phoneme, the length of the corresponding mel-
spectrogram frame was calculated using the following transformation
parameters:

  * •

Size of Fast Fourie Transform, as well as the window size = 1024

  * •

Hop length = 256

  * •

Number of mel filterbanks = 128

  * •

Type of window - Hann

Four experiments were conducted in order to investigate the effect of
embeddings on black box predictions. The following experiments were conducted,
with the type of input differing as follows: encoded phonemes; encoded
phonemes concatenated with a random noise of embedding size; encoded phonemes
concatenated with an embedding of the same speaker; encoded phonemes
concatenated with an embedding of the same utterance.

The following metrics were employed for the evaluation of the models: MAE,
RMSE, weighted duration error rare and the concordance correlation coefficient
[16]. The weighted DER metric was devised to quantify a range of model errors.
Given that phoneme duration is a relatively subjective value, with each
individual pronouncing the same word in a distinct manner, we have devised a
metric that allows for minor deviations from the target values to be
compensated for, while larger discrepancies are reflected in a larger error
coefficient, as outlined in eq. 2, 3

$f_{wder}(x,\hat{x})=\begin{cases}1,&|{x=\hat{x}}|\leq 1\\\
|x-\hat{x}|^{-1},&otherwise\end{cases}$ (2)

$WDER=\frac{1}{N}\sum^{n}_{i=0}f_{wder}(x_{i},\hat{x_{i}})$ (3)

The results of the model following four experiments are presented in Table 4.

experiment/result |  MAE $\downarrow$ |  RMSE$\downarrow$ |  WDER$\uparrow$ |  CCC$\uparrow$
---|---|---|---|---
encoded phonemes | 2.045 | 3.836 | 0.9309 | 0.771
encoded phonemes +random noise | 2.198 | 4.366 | 0.9364 | 0.7328
encoded phonemes +speaker embedding | 1.87 | 3.607 | 0.949 | 0.801
encoded phonemes +utterance embedding | 1.869 | 3.503 | 0.9501 | 0.8038
Table 4: The results of the duration predictor experiments

As illustrated in Table 4, the results demonstrated that the embeddings of the
same utterance exhibited the most optimal performance, indicating that the
phoneme length information is indeed retained within the embeddings.
Concurrently, the results with embeddings of another utterance by the same
speaker also demonstrated good results, indicating the retention of phoneme
duration information. The experiment in which random noise was added to the
embeddings demonstrated a decline in the results, which lends support to the
hypothesis that it is the speaker’s embeddings that retain phoneme length
information.

##  5 EXPERIMENTS AND TRAINING RESULTS FOR SSTC

###  5.1 Dataset description

The SSTC dataset utilises the Librispeech [17] train-clean-100, train-
clean-360, dev.clean, test.clean datasets as the source speaker dataset, and
the VoxCeleb 2 dev [18], VoxCeleb 1 test [19] datasets as the target speaker
datasets.

The dataset was designed to identify the original speaker in modified audio.
The audio was modified using 16 different voice conversion models.

###  5.2 Model training

A detailed description of the architectural design of the model used in this
Challenge can be found in Section 3. The model was trained exclusively on the
LibriSpeech corpus. It should be noted that no pre-training was applied to the
data from other sources. The model was trained for 30 epochs on a Tesla A100
with 40 GB.

The model was trained on the SSTC training subset for 22000 steps, one epoch
lasts 9100 steps, or 22 hours 34 minutes. A batch size of 52 items was
employed, with the number of classes for AM-Softmax being 1172. For the
purposes of training, the x3 Tesla A100 40 GB was employed, for validation and
test x1 Tesla P100 12Gb. Due to the limited timeframe of the Challenge, it was
not possible to conduct training on more epochs. The results of the training
are presented in Table 5.

subset | EER %
---|---
development | 46.981
test | 44.809
Table 5: Results on SSTC

###  5.3 Model ensembling

To enhance the performance of our model, we decided to apply the stacking
ensemble technique. As a second algorithm for model composition, we selected
the baseline model, which was provided by the organisers of the competition.
As a solver algorithm, we employ a variety of averaging techniques. The final
results, denoted by $Y_{i}$, were obtained by combining the vector of results
from our model, $X_{1}$, with those from the second model, $X_{2}$. This was
done in accordance with the following eq. 4, 5, 6:

$Y_{1}=\frac{1}{2}(X_{1}-0.99)*100+\frac{1}{2}X_{2}$ (4)

$Y_{2}=0.35*(X_{1}-0.99)*100+0.65*X_{2}$ (5)

$Y_{3}=0.25*(X_{1}-0.99)*100+0.75*X_{2}$ (6)

The results for these ensemble techniques are presented in Table 6.

ensemble technique | EER %
---|---
$Y_{1}$ | 23.625
$Y_{2}$ | 20.762
$Y_{3}$ | 20.669
Table 6: Results of different ensemble techniques.

We found that stacking two models allowed us to improve the quality of
predictions on the test sample by almost two times.

##  6 conclusion

In this work, we explored the potential of an automatic speaker verification
system for a range of applications. Our findings suggest that the model
trained for user verification may offer insights that could enhance the
performance of text-to-speech models in terms of duration prediction.

We also explored the possibility of training the model to identify the
original speaker in audio that has been modified with voice conversion. While
we were able to achieve an EER=20, we believe that the dataset proposed in the
SSTC competition offers a promising field for further research.

## References

  * [1] Z. Wu, T. Kinnunen, N. Evans et al.,  “Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,”  in INTERSPEECH 2015, Automatic Speaker Verification Spoofing and Countermeasures Challenge, colocated with INTERSPEECH 2015, September 6-10, 2015, Dresden, Germany, ISCA, Ed., Dresden, 2015.
  * [2] H. Delgado, M. Todisco, M. Sahidullah et al.,  “Asvspoof 2017 version 2.0: meta-data analysis and baseline enhancements,”  in Proc. The Speaker and Language Recognition Workshop (Odyssey 2018), 2018, pp. 296–303.
  * [3] X. Wang, J. Yamagishi, M. Todisco et al.,  “Asvspoof 2019: A large-scale public database of synthesized, converted and replayed speech,” 2020.
  * [4] J. Yamagishi, X. Wang, M. Todisco et al.,  “Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection,” 2021.
  * [5] Y. Zang, Y. Zhang, M. Heydari and Z. Duan,  “Singfake: Singing voice deepfake detection,” 2024.
  * [6] N. Holighaus, M. Dorfler, G. A. Velasco and T. Grill,  “A framework for invertible, real-time constant-q transforms,”  2012\.
  * [7] A. Dosovitskiy, L. Beyer, A. Kolesnikov et al.,  “An image is worth 16x16 words: Transformers for image recognition at scale,” 2021.
  * [8] A. Suni, D. Aalto, T. Raitio et al.,  “Wavelets for intonation modeling in hmm speech synthesis,”  01 2013.
  * [9] K. Hirose and J. Tao,  “Speech prosody in speech synthesis: Modeling and generation of prosody for high quality and flexible speech synthesis,”  2015\.
  * [10] G. R. Lee, R. Gommers, F. Waselewski et al.,  “Pywavelets: A python package for wavelet analysis,”  Journal of Open Source Software, vol. 4, no. 36, pp. 1237, 2019.
  * [11] F. Wang, J. Cheng, W. Liu and H. Liu,  “Additive margin softmax for face verification,”  IEEE Signal Processing Letters, vol. 25, no. 7, pp. 926–930, July 2018.
  * [12] J. Kominek and A. W. Black,  “The cmu arctic speech databases,”  in Proc. 5th ISCA Workshop on Speech Synthesis (SSW 5), 2004, pp. 223–224.
  * [13] Y. Koizumi, H. Zen, S. Karita et al.,  “Libritts-r: A restored multi-speaker text-to-speech corpus,” 2023.
  * [14] J. Deng, J. Guo, J. Yang et al.,  “Arcface: Additive angular margin loss for deep face recognition,”  IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 5962–5979, Oct. 2022.
  * [15] M. McAuliffe, M. Socolof, S. Mihuc et al.,  “Montreal forced aligner: Trainable text-speech alignment using kaldi,”  08 2017, pp. 498–502.
  * [16] L. I.-K. Lin,  “A concordance correlation coefficient to evaluate reproducibility,”  Biometrics, vol. 45, no. 1, pp. 255–268, 1989.
  * [17] V. Panayotov, G. Chen, D. Povey and S. Khudanpur,  “Librispeech: An asr corpus based on public domain audio books,”  in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.
  * [18] J. S. Chung, A. Nagrani and A. Zisserman,  “Voxceleb2: Deep speaker recognition,”  in Interspeech 2018. Sept. 2018, interspeech2018, ISCA.
  * [19] A. Nagrani, J. S. Chung and A. Zisserman,  “Voxceleb: A large-scale speaker identification dataset,”  in Interspeech 2017. Aug. 2017, interspeech2017, ISCA.