# AI-assisted Tagging of Deepfake Audio Calls using Challenge-Response

Govind Mittal1, Arthur Jakobsson2, Kelly O. Marshall1, Chinmay Hegde1, Nasir
Memon1
1New York University, Tandon School of Engineering, Brooklyn, NY
2Carnegie Mellon University, Pittsburgh, PA
{mittal, km3888, chinmay.h<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Scammers are aggressively leveraging AI voice-cloning technology for social
engineering attacks, a situation significantly worsened by the advent of audio
Real-time Deepfakes (RTDFs). RTDFs can clone a target’s voice in real-time
over phone calls, making these interactions highly interactive and thus far
more convincing. Our research confidently addresses the gap in the existing
literature on deepfake detection, which has largely been ineffective against
RTDF threats. We introduce a robust challenge-response-based method to detect
deepfake audio calls, pioneering a comprehensive taxonomy of audio challenges.
Our evaluation pitches 20 prospective challenges against a leading voice-
cloning system.

We have compiled a novel open-source challenge dataset with contributions from
100 smartphone and desktop users, yielding 18,600 original and 1.6 million
deepfake samples. Through rigorous machine and human evaluations of this
dataset, we achieved a deepfake detection rate of 86% and an 80% AUC score,
respectively. Notably, utilizing a set of 11 challenges significantly enhances
detection capabilities. Our findings reveal that combining human intuition
with machine precision offers complementary advantages. Consequently, we have
developed an innovative human-AI collaborative system that melds human
discernment with algorithmic accuracy, boosting final joint accuracy to 82.9%.
This system highlights the significant advantage of AI-assisted pre-screening
in call verification processes. https://mittalgovind.github.io/autch-samples/

##  1 Introduction

Recent advancements in synthetic speech generation have obscured the
distinction between authentic and fabricated media [18, 14, 24]. The impact of
such advancements is felt particularly with the advent of voice-cloning tools
that can generate speech that sounds authentic in real-time [26, 12, 25, 43,
41]. The existence of such tools presents a critical vulnerability for
exploitation during social engineering and raises an urgent question: _How can
one ascertain the authenticity of a caller in an age where synthetic voices
are indistinguishable from real ones?_

Phone call-based social engineering attacks are highly plausible, especially
since 2019, when online interactions became mainstream. An example of a phone
scam is robocalls, whose numbers in the U.S. peaked at an astonishing 58.5
billion. Robocalls even used the voices of the U.S. President Biden [7] and
Mayor of New York City for voter suppression and outreach [8]. This concerning
escalation prompted repeated warnings from the FBI [21], the FTC [36], and an
eventual prohibition against robocalls by the FCC [20].

Consider a more believable and targeted form of phone scams than robocalls
leveraging _Real-time Deepfakes or RTDFs_. Such deepfakes make scam calls
sound convincingly close to a target’s voice, and being real-time, they become
interactive and capable of conversing. Such fake calls are more risky as they
are more believable and, hence, more persuasive. We will shortly discuss how
we used RTDF’s inherent interactivity to tackle them.

One could assume that decades of technological advancements in voice calling
should be able to counter most scams. However, several incidents where
scammers exploited audio deepfakes highlight the inadequacy of current
telecommunications and regulatory measures to contain such threats. Such
incidents include convincing a finance worker to remit $25 million by sounding
like their chief financial officer over a group video call [6], demanding a
$50,000 ransom by mimicking a daughter’s distressed voice to deceive her
mother [16] and defrauding an energy company of $243,000 by impersonating
their boss’ voice over the phone [22]. Moreover, a McAfee survey of 7,000
individuals worldwide found that 25% had encountered an AI voice cloning scam
or knew someone who had [29], revealing a troubling reality: the end users
have been left responsible for discerning a genuine caller from an imposter.

In an ideal scenario, an aware individual would approach every call with a
healthy degree of skepticism, and a suspicious caller could never make any
gains. In reality, receiving a call from a known caller ID [2] and the mere
recognition of a familiar voice can swiftly establish trust, even amidst noise
or voice distortions. Typically, an explanation from the caller, asserting
they are under the weather or in an area with poor reception, is enough to
dispel any doubts [42]. Moreover, sophisticated RTDFs now challenge
conventional security protocols, such as speaker verification systems and
liveness detection in KYC processes, making them prone to evasion [3, 4, 5].

Numerous audio deepfake detection methods have been developed [47]. However,
they are primarily designed for static content and have limited efficacy
against RTDFs. These methods assume a non-interactive scenario and work
offline, allowing impostors ample time to refine manipulated content with
sophisticated editing tools.

In this work, we depart from the traditional techniques and leverage
challenge-response to pursue an alternative approach to detecting deepfake
audio calls. Challenge-response mechanisms, traditionally pivotal in
distinguishing bots from humans via the ubiquitous Captchas, have taken on new
ground with the introduction of Gotcha [32] and D-Captcha [46] for unmasking
video and audio deepfakes. These pioneering studies capitalize on the
asymmetric advantage where the burden to maintain high quality in real-time,
under challenging situations, is now squarely on the imposter caller. This
work aims to harness the full spectrum of audio challenges during phone calls
and orchestrate a collaborative dynamic between humans and AI, thereby
fortifying detection accuracy and confidence.

We began by curating a detailed taxonomy of audio challenges, categorizing
them into eight main types and 22 sub-types, including vocal distortions
(e.g., whispering), waveform manipulations (e.g., high-pitch speaking),
language-specific articulations (e.g., rolling R sounds), environmental noise
(e.g., clapping while speaking), and background playbacks (e.g., talking over
music). We selected and evaluated 20 prospective challenges from the taxonomy.

Our investigation revolves around three research questions:

  1. RQ1:

Do challenges enhance machine detection?

  2. RQ2:

Can human evaluators harness these challenges to sharpen their discernment?

  3. RQ3:

Does supporting humans with automated detectors further improve overall
performance?

We collected an open-source novel dataset to address these questions by
engaging 100 participants across mobile and desktop interfaces to produce
18,600 original voice recordings. We subsequently generated 1.6 million
samples through a state-of-the-art one-shot voice-cloning technology.

We performed machine evaluation***we used ‘Machines’ and ‘AI’ interchangeably
by designing a non-intrusive degradation metric for assessing audio samples on
compliance, naturalness, and the preservation of word information. This metric
unveiled that a select group of 11 challenges dramatically boosted the
deepfake detection performance (Area Under Curve or AUC) to 86.7%, compared to
56.0% in deepfakes of everyday speech, highlighting the benefit of using
challenges to aid machines. Notably, challenges like whispering, cupping
mouth, and playback were among the most effective for machines.

Concurrently, we performed a human evaluation on a harder subset of the whole
dataset. SpeechBrain’s speaker-verification system [37] classified all
deepfake samples in this subset as genuine. Humans assessed the compliance and
quality of each audio sample and achieved an AUC of 80.1%, with challenges
bringing out discernible degradation in deepfakes.

Informed by these results and the propensity of humans to produce more false
positives, we envisioned a framework where machines assisted humans in making
decisions. This integration makes our approach scalable while keeping the
outcome interpretable with human oversight. A subsequent human evaluation with
top-performing challenges, using a balanced dataset, revealed that machine
assistance increased human detection accuracy from 72.3% to 78.5%.
Furthermore, machine assistance significantly boosted human confidence and
rectified their errors in 43% of cases (when machines were correct) while
causing misjudgments in 29% (when machines were wrong).

Taking this integration a step further, in scenarios where humans were
uncertain, machines were allowed to take charge and make the final call. This
blend of Human-AI contributions, at a ratio of 56:44, enhanced the overall
detection accuracy to 82.9%, representing a 14.3% improvement over the human-
only baseline (with machine accuracy standing at 85%). This approach
underscores that while humans maintain their primary role in decision-making,
strategic machine involvement can dramatically enhance the accuracy of
detecting fake calls.

Our findings validated the instrumental role of challenge-response mechanisms
in bolstering both machine and human capabilities in audio deepfake detection,
showcasing a synergy between human insight and AI accuracy that mitigates the
threats posed by real-time synthetic speech technologies.

##### Distinction from Robocalls and Liveness Detection.

Robocalls are like spam, distributed en masse to the population in an
automated manner. Generally, the message is untargeted and is spread to a
large population. However, RTDFs (context for this work) are targeted and
possibly curated for the particular receiver.

In principle, liveness detection mechanisms are ineffective against RTDFs, as
the imposter generates deepfakes in real-time; hence, they are already live.
Also, there is a person behind the call who can understand the instructions
and can converse. Thus, each interaction could unroll differently.

##  2 Background

##### Definition.

An audio deepfake refers to a digital impersonation in which an imposter
mimics audio characteristics to convincingly match a specific target’s
likeness. When such impersonations can be done live with sufficient fidelity,
we call them RTDFs.

###  2.1 Cognitive Bias and Human Susceptibility

In the context of deepfakes, individuals are left to navigate the landscape on
their own, letting scammers abuse their trust during phone calls. Two pivotal
factors contribute to the phenomenon of rapid trust-building over calls.
First, there exists a general human acknowledgment of the fallibility of our
auditory perception. Mishearing is a shared experience, arguably more so than
visual misinterpretation. We naturally integrate multi-sensory inputs into
listening, such as live transcription, facial expressions, mouth movements,
and the conversation context, to enhance our ability to discern speech. These
cues are sufficiently potent to augment and even override auditory perception,
as demonstrated by the McGurk effect [30]. Under this effect, the visual
information from seeing a person’s mouth speak changes how one hears the
speech, leading to misheard syllables. This phenomenon highlights our innate
understanding that auditory signals could be unreliable, which can get
amplified over audio calls where the channel is noisy and the visual component
is absent.

Second, the technological landscape has molded our expectations of sound
quality. Individuals who grew accustomed to the quirks of early mobile
telephony – limited bandwidth, dropped calls, background noise, and frequent
verbal confirmations to ensure mutual understanding – have been conditioned to
interpret and understand communication even in subpar auditory conditions.
Despite improvements in audio clarity, exposure to imperfect sound has
ingrained tolerance for noisy or unclear speech in us. Such tolerance
accentuates our inherent bias to trust noisy speech.

This evolutionary and technological adaptation in auditory perception is
evident in how humans rate audio quality. When deploying mean opinion scores
(MOS) for speech naturalness assessment, the scale customarily spans from 1
(being incomprehensible) to 5 (akin to face-to-face). Nonetheless, the average
difference between the highest and lowest MOS scores is often only two points
and skewed towards the higher end [38]. This indicates a degree of leniency or
an adaptive bias in our auditory judgments, underscoring our propensity to
make sense of auditory inputs despite the quality.

This innate human tendency to trust auditory inputs instead of mistrusting
them offers a potent avenue for exploitation in social engineering attacks
employing audio deepfakes.

###  2.2 Problem Description

##### Threat Model.

Three parties are involved in this scenario: an impostor, the intended
identity (the target), and a defender. The defender receives a call of a
sensitive nature, answering to interact directly with the target. However, an
impostor may employ AI-generated speech synthesis tools to communicate with
the defender, masquerading as the target. The primary objective of the
defender is to authenticate the caller’s identity before proceeding further.

A sensitive call refers to a remote communication initiated to acquire
something from the recipient, who possesses both the capability and
willingness to grant it. Such calls include job interviews, phone banking,
disreputable interactions, and pranks.

Defenders. The defender does not assume any trust in the potential imposter or
their devices. The defender requests imposters to perform a specific task. The
nature of requests made is public. However, a seed is initialized for
randomizing the task. No identifiable target information, such as voice
biometrics, is collected through an extensive enrollment process. The defender
works against a strong threat model, as identity information can help them.

Imposters. An imposter can access sufficient computational resources to run
voice-cloning software in real-time. They can discern and respond to requests
made to them.

An imposter can obtain a high-quality speech sample of the target, potentially
via social media or an unsolicited phone call (pre-recorded voicemail
greetings). This scenario is particularly impactful, as it requires scant data
and can cause widespread harm.

##### Hypothesis.

Speech communication constitutes several vital elements, including phonetics,
articulation, pitch, tone, rhythm, stress, voice quality, and fluency, all of
which combine to create a distinct auditory experience. An audio deepfake
system should fail to maintain fidelity in real time while supporting all such
elements.

##  3 Related Work

##### Voice-Cloning.

Voice cloning systems aim to separate speaker identity from content in a given
target’s data and replace it with the desired content at inference time. Such
speech generation systems broadly differ in their form of input: text or
speech. In our analysis, Text-to-Speech models [39, 12] excelled in content
modulation but fell short in capturing speech nuances like emotions or
prosody, severely limiting them against our challenges. Our focus thus shifted
to speech-to-speech systems, which convert a source speech into the targeted
speaker’s voice while maintaining subtle elements of speech.

Among the various voice converters available, such as PPG-VC [28],
StarGANv2-VC [27], FREE-VC [25], StreamVC [45], and KNN-VC [9], we choose
FREE-VC for our evaluation. As FREE-VC is noted for its open-
source†††https://github.com/OlaWod/FreeVC any-to-any voice cloning
capabilities, outperforming earlier models and matching recent advancements,
with Korshunov et al. [23] considering it as a high threat to speaker
recognition systems, we assume it will be an imposter’s top choice.

##### Speaker Verification.

Speaker Verification aims to match the speaker in an unknown speech sample to
a clean reference speech sample. In our work, when we test SpeechBrain [37] on
our original and deepfake regular speech dataset, it scores a near-perfect
98.5%. However, the availability of a clean reference file makes such an
approach impractical and incompatible with our threat model.

##### Challenge-Response Systems.

Authentication schemes utilizing challenge-response mechanisms are used in the
ubiquitous CAPTCHA to differentiate bots masquerading as humans on the
internet. Audio CAPTCHAs [19] extends this approach for people with visual
impairment. The Gotcha system [32] introduced a challenge-response approach
for real-time video deepfake detection, analyzing eight video challenges. This
study concluded that challenges lead to consistent and measurable degradation
in detecting real-time video deepfakes, with a focus on interpretability.

D-CAPTCHA [46] developed a challenge-response system for audio calls,
evaluating nine challenges and measuring detection performance based on
realism, compliance, and identity against deepfake generators, with
StarGANv2-VC [27] being the most potent deepfake generator tested. Their
research is a vital proof-of-concept for a more systematic and robust
approach, indicating the promising potential of challenge-response systems in
detecting deepfake audio calls. Our work builds upon this by exploring 20
challenges and developing a pre-screener for calls that assists and
collaborates with human receivers.

##### Human Evaluation of Deepfakes.

Muller et al. [34] conducted a study in which participants competed against an
AI model, RawNet2 [40], to identify audio deepfakes. Participants were shown
the AI’s classification after listening to an audio clip and deciding its
authenticity. The study compared human performance against AI in detecting
deepfakes without allowing participants to revise their choices. The
conclusion was that humans and AI exhibit similar strengths and weaknesses
across various spoofing tasks from ASVSpoof2019 [35]. Contrary to their
findings, our work provides empirical evidence that human and AI performances
are not strongly correlated, often complementing each other’s decision-making.

Table 1: Taxonomy of Audio Challenges encompassing eight categories and 22
subcategories with examples. We evaluated the bold numbered examples as 20
prospective challenges.

data/taxonomy_only

##  4 Speech Challenges and their Taxonomy

##### Definition.

An audio challenge is:

  * •

a task plausible to perform during a phone call,

  * •

induces degradation in real-time deepfakes,

  * •

supports randomization, and

  * •

optionally be verifiable by humans.

This section describes a taxonomy of audio tasks that challenge RTDFs to adapt
to novel inputs. RTDFs facing unusual voice patterns produce speech artifacts,
which later aid us in discovering their presence on calls. We delineated the
taxonomy in Table 1, intending to stimulate variation in several factors that
constitute phone calls, like linguistics, tone of voice, background, and
behavioral cues.

###  4.1 Lingual Challenges

Lingual challenges prompt individual to alter their vocal characteristics in
distinctive ways. These challenges exploit a human vocal system’s degrees of
freedom, such as vocal fold shape, peripherals movements, lung pressure,
resonance and phonation modes.

Vocal Distortions include challenges that manipulate mouth peripherals
(including the tongue, lips, and teeth) and vocal chord resonance. We sub-
categorize them as follows:

  * •

Vocal Peripherals: The movements of teeth, lips, and tongue are crucial in
producing distinct speech sounds. Impeding the movement of these organs alters
speech patterns. E.g., speaking with a pen between the teeth, with the tongue
pressed to the palate, or with lips parted but stationary. Such challenges
restrict articulation of syllables, which hampers speech recognition [48].

  * •

Whisper: Whispering creates a unique sound characterized by minimal vibrations
of the vocal cords, leading to a toneless and less distinct output. Deepfakes
trained primarily on everyday speech might be opaque to this distortion and
fail to replicate its nuances. Furthermore, whispering makes even speaker
verification systems unreliable, as it can sound the same between two
individuals.

  * •

Vocal Cavity: Modifying the size of the vocal tract cavity impacts the
resonance in sound. Speaking while holding one’s nose results in a nasal
"buzzy" voice. In contrast, humming, which also narrows the vocal tract
cavity, was excluded from our consideration due to its constraints on word
articulation and lower audibility.

Waveform Challenges involve modifying key speech characteristics in the sound
waveform, including frequency (pitch), amplitude (loudness), and temporal
(speed). Deepfake generators focusing solely on speech content and speaker
identity may overlook these modulations.

  * •

Frequency: Voice pitch can vary, from speaking unusually low or high pitches
to singing melodies like "Happy Birthday." Singing, in particular, presents a
stark deviation from spoken speech and tests the waveform diversity an RTDF
generator can handle [15].

  * •

Amplitude: The ability to modulate speaking volume offers a natural test for
detecting RTDFs using amplitude variations.

  * •

Temporal: Abnormal speech rates may present challenges for RTDF generation
trained on average speaking speeds.

Language and Articulation Challenges involve subjects articulating language
patterns uncommon in everyday speech.

  * •

Difficult Sequences: Pronouncing phonetically complex or foreign words, like
‘Worcestershire,’ making guesses that lead to incorrect pronunciations and
filler words.

  * •

Mimicry: Users mimicking known accents vary their natural vocal style. This
variation can confound RTDF generators, which get trained on targeted
datasets. It would lack matching target data for such mimicry.

  * •

Phonetics: Rolling Rs or tongue clicks challenge the ability to reproduce
sounds not native to the speaker’s language. However, genuine speakers may
themselves struggle.

  * •

Deception: An unexpected command to cease speaking tests reaction times;
genuine speakers might pause or stop altogether. In contrast, an RTDF,
especially a text-to-speech (TTS) model, may not exhibit such reactive pauses.

Tone of Voice.

  * •

_Emotion:_ Speaking with a discernible emotion like happiness, sadness, or an
authoritative tone. Replicating the nuanced emotional expressions in speech is
a complex task for TTS systems.

  * •

_Phonology:_ Challenges in this category might include altering inflection,
particularly at the end of sentences, as in asking a question.

###  4.2 Non-lingual Challenges

Challenges that rely on noises created by humans, their environment, or speech
patterns unique to a user constitute this category.

Noise Challenges induce sounds from the speaker or their environment to create
artifacts in deepfakes. As these challenges are in-essence noises, deepfake
training datasets weed them out, making them vulnerable against them.

  * •

_Vocal Noises_ includes coughs, whistles, and loud breathing.

  * •

_Non-vocal Noises_ include clapping, snapping fingers, or manipulating the
microphone.

  * •

_Environmental Noises_ include background sounds like bird songs, distant
conversations, or traffic noise. Such sounds are less reliable for
authentication due to their variability and stochastic presence.

  * •

_Echo/Reverb:_ Involves creating feedback loops or reverberations during the
call, which can result in multiple overlaid voices and sounds. Disabling echo
cancellation‡‡‡https://support.zoom.us/hc/en-us/articles/115003279466-Using-
and-preserving-original-sound-in-a-meeting and looping back the received audio
can simulate this effect.

Behavioral Challenges (out-of-scope) utilize known attributes of an
individual’s speaking style or biometrics to verify their identity. We do not
explore this subcategory because this work does not assume identity
information access.

  * •

_Unique Speaking Habits:_ Individuals have distinctive ways of expressing
agreement or responding to questions (such as "yes," "agreed," or "sure").
Habit-based challenges involve prompting subjects to use their characteristic
phrases or responses, aiding in identity verification. However, they can be
unreliable due to dependence on individual behavior.

  * •

_Biometric Authentication:_ A voice sample is biometrically matched to the
caller’s speech, e.g., voice prints for biometric identity authentication
[13].

###  4.3 Replay Challenges

This category involves playing recordings that may disrupt deepfake creation.
They introduce noise elements that might be mistaken for the speaker’s voice,
potentially resulting in artifacts in deepfake output. However, their
effectiveness is limited in scenarios where deepfake systems can accurately
distinguish between background and speaker audio.

Playback Challenges play human-audible audio while the participant speaks.

  * •

_Speech:_ Subjects are instructed to speak while a recorded voice is speaking.
The resultant artifacts emerge as RTDF systems struggle to process multiple
speakers. If the imposter enables speech separation, this will reduce the
generation throughput.

  * •

_Music:_ Subjects are requested to play music while speaking. Lyrical tracks
lead to artifacts as the participant’s voice overlaps voices in audio. At the
same time, instrumental music introduces non-speech audio patterns,
challenging the system’s voice conversion process.

Passive Distortions (Out-of-Scope) challenges entail manipulating the caller’s
voice or playing adversarial audio to impede deepfake systems. These methods
presume a level of trust in the caller’s end and their equipment, which is
beyond the scope of this work, where we assume no trust.

  * •

_Software Editing:_ If a trusted software application can control the caller’s
equipment, it could apply audio transformations to the acquired speech before
it reaches the RTDF.

  * •

_Adversarial Audio:_ We propose using targeted audio samples designed as
adversarial perturbations against deepfake systems, primarily those utilizing
MEL-spectrogram images. ‘This concept aligns with existing research on
convolutional neural networks [1] and face-swapping deepfakes [17].

##  5 Evaluation

###  5.1 Collecting Original Recordings and Generating Deepfakes

Challenge Selection: We explore 20 challenges from the taxonomy and enumerate
them in Table 1. We requested a consenting participant to perform each task
with instructions detailed in §A.1.

Ethics and Recruitment. After getting approved by our IRB, we recruited
English speakers in the US using Prolific. The subjects consented to be video
and audio recorded while doing the tasks and were paid appropriately. We
stored the recorded data on a secure server designed specifically for high-
risk data. We will enforce ethical guard rails against indemnification of any
subject through a signed data release form. Both audio and video parts of the
data will be made available for academic research.

Study Design. We recorded subjects on both desktops and smartphone
environments. On the desktop environment, we explored 20 challenges, excluding
regular speech in challenge #0 or the no-challenge. For 19 out of 20
challenges, we requested each participant to read ten phonetically diverse
sentences per challenge (see Table 6). Challenges involving coughing or
whistling had only one instance. Thus, each successful participant, using a
desktop, recorded 201 samples.

We intended the study on the smartphone to simulate a person on the move,
potentially with less freedom to perform tasks. It, hence, was a strict subset
of the desktop. Specifically, we excluded the playback challenges (#18 - #22),
as they need a secondary device for playing the background sounds, which would
be an unrealistic assumption. Thus, each successful participant using a
smartphone recorded 171 samples.

We built the experiments on the Gorilla Experiment Builder. 77% subjects who
started the experiment also completed it, leading to the original data
collection of 100 subjects (actually 102, explained soon). The subjects were
split equally between smartphone and desktop environments, yielding 18,600
original video or audio recordings. In comparison, a prevalent speech dataset,
VCTK [44], contains 110 speakers and 400 sentence recordings, each with about
44,000 samples. Our dataset is comparable to it while being highly diverse.
Instruments used for creating the experiment are available at
https://app.gorilla.sc/openmaterials/722500.

Although this work only focuses on audio, we intentionally recorded videos to
aid in final compliance checks and prevent false rejections of submissions.
Not all audio samples have corresponding videos, as the video recording
instrument on Gorilla was in its development stage, so it worked only two-
thirds of the time.

Data Pre-processing. From the recorded samples (.webm), we strip the 48kHz
audio stream and down-sample them to 16kHz lossless audio (.wav) files for
downstream tasks. We use WhisperX [11] to transcribe the down-sampled audio
samples and obtain timestamps for speech segments. We concatenate the
segmented parts to clip the speech’s start and end and remove silent
intervals.

Compliance Check of Original Recordings: We recruit 45 human evaluators to
verify if the pre-processed audio recordings contain the required challenge.
To reduce the cost of evaluating all 18,600 recordings, we randomly sample one
audio recording per subject per challenge for a compliance check. The
assumption is that if a subject did the challenge once correctly, they could
do it multiple times and vice-versa. This results in a subset of 1,860
samples.

Each sample gets verified by three evaluators: a native English monolingual, a
non-native English multi-lingual, and an English-speaking tie-breaker. The
goal is to be more inclusive and reduce demographic bias. We made the
compliance decision using majority voting. We manually inspected videos of
eight low performers ($<70\%$ across challenges), revealing that two did not
comply. However, the other six were noisy cases (such as children’s sounds in
the background), which we consider essential for diversity. Hence, the final
dataset consists of 100 individual identities, with a mean compliance of 84%

Across challenges, we observe 89% mean compliance. Evaluators reported
challenges involving speaking (a) while holding nose (43%), (b) in a different
accent (73%), or (c) in a low pitch (74%) had low compliance. Similarly, video
inspection revealed that subjects with low compliance scores got confused
during the task and could not do an impromptu accent. On the other hand,
subjects did comply by holding their nose or speaking in a low pitch,
_suggesting that these challenges were problematic for human evaluators to
verify_.

Generating Deepfakes: To generate deepfakes, we consider each of our 100
participants as a target, with all others acting as imposters, resulting in
$100\times 99=9,900$ unique target-imposter pairs. Our challenges utilize one-
sentence ($4.5\pm 1.1$s) samples of both the target’s and imposter’s speech.
This setup intentionally varies the speech content between the target and the
imposter to reflect realistic constraints. We convert all collected samples
into deepfakes, generating a total of 1.6 million samples. Using SpeechBrain’s
speaker verification model [37], we verified 57% of regular speech deepfake
samples as the target speaker.

With the dataset of original and deepfakes in place, we evaluate them next
with machines and, subsequently, humans.

###  5.2 RQ1. Machine Evaluation

We begin evaluating the effectiveness of our framework against voice-cloning
attacks using an automated authentication system. An impersonator aiming to
convincingly mimic a target individual during a call must (1) adhere to the
challenge in terms of task and content, (2) maintain the target speaker’s
identity, and (3) achieve a high level of realism.

Measuring the performance of each challenge depends on how much disruption it
can bring in the above criteria. However, incorporating the criterion of
identity preservation into our machine evaluation renders the method
intrusive, necessitating a reference sample of the target’s voice. This
requirement would compromise the practicability of our approach, introducing a
significant constraint. Therefore, we omit the speaker identity check and
focus on evaluating audio samples against three non-intrusive criteria:

  1. 1.

_Compliance to the task_ : Did the sample contain the corresponding challenge
task?

  2. 2.

_Compliance to the Content_ : Did the sample contain the requested text?

  3. 3.

_Realism_ : What is the naturalness of the sample?

Table 2: Right: Average degradation score $\mathcal{D}$ for both fake and
original audio samples, along with the Area-under-Curve (AUC), averaged across
all samples pertinent to that challenge. Left:$\Delta$ values represent the
mean difference in each challenge’s performance relative to the baseline
(ground truth) in terms of challenge compliance, realism, and content
compliance. Challenges marked with gray labels indicate under-performance (AUC
$<70\%$). For the No-Challenge condition, we anticipated a low AUC.

# |  Challenge |  $\Delta$ Compliance |  $\Delta$ NISQA |  $\Delta$ Word Info Lost |  Degradation in Fakes |  Degradation in Originals |  AUC
---|---|---|---|---|---|---|---
0 |  No Challenge |  - |  8% |  11% |  11% |  5% |  56%
1 |  Static Mouth |  22% |  10% |  57% |  31% |  11% |  85%
2 |  Cup mouth |  87% |  11% |  38% |  46% |  9% |  97%
3 |  Whisper |  61% |  27% |  63% |  53% |  22% |  92%
4 |  Speak Loudly |  34% |  7% |  13% |  26% |  16% |  62%
5 |  Speak softly |  36% |  12% |  22% |  29% |  16% |  71%
6 |  Read quickly |  12% |  8% |  35% |  24% |  14% |  67%
7 |  Read Slowly |  19% |  11% |  13% |  18% |  20% |  55%
8 |  Hold nose |  55% |  11% |  32% |  29% |  38% |  63%
9 |  High Pitch |  63% |  9% |  28% |  44% |  17% |  86%
10 |  Low Pitch |  35% |  10% |  20% |  25% |  15% |  65%
11 |  Foreign Words |  - |  10% |  82% |  47% |  6% |  97%
12 |  Accent |  25% |  9% |  26% |  21% |  18% |  57%
13 |  Sing |  18% |  10% |  32% |  21% |  9% |  71%
14 |  Emotions |  30% |  9% |  25% |  35% |  10% |  82%
15 |  Question |  14% |  7% |  19% |  16% |  8% |  60%
16 |  Cough/Whistle |  36% |  26% |  - |  38% |  34% |  54%
17 |  Clap |  12% |  24% |  33% |  22% |  19% |  51%
18 |  Crosstalk |  27% |  17% |  65% |  38% |  13% |  90%
19 |  Instrumental Playback |  96% |  24% |  40% |  53% |  17% |  98%
20 |  Lyric Playback |  26% |  25% |  61% |  38% |  17% |  85%
|  Average |  37.3% |  13.6% |  35.8% |  31.7% |  15.9% |  74.4%
|  Average (top 11) |  46.6% |  14.9% |  46.6% |  39.5% |  13.4% |  86.7%

Challenge Compliance. This aspect is treated as a binary classification
problem, using the original speech data mentioned in §5.1. We train eighteen
classifiers, each tailored to a specific challenge, to differentiate between
audio samples of a challenge and regular speech samples (the latter referred
to as challenge #0). The dataset is partitioned by subject identity for 5-fold
cross-validation, utilizing 90 identities and reserving 10 for a final test
set. Classification employs a fine-tuned Wav2Vec2 model [10], achieving, on
average, 93.1% AUC and 83.6% accuracy on test sets (human evaluators had rated
compliance to be 86.0%). Although we split the dataset to follow machine-
learning protocols, we use the compliance detectors on all identities.
Notably, we do not use any deepfakes for training or validation.

For the remaining two out of 20 challenges, involving speaking foreign words
(challenge #11) and asking questions (challenge #17), compliance is assessed
through speech transcription, as detailed in Table 5.

Content Compliance. We use Word Information Lost (WIL, [33]) for quantifying
the loss in transcribed speech content compared to the original script. We
preferred WIL over the standard Word Error Rate (WER), as it emphasizes
keywords while de-emphasizing fillers. This approach mirrors actual
conversations, where communication remains clear despite potential omissions
or errors in non-essential words.

All challenges have spoken content, except the one involving coughing or
whistling.

Realism. We utilize Non-Intrusive Speech Quality Assessment (NISQA, [31]) to
score realism. As a non-intrusive, deep-learning-based solution, NISQA
measures speech quality on multiple dimensions without requiring a reference
signal, aligning with general speech research applications. It predicts a
five-point mean opinion score (pMOS) akin to human evaluation for a given
audio sample.

A degradation score $\mathcal{D}~{}\in~{}[0,1]$ for an audio sample $x$ and
challenge $chal$ is defined, combining the three metrics. A higher score
indicates poorer deepfake performance, aligning with our challenge’s
objectives. We calculate the score as an average of the compliance detector
$C_{chal}(x)$, content verification WIL$(x)$, and realism NISQA$(x)$:

$\mathcal{D}_{chal}(x)=\frac{1}{3}\left[(1-C_{chal}(x))+WIL(x)+\left(1-\frac{\text{NISQA}(x)}{5}\right)\right]$

where $C_{chal}:\text{audio}\rightarrow\\{0,1\\}$ indicates compliance with
the challenge task, returning 1 for compliance and 0 otherwise.

Figure 1: Boxplot depicting the degradation scores assigned by machines for
both fake and original audio samples. Samples are arranged in ascending order
based on the median values of the fakes (indicated by white bars), with
outliers omitted. Higher scores indicate higher challenge performance (or
greater degradation). Challenges where the model under-performs are marked
with gray labels, correlating with the data in Table 2.

We assessed the degradation score $\mathcal{D}$ across 1.6 million fake and
18,600 original audio samples. Detailed in Table 2, our analysis spans task
compliance, realism, and word information loss across various challenges,
showcasing how fake and original samples diverge regarding degradation. Due to
the significant imbalance between fake and original samples in our automated
evaluation dataset, we omit accuracy. Instead, Figure 1 visually represents
the variance in machine-scored degradation scores across challenges.

Observations. Without specific auditory challenges, the observed degradation
was relatively low – 11% for deepfakes and 5% for original speech – indicating
that RTDFs can effectively replicate standard speech patterns.

Upon introducing auditory challenges, we observe a notable increase in
degradation for both deepfakes and original recordings, with deepfakes showing
more substantial declines in quality. This pattern was especially pronounced
in tasks involving foreign words, static mouth, and cross-talk, which resulted
in considerable word information loss. Furthermore, deepfakes struggled to
maintain compliance in scenarios requiring lyrical playback, cupping the
mouth, whispering, and producing a high-pitched voice, indicating a critical
vulnerability of current deepfake technologies. Lyrical playback, in
particular, introduced measurable vocal distortions, highlighting the
effectiveness of these challenges in compromising the fidelity of deepfake
audio.

Our exploration of 20 unique challenges revealed nine of them to be less
effective, with AUC scores falling below 70%. These less effective challenges
included variations in accent, reading pace (both slow and quick), volume
(specifically speaking loudly), pitch (notably low pitch), and specific
actions like clapping, coughing/whistling, and holding the nose. The deepfake
algorithms were notably proficient in mimicking tasks related to regular
speech patterns, such as the first six mentioned, demonstrating the
sophistication of these systems in replicating human vocal characteristics.
The challenge involving holding the nose displayed a low compliance rate from
participants (43%), contributing to its limited effectiveness.

The clap challenge, where participants clapped while speaking, revealed that
deepfakes transformed clap sounds into noisy peaks, potentially leading to
incorrect detections. This revelation indicates that despite some success in
distorting deepfake outputs, there is ample scope for further refinement.

The cough/whistle challenge was ultimately excluded from the strong challenge
set due to its lack of support for content randomization, as such challenges
are devoid of spoken content. Without randomization, an imposter can play a
pre-computed clean recording of the task to the defender to bypass detection.

Our analysis further delved into the effects of frequency modulation,
particularly comparing high-pitch and low-pitch challenges. We observed that
high-pitch challenges induced more significant degradation (44%) in deepfake
outputs than their low-pitch counterparts (25%). The observed discrepancy in
machine detection efficiency between high and low-pitch challenges points to
limitations in the training datasets of deepfake models, such as the VCTK
corpus, which lacks a diverse range of high-pitch voice samples. This
limitation emphasizes dataset diversity’s critical role in enhancing deepfake
quality.

Despite the underperformance of specific tasks, we included them in our
analysis for a comprehensive evaluation and to facilitate comparison against
Yasur et al. [46]. The authors identified six out of the nine ineffective
tasks as beneficial against an RTDF model named StarGANv2-VC [27]. However,
these tasks fell short when tested against the more sophisticated FREE-VC
model [25], suggesting that our remaining 11 challenges offer a higher level
of degradation even against a more advanced adversary.

Takeaways. We developed a comprehensive and robust machine evaluation
framework and identified 11 tasks that can effectively serve as challenges.
The challenges enable machines to achieve an AUC of 86.7%, in comparison to
56.0% for no-challenge. These findings highlight the vital role of challenges
in identifying the strengths and limitations of deepfakes, particularly when
such tasks involve complex auditory challenges that extend beyond simple
speech imitation.

Figure 2: Boxplot of human degradation scores for Vanilla Human Evaluation,
with outliers removed. The plot order is sorted by the fake boxplot’s median
(white bar). Corresponding AUC is in parenthesis, with a mean AUC of 73.7%
across 21 challenges and 80.1% for the selected 12 challenges.

###  5.3 Human Evaluation

Having established that challenges enable machine evaluations, we question
whether similar results hold for humans. Support for human assessment becomes
essential when advanced deepfake detection tools are inaccessible,
interpretability is desirable, or both. In this subsection, we describe
details of a vanilla human evaluation we conducted and extended it with two
machine-assisted evaluations.

Dataset Selection. As the manual evaluation of 1.6 million audio files is
impractical, we focused on a subset of the data that could present the most
difficulty to human evaluators.

The full dataset has 9,900 unique imposter-target speaker pairs. We utilized
the SpeechBrain speaker recognition framework [37] to compute each pair’s
matching score ($\in[-1,1]$) while keeping the same speech content between the
pair samples. We sub-selected a unique set of pairs that were false positives.
All pairs obtained matching scores above 0.50, indicating a high likelihood of
the pair being considered the same speaker well beyond the usual threshold of
0.25. We selected 147 high-quality samples per challenge from this subset,
each with a $pMOS$ of $4.50\pm 0.25$, yielding a balanced dataset of 6,174
audio samples for human assessment.

####  5.3.1 RQ2. Vanilla Human Evaluation

Our human evaluation process mirrored the compliance check methodology for
original recordings outlined in §5.1. We recruited 63 evaluators through
Prolific, categorized into three groups: native English monolinguals, non-
native English multilinguals, and English-speaking tie-breakers.

We tasked evaluators with assessing each audio sample’s compliance to the
given task ($\mathcal{C}$) and its overall quality ($\mathcal{Q}$) on a
5-point scale, following the ITU P.808 standard for subjective audio quality
assessments. The human-scored degradation ($\mathcal{H}$) for an audio sample
$x$ is calculated as follows:

$\mathcal{H}(x)=1-\min(\mathcal{C}(x),\mathcal{Q}(x)/5).$

Here, $\mathcal{C}:x\rightarrow\\{0,1\\}$ gives 1 if compliant and 0
otherwise. This formula assigns either a maximum degradation score to a sample
for compliance failure or a score drop using the human-rated quality. We
determine the final score for each sample through majority voting among these
evaluators. Materials and instruments for this evaluation are available for
preview at https://app.gorilla.sc/openmaterials/769577.

Fig. 2 displays the distribution of human scores across 21 prospective
challenges and the corresponding AUC for each challenge. We ordered the
boxplots by the median degradation observed in the fake dataset.

Humans vs Machines. Human evaluators achieved an AUC of 74.1% in detecting
deepfakes across 21 challenges. This performance is comparable to machine
evaluations (74.4%), albeit conducted on a challenging, high-quality dataset.

A comparison between Fig. 2 and 4 reveals a shift in the ranking of
challenges, with human scores generally higher than those obtained by
machines. Moreover, the correlation between human and machine AUC scores, at
0.476, suggests a moderate positive relationship. These observations revealed
that while humans and machines align on particular challenges, they provide
complementary strengths.

Humans demonstrated superior performance over machines in challenges that
involved Speaking Loudly, Low Pitch, Cough/Whistle, and Clapping. Conversely,
they were less effective in challenges associated with Static Mouth, Foreign
Words, Singing, and Emotion Display, pointing to a human sensitivity gap in
detecting subtleties within these challenges.

Although the disparity between human and machine performance appears nominal,
each exhibits distinct advantages and limitations. This gap becomes more
pronounced when focusing on the 11 machine-validated effective challenges.
Here, machine evaluations achieved an AUC of 86.7% in deepfake detection,
compared to 80.1% by humans. The difference raises the prospect of combining
human and machine efforts to enhance overall performance in deepfake
detection.

####  5.3.2 RQ3. Machine-Supported Human Evaluation

We perform additional human evaluations to explore the potential of machine
assistance in enhancing human capability to detect deepfakes. We focused this
investigation on the 12 challenges (including no-challenge) where we deemed
machine performance satisfactory, as detailed in §5.2.

For this evaluation, we selected a subset of 84 samples from the high-quality
dataset rather than the original 147. This approach allowed us to maximize our
resources and recruit a larger pool of evaluators, resulting in a balanced
dataset comprising 2,016 deepfake and original audio samples.

Study Design. We recruited a new, gender-balanced cohort of 100 human
evaluators from the Prolific platform. The recruitment process included an
orientation on the challenges the participants will encounter, along with
specific examples for each. We used a preliminary quiz based on original
challenge samples to screen participants, with 91 passing.

We informed the participants about the task and emphasized that the machine’s
verdict has 85% accuracy (computed by choosing $\mathcal{D}_{chal}=0.25$ as
the threshold for all challenges). We ask each participant to listen to an
audio sample and determine its authenticity ("Real" or "Fake"), as well as
their confidence level in this initial decision. After making their initial
judgment, the machine’s verdict was disclosed to the participant, sometimes
accompanied by a brief explanation. Additionally, we based the explanations on
one of three criteria we had established in §5.2: the failure to perform the
specified task, a mismatch in spoken text, or the presence of vocal
distortions.

After participants received the machine’s verdict and its rationale, we gave
them the option to revise their decision and confidence level or to stick with
their initial assessment. This method generated a total of 8,607 responses.
Instruments are available to preview at
https://app.gorilla.sc/openmaterials/769578.

We present the outcomes in the left half of Table 4, analyzing initial
judgments, machine-assisted adjustments, and the resulting accuracy changes
across the challenges. Table 3 further breaks down the responses across the
eight possible scenarios, noting any changes in confidence levels.

Table 3: Effect of Machine Predictions on Initial Human Decisions under all
permutations. A participant makes an initial decision, acknowledges the
machine’s verdict, and makes their final decision. ✓implies correct
prediction, and ✗implies otherwise. The last column indicates the mean and
standard deviation of change in human confidence.

Scenario |  I |  M |  F |  Count |  $\Delta$Conf.(%)
---|---|---|---|---|---
Correct Agreement |  ✓ |  ✓ |  ✓ |  5462 |  $+8.1\pm 15.8$
Machine Corrected |  ✗ |  ✓ |  ✓ |  781 |  $-10.0\pm 12.6$
Self Corrected |  ✗ |  ✗ |  ✓ |  5 |  $-9.9\pm 20.0$
No Change (Initial Correct) |  ✓ |  ✗ |  ✓ |  515 |  $-5.4\pm 18.9$
Self Misled |  ✓ |  ✓ |  ✗ |  27 |  $-6.9\pm 19.2$
Machine Misled |  ✓ |  ✗ |  ✗ |  216 |  $-8.6\pm 17.1$
No Change (Initial Wrong) |  ✗ |  ✓ |  ✗ |  1039 |  $-2.2\pm 17.6$
Incorrect Agreement |  ✗ |  ✗ |  ✗ |  562 |  $+8.8\pm 13.0$
I = Initial Decision, M = Machine Prediction, F = Final Decision

Table 4: Human Evaluation with AI assistance and collaboration.

Challenge |  Initial Accuracy |  Assisted Accuracy |  Increase |  Human-AI Collab. |  Increase
---|---|---|---|---|---
No Challenge |  52.2 |  66.7 |  $\uparrow$27.7% |  88.4 |  $\uparrow$69.7%
Static Mouth |  58.9 |  68.8 |  $\uparrow$16.8% |  71.1 |  $\uparrow$20.9%
Cup mouth |  85.0 |  91.2 |  $\uparrow$7.4% |  93.9 |  $\uparrow$10.6%
Whisper |  69.6 |  78.1 |  $\uparrow$12.2% |  81.7 |  $\uparrow$17.9%
Speak softly |  61.8 |  63.4 |  $\uparrow$2.5% |  66.5 |  $\uparrow$7.5%
High Pitch |  84.5 |  90.5 |  $\uparrow$7.1% |  90.6 |  $\uparrow$7.1%
Foreign Words |  58.5 |  71.3 |  $\uparrow$21.9% |  85.4 |  $\uparrow$46.3%
Sing |  68.7 |  68.6 |  $\downarrow$0.2% |  68.8 |  $\uparrow$0.4%
Emotions |  68.3 |  74.0 |  $\uparrow$8.4% |  77.3 |  $\uparrow$12.3%
Crosstalk |  80.2 |  85.1 |  $\uparrow$6.1% |  86.3 |  $\uparrow$7.2%
Instr. Playback |  89.2 |  94.6 |  $\uparrow$5.9% |  95.1 |  $\uparrow$6.3%
Lyric Playback |  90.2 |  90.7 |  $\uparrow$0.5% |  89.1 |  $\downarrow$1.2%
Average |  72.3 |  78.5 |  $\uparrow$8.5% |  82.9 |  $\uparrow$14.3%

Machine-Assisting Humans. Initial decisions achieved a 72.3% accuracy rate,
while updated decisions improved to 78.3%, with machine assistance enhancing
the original accuracy by 8.5% overall.

In 11 out of 12 challenges, machine predictions increased the average human
accuracy. This effect was more noticeable in nuanced challenges such as
Static-Mouth, Cupping-Mouth, and Whisper and less effective in scenarios like
No-Challenge and Foreign Words.

When machine predictions aligned with human decisions, they increased human
confidence, regardless of the correctness. Conversely, when predictions
diverged, they reduced human confidence. Notably, when humans initially erred
and machines provided the correct guidance, humans changed their decisions 43%
of the time. However, when humans were correct and machines erred, the latter
misled humans 29% of the time. This disparity highlights the beneficial role
of machines in correcting human errors and their relatively lesser impact in
misleading humans when they are correct.

When comparing the presence of rationale against providing no rationale, we
observe no significant difference between humans accepting that decision and
changing their mistakes (42.4% vs 43.0%).

Figure 3: Tradeoff between mean deepfake detection accuracy and proportion of
human decisions considered. AI (or machines) overrides human decisions if it
has relatively higher confidence. Temperature calibrates the confidence of AI
predictions, with lower temperatures overriding more number human predictions.
A star represents the point of intersection, denoting an equilibrium between a
number of human predictions and accuracy.

Humans-AI Collaboration. The machine-assistance framework kept machines as a
warning signal and prevented them from making unilateral final decisions. This
framework necessitates convincing humans to correct their initial judgments
based on the machine’s input, which could be cumbersome. To bypass this, we
introduced a mechanism allowing the machine to override human decisions when
it exhibited higher confidence (and not correctness of decision).

We computed the machine confidence for each audio sample’s decision by
utilizing the machine scores from §5.2. The machine confidence was the
normalized difference between a predefined threshold ($\tau=0.25$) and the
sample’s score. Despite machines demonstrating superior accuracy (85.1%), they
often exhibited lower confidence than humans. We counteracted this by scaling
the machine’s confidence using a temperature parameter.

Fig. 3 illustrates the tradeoff between retaining human decisions post-machine
intervention and the mean accuracy achieved, and the right half of Table 4
details the accuracy breakdown. At the intersection point in Fig. 3, a balance
of 56% human to 44% machine decisions was observed, which achieved an accuracy
of 82.9%. This balance indicates that a synergistic approach between humans
and AI can significantly enhance accuracy while ensuring that humans retain
decision-making authority in the majority of cases.

Takeaways. Challenges enable human capabilities to detect deepfakes audios,
similar to machines, even on a difficult dataset. Comparing human detection to
machines reveals both similarities and differences depending on the challenge
type, suggesting a complementary relationship between human intuition and
machine precision. When combined, machine tagging aided in assisting humans,
where their performance was lacking. Further collaboration between humans and
machines resulted in an even further boost in deepfake detection performance.

##  6 Discussion

##### Practical Design Recommendations.

To ensure a practical implementation of the tagging system, we offer the
following guidelines:

_On the caller’s side:_

  * •

Use single, succinct sentences for tasks.

  * •

Enhance compliance with high-quality sample audio.

  * •

Avoid challenges that confuse both machines and humans, such as those
involving human cognition tests or deception.

  * •

For playback, utilize an automated sound for ease of use.

_On the receiver’s side:_

  * •

Employ two categories of tags: "Deepfake-Likely" and "Deepfake-Certainly," to
indicate machine confidence.

  * •

Providing a rationale is not necessary.

Figure 4: Boxplot of user ratings for the 12 valid challenges.

##### Usability.

For any human-centered security system, usability is paramount. While
researchers aim to make their systems as secure as possible, such efforts
would only be practical if they considered usability.

In Fig. 4, we present the usability ratings of the 12 valid challenges as
rated by participants who contributed to our audio-video dataset. Our findings
indicate a known trend that challenges with lower usability scores, such as
Playbacks and Emotions, lead to significant degradation in deepfakes.
Interestingly, moderately usable challenges, such as Whispering, High-Pitch,
and Foreign Words, also induce comparable levels of degradation. The
observation suggests that more preferred challenges could be prioritized for
deployment, resorting to less preferred ones only if necessary.

We acknowledge that our proposed challenge-response framework has manual
components and requires callers to actively engage in a task before being
authenticated. However, our data shows that a caller typically takes a mere 6
seconds to speak out a sentence and finish the challenge.

On the defender’s (receiver’s) side, there is a desire for quick verification
of callers before proceeding with the conversation. To accommodate this, we
employ AI as a tagging system to aid the defender in making confident
decisions. The design decision to keep humans in control acknowledges the
fallibility of AI and the nature of telephone communications, where a call,
unlike spam emails, cannot be retrieved once disconnected.

##### Robustness against Countermeasures.

An adaptive adversary can design countermeasures against our approach. Such
countermeasures include the possibility of adversaries pre-empting challenges
with prepared responses or enhancing deepfake generation techniques to
circumvent detection.

Given their public nature, adversaries might anticipate our challenges and
prepare responses in advance, using sophisticated editing tools to hide
detectable anomalies. To prevent an adversary from preparing responses pre-
call, our challenges feature randomizable elements, such as varying speech and
background noise, introduced at the call’s outset. This element of randomness
necessitates real-time engagement from adversaries, thwarting the use of pre-
recorded responses.

Furthermore, as deepfake technology advances, it may develop the capability to
bypass our current challenges. Anticipating this, we can employ a sequence of
challenges tailored to the specific security requirements, strengthening our
defense, alike current CAPTCHA implementations. Our system’s malleability also
allows for the seamless incorporation of new challenges to stay ahead of
technological advancements in deepfake generation.

Therefore, while adversaries might develop countermeasures to our challenge-
response system, the inherent design features of our system—such as its
randomness and adaptability—are aimed at preserving its effectiveness against
both current and future threats, thereby ensuring its continued robustness
against evolving deepfake technologies and adaptive adversaries.

##  7 Limitations and Future Work

Compounding Challenges: Future work should explore compounding challenges,
where multiple challenges are executed simultaneously rather than
sequentially, to enhance robustness without increasing complexity.

Data-rich Scenarios: Further research is needed to assess detection efficacy
in environments with extensive adversary access to data, testing the system’s
resilience against more sophisticated deepfakes.

Unified Challenge Detectors: Developing a unified detector for all challenges
could streamline deployment and improve efficiency, moving away from separate
detectors for each challenge type.

##  8 Conclusion

In this study, we design a challenge-response approach to enhance humans’ and
machines’ detection of interactive deepfake audio calls. We developed a
comprehensive taxonomy of audio challenges, from which we selected 11
particularly effective instances out of an initial set of 20. These challenges
were instrumental in degrading the quality of deepfakes, thereby enhancing
machines in their detection tasks and enabling humans to discern deepfakes,
even under a hard dataset effectively. By fostering a collaborative dynamic
between humans and machines, we achieved a balance that leverages the
strengths of both—human oversight and machine efficiency. This complementary
strength has laid the groundwork for an innovative tagging system aimed at
pre-screening calls, marking a significant advancement in audio deepfake
detection independent of the target identity.

## Acknowledgements

This work was supported by the National Science Foundation under SaTC grant
number 2154119 and by Google Research under the Cyber NYC Program.

## References

  * [1] Adversarial Perturbations of Deep Neural Networks, pages 311–342.  2017\.
  * [2] Caller id spoofing: How to spot and avoid spoofed calls.  Norton Blog, 2023.  [Accessed: 23-Nov-2023].
  * [3] How does biometrics voice recognition work?  KYCAML Guide Blog, Jan 2023.  Accessed: 2024-02-15.
  * [4] Voice deepfakes are coming for your bank balance.  The New York Times, Aug 2023.  Accessed: 2024-02-15.
  * [5] Will generative ai kill kyc authentication?  CSO Online, Oct 2023.  Accessed: 2024-02-15.
  * [6] https://www.cnn.com/2024/02/04/asia/deepfake-cfo-scam-hong-kong-intl-hnk/index.html, Feb 2024.  Accessed: 2024-02-15.
  * [7] Fake joe biden robocall tells new hampshire democrats not to vote tuesday.  https://www.nbcnews.com/politics/2024-election/fake-joe-biden-robocall-tells-new-hampshire-democrats-not-vote-tuesday-rcna134984, Feb 2024.  Accessed: 2024-02-15.
  * [8] APNews.  Can New York’s mayor speak Mandarin? No, but with AI he’s making robocalls in different languages.  https://apnews.com/article/nyc-mayor-ai-robocalls-foreign-languages-30517885466994e5f1f54745c08691e0.  [Accessed: 23-Nov-2023].
  * [9] Matthew Baas, Benjamin van Niekerk, and Herman Kamper.  Voice conversion with just nearest neighbors.  In Interspeech, 2023.
  * [10] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  CoRR, abs/2006.11477, 2020.
  * [11] Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman.  Whisperx: Time-accurate speech transcription of long-form audio.  INTERSPEECH 2023, 2023.
  * [12] James Betker.  Better speech synthesis through scaling.  arXiv preprint arXiv:2305.07243, 2023.
  * [13] Andrew Boles and Paul Rad.  Voice biometrics: Deep learning-based voiceprint authentication system.  In 2017 12th System of Systems Engineering Conference (SoSE), pages 1–6. IEEE, 2017.
  * [14] Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al.  Audiolm: a language modeling approach to audio generation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [15] Anurag Chowdhury, Austin Cozzo, and Arun Ross.  Domain adaptation for speaker recognition in singing and spoken voice.  In ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7192–7196, 2022.
  * [16] CNN.  ’Mom, these bad men have me’: She believes scammers cloned her daughter’s voice in a fake kidnapping.  Online, 2023.  [Accessed: 23-Nov-2023].
  * [17] Junhao Dong, Yuan Wang, Jianhuang Lai, and Xiaohua Xie.  Restricted black-box adversarial attack against deepfake face swapping.  IEEE Transactions on Information Forensics and Security, 2023.
  * [18] Xu Tan Rongjie Huang Songxiang Liu Xuankai Chang Jiatong Shi Sheng Zhao Jiang Bian Xixin Wu Zhou Zhao Helen Meng Dongchao Yang, Jinchuan Tian.  Uniaudio: An audio foundation model toward universal audio generation.  arXiv preprint arXiv:2310.00704, 2023.
  * [19] Valerie Fanelle, Sepideh Karimi, Aditi Shah, Bharath Subramanian, and Sauvik Das.  Blind and human: Exploring more usable audio $\\{$CAPTCHA$\\}$ designs.  In Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020), pages 111–125, 2020.
  * [20] Federal Communications Commission.  Fcc makes ai-generated voices in robocalls illegal.  https://www.fcc.gov/document/fcc-makes-ai-generated-voices-robocalls-illegal, Feb 2024.  Accessed: 2024-02-15.
  * [21] Internet Crime Complaint Center (IC3).  Malicious Actors Almost Certainly Will Leverage Synthetic Content for Cyber and Foreign Influence Operations.  Online, 2021.  [Accessed: 23-Nov-2023].
  * [22] Wall Street Journal.  Fraudsters used ai to mimic ceo’s voice in unusual cybercrime case.  https://www.wsj.com/articles/fraudsters-use-ai-to-mimic-ceos-voice-in-unusual-cybercrime-case-11567157402, 2019\.  [Accessed: 23-Nov-2023].
  * [23] Pavel Korshunov, Haolin Chen, Philip N Garner, and Sebastien Marcel.  Vulnerability of automatic identity recognition to audio-visual deepfakes.  2023\.
  * [24] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi.  Audiogen: Textually guided audio generation.  In The Eleventh International Conference on Learning Representations, 2023.
  * [25] Jingyi Li, Weiping Tu, and Li Xiao.  Freevc: Towards high-quality text-free one-shot voice conversion.  In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023.
  * [26] Yinghao Aaron Li, Cong Han, Vinay S. Raghavan, Gavin Mischler, and Nima Mesgarani.  Styletts 2: Towards human-level text-to-speech through style diffusion and adversarial training with large speech language models.  In Advances in Neural Information Processing Systems, 2023.
  * [27] Yinghao Aaron Li, Ali Zare, and Nima Mesgarani.  StarGANv2-VC: A Diverse, Unsupervised, Non-Parallel Framework for Natural-Sounding Voice Conversion.  In Proc. Interspeech 2021, pages 1349–1353, 2021.
  * [28] Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, and Helen Meng.  Any-to-many voice conversion with location-relative sequence-to-sequence modeling.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1717–1728, 2021.
  * [29] McAfee.  Beware the Artificial Impostor.  https://www.mcafee.com/content/dam/consumer/en-us/resources/cybersecurity/artificial-intelligence/rp-beware-the-artificial-impostor-report.pdf.  [Accessed: 23-Nov-2023].
  * [30] Harry McGurk and John MacDonald.  Hearing lips and seeing voices.  Nature, 264(5588):746–748, 1976.
  * [31] Gabriel Mittag, Babak Naderi, Assmaa Chehadi, and Sebastian Möller.  Nisqa: A deep cnn-self-attention model for multidimensional speech quality prediction with crowdsourced datasets.  In Interspeech 2021. ISCA, 2021.
  * [32] Govind Mittal, Chinmay Hegde, and Nasir Memon.  Gotcha: A challenge-response system for real-time deepfake detection.  arXiv preprint arXiv:2210.06186, 2022.
  * [33] Andrew Morris, Viktoria Maier, and Phil Green.  From wer and ril to mer and wil: improved evaluation measures for connected speech recognition.  10 2004.
  * [34] Nicolas M Müller, Karla Pizzi, and Jennifer Williams.  Human perception of audio deepfakes.  In Proceedings of the 1st International Workshop on Deepfake Detection for Audio Multimedia, pages 85–91, 2022.
  * [35] Andreas Nautsch, Xin Wang, Nicholas Evans, Tomi H Kinnunen, Ville Vestman, Massimiliano Todisco, Héctor Delgado, Md Sahidullah, Junichi Yamagishi, and Kong Aik Lee.  Asvspoof 2019: spoofing countermeasures for the detection of synthesized, converted and replayed speech.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(2):252–265, 2021.
  * [36] NPR.  That panicky call from a relative? it could be a thief using a voice clone, ftc warns.  https://www.gpb.org/news/2023/03/22/panicky-call-relative-it-could-be-thief-using-voice-clone-ftc-warns, 2023\.  [Accessed: 23-Nov-2023].
  * [37] Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio.  SpeechBrain: A general-purpose speech toolkit, 2021.  arXiv:2106.04624.
  * [38] Andrew Rosenberg and Bhuvana Ramabhadran.  Bias and statistical significance in evaluating speech synthesis with mean opinion scores.  In Interspeech, pages 3976–3980, 2017.
  * [39] suno-ai.  Bark: Text-Prompted Generative Audio Model.  https://github.com/suno-ai/bark, 2023.  Accessed: 2024-02-15.
  * [40] Hemlata Tak, Jose Patino, Massimiliano Todisco, Andreas Nautsch, Nicholas Evans, and Anthony Larcher.  End-to-end anti-spoofing with rawnet2.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6369–6373. IEEE, 2021.
  * [41] Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al.  Neural codec language models are zero-shot text to speech synthesizers.  arXiv preprint arXiv:2301.02111, 2023.
  * [42] Emily Wenger, Max Bronckers, Christian Cianfarani, Jenna Cryan, Angela Sha, Haitao Zheng, and Ben Y. Zhao.  "hello, it’s me": Deep learning-based speech synthesis attacks in the real world.  In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS ’21, page 235–251, New York, NY, USA, 2021. Association for Computing Machinery.
  * [43] Yihan Wu, Xu Tan, Bohan Li, Lei He, Sheng Zhao, Ruihua Song, Tao Qin, and Tie-Yan Liu.  AdaSpeech 4: Adaptive Text to Speech in Zero-Shot Scenarios.  In Proc. Interspeech 2022, pages 2568–2572, 2022.
  * [44] Junichi Yamagishi, Christophe Veaux, Kirsten MacDonald, et al.  Cstr vctk corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92).  University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2019.
  * [45] Yang Yang, Yury Kartynnik, Pen Li, Jiuqiang Tang, Xing Li, George Sung, and Matthias Grundmann.  Streamvc: Real-time low-latency voice conversion.  2024\.
  * [46] Lior Yasur, Guy Frankovits, Fred M. Grabovski, and Yisroel Mirsky.  Deepfake captcha: A method for preventing fake calls.  In Proceedings of the 2023 ACM Asia Conference on Computer and Communications Security, ASIA CCS ’23, page 608–622, New York, NY, USA, 2023\. Association for Computing Machinery.
  * [47] Jiangyan Yi, Chenglong Wang, Jianhua Tao, Xiaohui Zhang, Chu Yuan Zhang, and Yan Zhao.  Audio deepfake detection: A survey.  arXiv preprint arXiv:2308.14970, 2023.
  * [48] Linghan Zhang, Sheng Tan, Yingying Chen, and Jie Yang.  A phoneme localization based liveness detection for text-independent speaker verification.  IEEE Transactions on Mobile Computing, pages 1–14, 2022.

## Appendix

###  A.1 Data Collection Instructions

Thus, we request a consenting participant to perform the following tasks:

  1. 0.

Speak the sentence normally.

_Vocal Distortions_

  2. 1.

Keep their mouth and lips static, and speak the sentence.

  3. 2.

Cup your mouth and Speak the sentence.

  4. 3.

Whisper the sentence.

_Waveform Manipulation_

  5. 4.

Speak the sentence loudly.

  6. 5.

Speak the sentence softly.

  7. 6.

Read out the sentence quickly.

  8. 7.

Read out the sentence slowly.

  9. 8.

Speak the sentence in a high pitch.

  10. 9.

Speak the sentence in a low pitch.

  11. 10.

Hold their nose and read the sentence.

_Language and Articulation_

  12. 11.

Speak a transliteration of a foreign sentence.

  13. 12.

Speak the sentence with a known accent, such as Russian or British.

  14. 13.

Sing the sentence.

_Tone of Voice_

  15. 14.

Speak the sentence with happiness or excitement.

  16. 15.

Speak the question.

_Non-Lingual Noises_

  17. 16.

Cough or Whistle.

  18. 17.

Speak the sentence while clapping.

_Playback Challenges:
(done while playing audio on a smartphone)_

  19. 18.

Speak the sentence clearly amidst cross-talk; while playing recording of a
matching sentence.

  20. 19.

Speak the sentence while playing instrumental music.

  21. 20.

Speak the sentence while playing lyrical music.

Figure 5: Histogram of compliance across participants. Figure 6: Compliance across original challenge recordings. Table 5: Performance of Challenge Compliance Detectors. All accuracies and area-under-curve numbers are in percentage points. Test accuracy is on the 10 heldout identities, while validation is across 5 trained models during 5-fold cross validation on the corresponding folds. Compare the performance with the compliance numbers from human evaluation. Index | Challenge |  Human | Compliance Check using Finetuned Wav2Vec2
---|---|---|---
|  |  Baseline | Val Acc. | Test Acc. | AUC | Precision | Recall | F1
1 | Static Mouth |  85.9 | $89.8\pm 3.1$ | 88.2 | 96.6 | 0.89 | 0.87 | 0.88
2 | Cup mouth |  86.9 | $92.0\pm 2.0$ | 89.5 | 97.1 | 0.95 | 0.84 | 0.89
3 | Whisper |  94.9 | $92.9\pm 2.4$ | 90.9 | 93.1 | 0.92 | 0.9 | 0.91
4 | Speak Loudly |  86.9 | $76.1\pm 3.8$ | 70.5 | 74.8 | 0.73 | 0.65 | 0.69
5 | Speak softly |  85.9 | $80.7\pm 3.8$ | 79.1 | 84.7 | 0.76 | 0.85 | 0.8
6 | Read quickly |  98.0 | $86.8\pm 2.6$ | 83.2 | 90.7 | 0.92 | 0.73 | 0.81
7 | Read Slowly |  89.9 | $80.2\pm 1.1$ | 79.1 | 87.5 | 0.8 | 0.77 | 0.79
8* | Hold nose |  43.4 | $83.6\pm 3.6$ | 90.0 | 92.8 | 0.94 | 0.85 | 0.90
9* | High Pitch |  86.9 | $76.1\pm 3.8$ | 81.8 | 91.2 | 0.95 | 0.67 | 0.79
10 | Low Pitch |  73.7 | $77.1\pm 3.9$ | 77.3 | 85.9 | 0.75 | 0.83 | 0.78
13 | Accent |  72.7 | $77.9\pm 2.2$ | 79.5 | 85.9 | 0.81 | 0.77 | 0.79
14 | Sing |  87.9 | $90.0\pm 2.0$ | 90.0 | 96.8 | 0.89 | 0.91 | 0.9
16 | Emotions |  89.9 | $86.2\pm 3.7$ | 88.6 | 95.6 | 0.98 | 0.79 | 0.87
18 | Cough/Whistle |  90.9 | $96.3\pm 1.8$ | 96.7 | 99.3 | 0.82 | 0.82 | 0.82
19 | Clap |  84.7 | $92.1\pm 2.8$ | 94.1 | 98.8 | 0.95 | 0.93 | 0.94
20 | Cross-talk |  91.5 | $94.1\pm 2.5$ | 98.8 | 100.0 | 1.0 | 0.96 | 0.98
21 | Instr. Playback |  95.7 | $95.2\pm 3.7$ | 98.1 | 99.6 | 0.98 | 0.96 | 0.97
22 | Lyric Playback |  100.0 | $95.9\pm 2.3$ | 98.1 | 100.0 | 0.94 | 1.0 | 0.97
| Average (across all) |  86.0 | 87.5 | 83.6 | 93.1 | 0.89 | 0.82 | 0.85
|  |  | Heuristic-based Compliance Check |
12 | Foreign Words |  99.0 | WER compared to the specific foreign script used. |
17 | Question |  92.9 | Transcribed sentence has a question mark |
Table 6: Scripts used for all tasks General Sentences
---
The quick brown fox jumps over the lazy dog.
With measured words, she explained the strange rhythm of the universe.
Bright violets grow under the dense canopy, shading their vivid colors.
Jazz and classical music often feature complex harmonies and rhythms.
Six whimsical wizards make powerful, rhythmic chants deep in the forest.
The doctor’s prescription advised taking unique, holistic measures.
The swift cheetah is an elegant and extraordinary creature.
The rugged path wound around the steep hill to the quaint village.
Gazing at the azure sky in July, the dreamy poet wrote verses.
The glum monk thought deeply about life’s complex queries.
Questions
Does the quick brown fox jump over the lazy dog?
Can she explain the strange rhythm of the universe with measured words?
Do bright violets grow under the dense canopy, shading their vivid colors?
Do jazz and classical music often feature complex harmonies and rhythms?
Do six whimsical wizards make powerful, rhythmic chants deep in the forest?
Did the doctor’s prescription advise taking unique, holistic measures?
Is the swift cheetah an elegant and extraordinary creature?
Does the rugged path wind around the steep hill to the quaint village?
While gazing at the azure sky in July, did the dreamy poet write verses?
Did the glum monk think deeply about life’s complex queries?
Foreign Sentences
Finnish: ’En ymmarra mitaan mita tarkoitat’
French: ’Le coeur a ses raisons que la raison ne connait point.’
Spanish: ’El rapido zorro marron salta sobre el perro perezoso.’
German: ’Fuchse lieben es, schnell uber die grauen Hunde zu springen.’
Chinese: ’Feng yu zhi hou, bi jian cai hong.’
Russian: ’Na dvore trava, na trave drova.’
Arabic: ’Al-hayatu jamilatun ala al-raghmi min kulli shay’.
Hindi: ’Bandar bhagate hue chhat par chadh gaya.’
Japanese: ’Sakura no hana ga haru no kaze ni mau.’
Swahili: ’Simba anapenda kupumzika chini ya mti mkubwa.’
(a) Age distribution across predefined intervals. Figure 7: Distributions of
demographic variables in the dataset. (a) Age distribution (b) Distribution of
individuals by Sex (c) Distribution by Race categories. 29 out of 100
participants demographics had expired on Prolific.