# Contrastive Chain-of-Thought Prompting

Grant Kruttschnitt Jay Shim Alyssa Ma Daniel Kim
Benjamin Chek Athul Anand Kevin Zhu Sean O’Brien
Algoverse AI Research
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Rapidly increasing model scales coupled with steering methods such as chain-
of-thought prompting (Wei et al., 2022b) have led to drastic improvements in
language model reasoning. At the same time, models struggle with compositional
generalization and are far from human performance on many reasoning-based
benchmarks. Leveraging the success of chain-of-thought prompting, and also
taking inspiration from context-aware decoding (CAD) (Shi et al., 2023), we
explore input-based contrasting methods to further encourage the type of
reasoning induced by chain-of-thought prompting. While work remains to
stabilize these results across datasets and models, the improvements we find
warrant further investigation into input-based steering methods for context-
aware reasoning.

Contrastive Chain-of-Thought Prompting

Grant Kruttschnitt Jay Shim Alyssa Ma Daniel Kim Benjamin Chek Athul Anand
Kevin Zhu Sean O’Brien Algoverse AI Research<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large Language Models (LLMs) have revolutionized natural language processing
by leveraging vast amounts of data to generate human-like text. These models
have gained attention for their ability to perform a wide range of language
tasks, from text generation and translation to question answering and
summarization. One such technique that has propelled the performance of LLMs
is Chain-of-Thought (CoT) prompting (Wei et al., 2022b) . By applying CoT
prompting, researchers have found substantial improvement in various tasks
such as arithmetic and common sense reasoning, surpassing the efficacy of
traditional methods focused on scaling model size (Wang et al., 2023; Wei et
al., 2023). In a similar vein, Context-Aware Decoding (CAD) emerges as another
approach to augment LLM performance (Liu et al., 2021; Li et al., 2023a;
O’Brien and Lewis, 2023), priotizing relevant contextual information over pre-
existing knowledge (Shi et al., 2023). While these advancements underscore
significant progress, an ongoing question persists regarding the efficacy of
combining CAD with CoT or with amateur prompts, which typically contain less
context.

To address this question, the study aims to investigate how CoT prompts’
ability to capture sequential dependencies can be complemented by the
discriminative ability of CAD, particularly when the context contradicts prior
knowledge (Longpre et al., 2021; Zhou et al., 2023). In such cases, we aim to
ensure that incorporating CoT prompting does not detract from the outputs
derived from the provided context. Ultimately, the goal is to create faithful
models capable of accurately executing mathematical and reasoning tasks while
avoiding the misinterpretation of crucial contextual cues (Maynez et al.,
2020; Pagnoni et al., 2021).

##  2 Method

###  2.1 Background

We take a query x which we feed to our language model $\theta$, to generate a
response y using greedy decoding. The response is sampled autoregressively
from two probability distributions given by query $x$, CoT prompt c, and an
amateur model (3.2.1) with less context amt (Shi et al., 2023). Using a
contrastive strength of $\alpha$, we create a new distribution:

$y_{t}\propto\frac{p_{\theta}(y_{t}|\mathbf{c,x,y_{<t}})^{\alpha+1}}{p_{\theta}(y_{t}|\mathbf{amt,x,y_{<t}})^{\alpha}}$

We hypothesize that the model can effectively rely on the prompt c when
reasoning with the input while simultaneously penalizing incorrect reasoning
behaviors and conciseness (amt) to provide an accurate output.

###  2.2 Context-Aware Decoding

For each method we utilize two different prompts: an 8-shot CoT prompt, and an
amateur prompt. To identify the next logit and thus the next token to add
autoregressively, we use the contrastive function below proposed by Shi et al.
(2023):

$y_{t}\sim\textrm{softmax}\left[(1+\alpha)\exp(\text{logit}_{\theta}(y_{t}|\mathbf{c,x,y_{<t}}))\right.$
$\left.-\alpha\exp(\text{logit}_{\theta}(y_{t}|\mathbf{amt,x,y_{<t}}))\right]$

The values of $\alpha$ are as follows, $0\leq\alpha\leq 1$, with higher alpha
values indicating stronger reliance on prompt c, the CoT prompt. From the
adjusted logits, we apply greedy decoding to generate the subsequent
character. In our analysis, a single model was used to generate probabilities
for both CoT and amateur prompts, but different models may also be used.

##  3 Experimental Setup

###  3.1 Datasets

We evaluated our method on three question-answering task datasets that measure
performance based on reasoning abilities: GSM8K (Cobbe et al., 2021), AQuA
(Ling et al., 2017), and CommonSenseQA (Talmor et al., 2019). We provide
multiple-choice options in the prompts of CommonSenseQA and AQuA, but do not
for GSM8K. GSM8K and AQuA are math datasets, while CommonSenseQA contains
common-sense reasoning problems.

###  3.2 Models and Prompting

We performed our experiments using Phi-1.5, Mistral 7B, and GPT-3.5 with
version 3.5-turbo-0125. In each of our experiments, we contrasted the logits
obtained from a 8-shot CoT prompt with the amateur. The exemplars in the few-
shot prompts are taken from the corresponding dataset used for evaluation.

The expert prompt shared across all experiments is displayed below.

Expert Prompt (8-shot CoT) Q: {question 1} A: {CoT 1} {answer 1} $\cdots$ Q:
{question 8} A: {CoT 8} {answer 8} Q: {new question} A:

####  3.2.1 Expert vs Amateur 1

In our main method, the amateur is given no context. By doing so, we increase
the potential of our context-aware decoding and follow the formulation of
(Sanchez et al., 2023)’s method.
The effectiveness on this method is that this will cause it to hopefully rely
on the context much more, since the tokens that do rely on it will be much
more likelier to occur.

Amateur 1 Prompt A:

####  3.2.2 Expert vs Amateur 2

In our second amateur, we feed an 8-shot CoT prompt with the questions
omitted. This should encourage faithfulness to the question context, as
demonstrated in factuality-based tasks by (Shi et al., 2023). That is because
it would reduce the probability of logits that would happen even if there’s no
question, and increase the probabilities of those that only appear because of
it, making the logits that eventually end in it more likely to rely on the
question.

Amateur 2 Prompt A: {answer 1} $\cdots$ A: {answer 8} Q: {new question} A:

####  3.2.3 Expert vs Amateur 3

In the third amateur, we feed an 8-shot prompt with questions and answers, but
without CoT reasoning. This should hopefully induce further step-by-step
reasoning, under the intuition that a reasoning chain uniquely preferred by a
CoT prompted model will exhibit more thorough reasoning.

Amateur 3 Prompt Q: {question 1} A: {non-COT answer 1} $\cdots$ Q: {question
8} A: {non-COT answer 8} Q: {new question} A:

###  3.3 Hyperparameter Search

Measuring performance on Mistral-7B (Jiang et al., 2023) using the
CommonSenseQA dataset (Talmor et al., 2019), we found an optimal $\alpha$
value of $0.8$ (see Table 1). This is a constant we used across all datasets
and models. In addition, we decided to use 0.5, because although it did not
have the second highest accuracy, we wanted another alpha away from 0.8. That
would allow us to see a more pronounced difference between the alphas.

$\alpha$ | Accuracy
---|---
0.5 | 0.500
0.7 | 0.620
0.8 | 0.646
0.9 | 0.59
Table 1: Accuracy of Different CAD levels while Contrasting CoT and Amateur 3
with Mistral on 200 questions of CommonSenseQA

###  3.4 Baselines

We first run our datasets through 8-shot CoT prompting, without contrasting
with any amateur prompt, setting the benchmark accuracy for the rest of the
contrasting experiments. We used the same 8-shot COT prompt as we did for our
experts.

##  4 Results

As seen in Table 2, CAD improved AQUA’s accuracy on Phi by a substantial
result, and the same for Mistral on CommonSenseQA. The other accuracy
improvements were much more modest, with three percent and below in
improvement. The AQUA improvement might not show that much importance,
however, because even the improved accuracy still mostly stays at the random
rate of guessing.

The CoT Contrast works best with Phi-1.5 on AQuA, while it works best with
Mistral 7B on CommonsenseQA. Therefore, we do not find that contrastive CoT
increases problem solving performance in all language models for all data.

GSM8K notably shows a significant decrease in accuracy on Phi-1.5 using CoT
contrast. This could be due to contamination in the Phi-1.5B dataset (Zhang et
al., 2024)

There was also contamination in the Mistral model with GMS8K, so that may have
affected the results there too(Zhang et al., 2024).

But the results demonstrate that contrastive CoT is successful when addressing
common-sense multiple choice questions. The only time that one of the amateurs
does not have a greater value than the baseline on multiple choice is GPT-3.5
on AquA. Some improvements are much more modest than others, but all other
model-database combinations that are muultiple choice do improve.

The Mistral multiple-choice offers more improvement than Phi’s multiple-choice
which could be due to Mistral being a bigger model with more parameters. It
having more parameters could allow it to use the contrasting more powerfully.

Table 5 shows an interesting property of the outputs, as it appears the
baseline evaluates its expressions with lower accuracy than most of the other
experiments, but still results in a higher accuracy. After conducting analysis
on model output for each of the experiments, it appears as though, for the
experiments, the mathematical calculations may be higher in accuracy in the
Amateurs, but the expressions themselves contain incorrect numerical values
which may indicate an incorrect translation from the word questions to
expressions.

The overall results demonstrate that contrasting with CoT prompting and
amateur models is mostly successful when addressing common-sense multiple
choice questions.

Model | Dataset | Baseline | Amateur 1 ($\alpha$ = 0.5) | Amateur 1 ($\alpha$ = 0.8)
---|---|---|---|---
Phi-1.5 | GSM8K | 34.3 | 26.9 | 23.9
Phi-1.5 | AQuA | 19.3 | 22.1 | 25.6
Phi-1.5 | CommonSenseQA | 24.5 | 24.1 | 24.9
Mistral 7B | GSM8K | 41.3 | 42.0 | 34.6
Mistral 7B | AQuA | 30.3 | 33.1 | 29.1
Mistral 7B | CommonSenseQA | 47.1 | 48.7 | 52.0

Table 2: Accuracies (in percentages) of baseline and contrastive CoT with
varying $\alpha$. CCoT typically outperforms standard CoT on arithmetic and
commonsense reasoning tasks.

Method | GMS8K | AQuA | CommonSense
---|---|---|---
Baseline | 58.7 | 52.8 | 61.4
Amateur 1($\alpha$ = 0.8) | 53.9 | 28.7 | 64.4

Table 3: GPT-3.5 Results on Amateur 1 on 0.8 Alpha. It does not perform as
well on GPT, with only one of the CCoT having a better result.

Model | Dataset | Baseline | Amateur 2 ($\alpha$ = 0.5) | Amateur 2 ($\alpha$ = 0.8) | Amateur 3($\alpha$ = 0.5) | Amateur 3($\alpha$ = 0.8)
---|---|---|---|---|---|---
Phi-1.5 | GSM8K | 34.3 | 30.2 | 26.3 | 33.4 | 31.6
Phi-1.5 | AQuA | 19.3 | 23.6 | 21.3 | 22.0 | 21.7
Phi-1.5 | CommonSenseQA | 24.5 | 23.5 | 15.6 | 24.1 | 24.9
Mistral 7B | GSM8K | 41.3 | 38.9 | 43.2 | 42.2 | 41.6
Mistral 7B | AQuA | 30.3 | 35.8 | 33.9 | 31.5 | 29.9
Mistral 7B | CommonSenseQA | 47.1 | 26.1 | 48.2 | 26.1 | 39.2

Table 4: Results on Other Amateurs. Amateur 2 seems promising because out of
the six categories, it has the best result in four of them.

| Baseline | Amateur 1 (0.5) | Amateur 1 (0.8) | Amateur 2 (0.5) | Amateur 2 (0.5) | Amateur 3 (0.5) | Amateur 3 (0.5) | Coherence Boosting (0.5) | Coherence Boosting (0.8)
---|---|---|---|---|---|---|---|---|---
Mean | 6.110 | 6.088 | 6.039 | 6.071 | 6.037 | 6.062 | 6.089 | 4.386 | 5.331
Proportion | 0.589 | 0.617 | 0.626 | 0.607 | 0.615 | 0.608 | 0.606 | 0.529 | 0.542

Table 5: Comparing mean number of sentences per output and the proportion of
expressions evaluated correctly for each experiment. Alpha values are in
parenthesis.

##  5 Related Work

This work is inspired by three main ideas: Context-Aware Decoding, Chain of
Thought, and prompting methods. Here we detail the most relevant papers in
these paths.

The first direction that this paper relates to is prompting. The last few
years have given way to major improvements in input part of prompt
instructions. This is apparent with the popularity of few-shot prompting by
(Brown et al., 2020), automatically learning prompts (Lester et al., 2021), or
providing model prompts by describing tasks (Wei et al., 2022a; Sanh et al.,
2022; Ouyang et al., 2022). This idea leads to our paper’s relation to Chain-
of-Thought Prompting.

Chain-of-Thought prompting was a leading discovery with the ability to improve
model performance significantly without scaling size. With such an improvement
in performance, one of the main focuses of this paper was trying to take the
improvement even further. Our main reference would be (Wei et al., 2023) for
its groundbreaking discovery and testing of arithmetic and common-sense
benchmarks. This paper uses similar benchmarks and prompting for testing chain
of thought with context-aware decoding.

Lastly, this paper relates to others that pushed for context-aware decoding
and its ability to decrease hallucination in LLMs. In particular, the paper
that was referenced was (Shi et al., 2023) for its novelty in getting LLMs to
pay more attention to the context provided. The context-aware decoding was
effectively used in this paper to improve the accuracy of models and decrease
hallucination.

##  6 Conclusion

Our findings highlight the impact of integrating expert and amateur models in
Context-Aware Decoding across diverse datasets. This study highlights the need
for future research prioritizing context-sensitive applications in language
modeling. Further, there is a pressing need to explore the scalability of
these techniques and their relevance across even broader datasets and across
differing models and sizes. Future investigations could delve into testing a
wider range of alpha values to unveil potentially more substantial results,
aiding our understanding of the nuances within context-driven language
modeling.

##  7 Limitations

Our analysis could be limited in that the difference in strength between the
expert and amateur models is too inconsequential. That means that there is a
possibility that the models themselves have generated outputs that are not
dissimilar enough to impact results.

Another limitation specifically on the Phi 1.5 GMS8K results could be the fact
that Phi 1.5 is possibly contaminated with GSM8K (Zhang et al., 2024). This
explains why our trials for Phi - 1.5 produced many unfinished and illogical
answers when responding to input prompts, yet performed much better on the
baseline compared on Amateur 1 on GSM8K (Table 2).

Phi-1.5 has also been known to work best with prompts in a QA format which
inherently makes the format of the Amateur 1 and 2 prompts difficult for the
model to synthesize outputs, and thus impact results (Li et al., 2023b). This
could also result because of the fact that Phi-1.5 is too weak of a model with
only 1.3 billion parameters to accurately display the effect of CoT
contrasting.

Additionally, GPT-3.5 Turbo is a much larger model compared to Phi-1.5 and
Mistral 7B. This could affect results too, because while it seems that the
bigger size could make it more effective, as it’s able to do contrasting more
effectively, which it might, the contrasting could also get diluted in the
billions of parameters.

Among the three, only GPT-3.5 Turbo is an instruction-tuned model, meaning
that it’s ability to reason through questions may not be a result of CoT
contrasting but rather because it already contains such capabilities. There
also might be things similar to our contrasting methods already hidden in
their instructions, which would make our methods less effective.

The under-performance of GPT-3.5’s GSM8K and AQuA results may be attributed to
GPT-3.5’s difficulty in consistently answering mathematical questions
accurately Frieder et al. (2023), as tested GPT versions of January 9, January
30, and GPT-4 extensively with GHOSTS, a natural language mathematics dataset,
and concluded they were not competent in advanced mathematical comprehension.
This could explain why the accuracy did not improve even with context-aware
decoding.

For future experiments, the first steps to improve should be to run
experiments on non-instruction tuned and much larger models that can better
reflect whether or not chain-of-thought vs. amateur contrasting improves the
accuracy of outputs. There could also be more experiments done on different
datasets, to see how the constrasting works differently for different types of
problems. Additionally, selecting an optimal $\alpha$ value for each
individual task, question or even token could potentially improve performance.

## References

  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021.  Training verifiers to solve math word problems.
  * Frieder et al. (2023) Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner. 2023.  Mathematical capabilities of chatgpt.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2023a) Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023a.  Contrastive decoding: Open-ended text generation as optimization.
  * Li et al. (2023b) Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023b.  Textbooks are all you need ii: phi-1.5 technical report.  _arXiv preprint arXiv:2309.05463_.
  * Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017.  Program induction by rationale generation : Learning to solve and explain algebraic word problems.
  * Liu et al. (2021) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021.  Dexperts: Decoding-time controlled text generation with experts and anti-experts.
  * Longpre et al. (2021) Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021.  Entity-based knowledge conflicts in question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * O’Brien and Lewis (2023) Sean O’Brien and Mike Lewis. 2023.  Contrastive decoding improves reasoning in large language models.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pagnoni et al. (2021) Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021.  Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4812–4829, Online. Association for Computational Linguistics.
  * Sanchez et al. (2023) Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman. 2023.  Stay on topic with classifier-free guidance.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.
  * Shi et al. (2023) Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Scott Wen tau Yih. 2023.  Trusting your evidence: Hallucinate less with context-aware decoding.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023.  Self-consistency improves chain of thought reasoning in language models.
  * Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a.  Finetuned language models are zero-shot learners.
  * Wei et al. (2022b) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. 2022b.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023.  Chain-of-thought prompting elicits reasoning in large language models.
  * Zhang et al. (2024) Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. 2024.  A careful examination of large language model performance on grade school arithmetic.
  * Zhou et al. (2023) Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023.  Context-faithful prompting for large language models.


# Statistical Design and Analysis for Robust Machine Learning: A Case Study
from COVID-19

Davide Pigoli111These authors contributed equally to this work.222Address for
correspondence: Department of Mathematics, King’s College London, Strand,
London WC2R 2LS, United Kingdom. Email<EMAIL_ADDRESS>King’s
College London, UK and The Alan Turing Institute, London, UK.  Kieran Baker∗
King’s College London, UK and The Alan Turing Institute, London, UK.  Jobie
Budd  University College London, UK.  Lorraine Butler  UK Health Security
Agency, London, UK.  Harry Coppock  Imperial College London, UK.  Sabrina
Egglestone  UK Health Security Agency, London, UK.  Steven G. Gilmour  King’s
College London, UK and The Alan Turing Institute, London, UK.  Chris Holmes
University of Oxford, UK and The Alan Turing Institute, London, UK.  David
Hurley  UK Health Security Agency, London, UK.  Radka Jersakova  The Alan
Turing Institute, London, UK.  Ivan Kiskin  University of Surrey, UK.
Vasiliki Koutra  King’s College London, UK and The Alan Turing Institute,
London, UK.  Jonathon Mellor  UK Health Security Agency, London, UK.  George
Nicholson  University of Oxford, UK.  Joe Packham  UK Health Security Agency,
London, UK.  Selina Patel  UK Health Security Agency, London, UK and
University College London, UK.  Richard Payne  UK Health Security Agency,
London, UK.  Stephen J. Roberts  University of Oxford, UK and The Alan Turing
Institute, London, UK.  Björn W. Schuller  The Alan Turing Institute, London,
UK and Imperial College London, UK.  Ana Tendero-Ca$\tilde{\mathrm{n}}$adas
UK Health Security Agency, London, UK and University of Brighton, UK.  Tracey
Thornley  University of Nottingham, UK.  Alexander Titcomb  UK Health Security
Agency, London, UK.

###### Abstract

Since early in the coronavirus disease 2019 (COVID-19) pandemic, there has
been interest in using artificial intelligence methods to predict COVID-19
infection status based on vocal audio signals, for example cough recordings.
However, existing studies have limitations in terms of data collection and of
the assessment of the performances of the proposed predictive models. This
paper rigorously assesses state-of-the-art machine learning techniques used to
predict COVID-19 infection status based on vocal audio signals, using a
dataset collected by the UK Health Security Agency. This dataset includes
acoustic recordings and extensive study participant meta-data. We provide
guidelines on testing the performance of methods to classify COVID-19
infection status based on acoustic features and we discuss how these can be
extended more generally to the development and assessment of predictive
methods based on public health datasets.

### Keywords:

UK COVID-19 Vocal Audio Dataset, Bioacoustic markers, Confounding, Choice of
test set, Matching.

##  1 Introduction

From the beginning of the coronavirus disease 2019 (COVID-19) pandemic, it has
been recognised that rapid and widespread testing is one of the most important
public health measures for containing the spread of the virus (World Health
Organization, 2020). The gold-standard reverse-transcription polymerase chain
reaction (RT-PCR) test is very sensitive and specific to severe acute
respiratory syndrome 2 (SARS-CoV-2) viral ribonucleic acid (RNA), but slow and
expensive to carry out, so is not considered practical for widespread
community testing. Lateral flow antigen tests offer a faster way to identify
COVID-19 positive individuals, especially those with high viral load. However,
they have cost and usability challenges and are less sensitive than a PCR
test. With the continuing impact of COVID-19, there is a requirement for
faster, simpler and cheaper ways to test for infection to reduce the impact of
widespread transmission.

One of the earliest identified symptoms of COVID-19 was a distinctive dry
cough. Since 2020, researchers (see, e.g., Laguarta et al., 2020; Han et al.,
2021; Brown et al., 2020) have explored the use of machine learning to
classify forced cough samples into those from COVID-19 positive and COVID-19
negative individuals. They have reported results which indicate high levels of
accurate classification, however, it is difficult to assess from these studies
how well the classifiers might actually perform in practice, as discussed in
Coppock et al. (2021).

In early 2021, a study was set up by the UK Health Security Agency (UKHSA) and
the Turing-RSS Health Data Laboratory – a working partnership between The Alan
Turing Institute and Royal Statistical Society – to rigorously assess the
feasibility of these methods as a public health tool. A team of statisticians
and computer scientists was brought together, with the aim of assessing what
levels of accuracy might be achievable in practice. An analysis of subject-
level covariates (meta-data) of the available data was used to choose training
and test sets to enable development and assessment of machine learning models
with a minimum of bias and as great a chance as possible to have a similar
performance when applied to the wider population.

A relevant issue when estimating the accuracy of machine learning methods on
public health datasets of observational nature, or surveys with a high level
of non-responses, is the presence of bias in the dataset, i.e., the dataset
does not reflect the population of interest in some important aspect. This can
lead to estimates of accuracy that cannot be replicated when the methods are
later applied in practice. A second issue is the presence of confounders,
i.e., variables that are correlated with the outcome and the predictors and
that the machine learning method can learn to predict instead of the actual
outcome of interest to obtain a greater accuracy within the dataset. It is
doubly problematic when bias and confounding are both present, in the sense
that confounding variables are correlated with the outcome of interest in the
dataset but not in the general population. This can lead to a machine learning
method that is extremely accurate in the dataset but useless for practical
purposes.

While the focus on this work is on the prediction of COVID-19 infection status
from acoustic features, many of the statistical issues discussed in this paper
are valid more generally for machine learning methods trained on observational
data, and the approach we suggest for the assessment of these methods in
Section 5 is an important contribution to the currently active research on
explainable AI (Watson, 2022; Rudin, 2019), in particular for healthcare
applications (see, for example, Babic et al., 2021).

##  2 Previous work on vocal acoustic features for predicting COVID-19 status

This section provides a discussion of some of the previously published papers
which have sparked interest in the use of acoustic biomarkers to predict
COVID-19 infection status and demonstrated seemingly promising results in this
direction.

Laguarta et al. (2020) used the MIT Open Voice model to predict COVID-19
infection status based on cough recordings, using crowd-sourced data collected
from a web form (5320 subjects), where COVID-19 infection status was mostly
self-reported. The authors report a sensitivity of $100\%$ on asymptomatic
patients (of the group of subjects diagnosed with an “official test”), but it
is unclear whether asymptomatic patients are properly held out for out-of-
sample prediction on the test set. With a corresponding specificity of
$83.2\%$, it is possible that the $100\%$ sensitivity is so high due to over-
fitting, rather than high predictive power. The absence of demographic
information (e.g., on geographical region or age) also makes it difficult to
assess the generalisability of the results.

Brown et al. (2020) collected a crowdsourced dataset comprised of submissions
from 6613 participants through a web-based and a mobile app (COVID-19 Sounds),
and predicted COVID-19 status based both on hand-crafted features extracted
from coughs and breathing sounds and features extracted using transfer
learning techniques from the same signals, using various machine learning
techniques, such as logistic regression, gradient boosting trees and Support
Vector Machines (SVMs). They reported an area under the curve of the Receiver
Operating Characteristic curve (ROC-AUC) of $80\%$. Their use of transfer
learning and handcrafted features therefore appeared to show promise. However,
the number of positive cases in the dataset is small, especially when
considering test/validation sample sizes. In addition, the authors selected
negative cases from countries with low infection prevalence at the time of
sampling, which may introduce bias associated with the mother tongue (through
its influence on speech physiology) that is not accounted for. To ensure the
classifier is not just classifying participants to their country, a more
appropriate approach may have been to select a group of COVID-19 negative
participants within a high-prevalence country. The effect of demographics on
the data and sound samples is not considered, perhaps due to the limitations
of the dataset. Also, people in the age bracket 20 to 49 years old seem to be
over-represented in the sample. Moreover, the imbalance between positive and
negative cases suggests that the use of the ROC curve might not be suitable,
and adjustments or alternative methods might be appropriate.

Han et al. (2021) discussed voice-based models for COVID-19 status that used
symptoms to classify positive and negative cases. The authors combined symptom
covariates with speech recordings. However, the performance greatly varied
across folds (wide standard deviation), which indicates epistemic uncertainty
and the requirement of more training data. It was noted that a much smaller
dataset was used compared to Brown et al. (2020) (with 343 participants coming
from 4 countries, unequally represented). The authors implemented the
synthetic minority oversampling technique (SMOTE), which adds synthetic data
to the minority class to achieve balance between the positive and negative
cases. There is no explicit information related to the way the positive cases
were chosen. Moreover, similar to Brown et al. (2020), there is a lack of
evaluation related to demographics. More recently, this issue was addressed by
Han et al. (2022), where it is indeed shown how biases and participant splits
can affect the performance of the method, using a dataset of 2478 volunteers
who self-reported COVID-19 test results.

To summarise, there are reports of accurate COVID-19 status prediction from
vocal acoustic features across the existing literature. However, there is also
a recognition of the need for large, clinically referenced datasets with
sufficient metadata and transparency when using machine learning and
artificial intelligence techniques for diagnostics. As mentioned above, Han et
al. (2022) discuss how the evaluation of these diagnostic procedures can be
affected by biases in the dataset. As we argue in this paper, a careful design
of the assessment procedure is also of paramount importance, and is only
possible when accurate metadata from study participants are available. We are
going to show that, when the study (i.e., both the data collection mechanisms
and the procedure to estimate and compare methods’ performance) is designed
specifically to assess out-of-sample generalisability and to investigate the
predictive power of the acoustic information, the results may no longer be as
impressive.

##  3 Data Collection

Developing and accurately assessing bioacoustics-based classification methods
requires a carefully designed study, both from the point of view of the data
collection and from the point of view of how to test and compare predictive
performances. Concerning the latter, we do not need to rely on a randomised
controlled trial, since it is possible to compare classification methods using
all the subjects and there is no need to rely on methods which require
counterfactuals. However, an important question is then how the performance of
these methods extends to the population of interest, as opposed to the
collected dataset.

Therefore, the main issue is to ensure that data are collected from a well
understood environment, with as much relevant metadata as possible, to ensure
that possible confounding variables are accounted for. In principle, we would
like the dataset to be representative of the population of interest, but in
practice we expect some biases to be present and collecting all the relevant
meta-data allows us to adjust the assessment procedure to account for these
biases. For example, the split of the data into training and test sets should
be done in such a way as to guarantee that the true out-of-sample performance
can be measured as well as possible.

Following the publication of initial studies reporting accurate classification
of COVID-19 infection status from vocal and respiratory audio, the UK Health
Security Agency (UKHSA, formerly NHS Test and Trace, the Joint Biosecurity
Centre, and Public Health England) were commissioned to collect a dataset to
allow for the independent evaluation of these studies. The “Speak up and help
beat coronavirus” study (UK Health Security Agency, 2021) was set up to
collect data for this purpose. A description of the resulting dataset (UK
COVID-19 Vocal Audio Dataset) and how to access it can be found in Budd et al.
(2022). Here, we give a quick overview and highlight the key variables used in
the design and evaluation of the predictive procedure described and assessed
in detail in Coppock et al. (2022). The primary analysis that we discuss in
this paper was based on data collected from this study between 01 March 2021
and 29 November 2021, giving 39850 submissions. It should be noted that the
dataset described in Budd et al. (2022) also contains submissions collected
after 29 November 2021. However, this was the date where the meta-data were
analysed to design the procedure to assess the methods’ performances. Thus,
data collected after this cut-off date were not considered in the primary
analysis, but they were used for some of the follow-up analysis described in
Coppock et al. (2022).

Volunteer participants recruited via two routes: (i) NHS Test and Trace
community testing in England and (ii) the REACT-1 (REal-time Assessment of
Community Transmission) survey (Riley et al., 2020). In the period covered by
the data collection, people were advised to seek a PCR-test through NHS Test
and Trace if they were experiencing COVID-19 symptoms, they were identified as
a close contact of a positive case, or following a positive rapid antigen
(lateral flow) test (until 11th January 2022). On the other hand, REACT-1 was
commissioned by the UK Department of Health and Social Care to estimate the
prevalence of SARS-CoV-2 infection in the community in England (and influenza
A and B in later survey rounds). It was carried out by Imperial College London
in partnership with Ipsos MORI using repeat, random, cross-sectional sampling
of the population.

Participants in the “Speak up and help beat coronavirus” study were asked to
complete an online survey on a smartphone, tablet, or computer. Health and
demographic information and audio recordings made via the microphone of the
participant’s device were collected. These data were linked with the
participants’ COVID-19 PCR test results and associated information via a
patient identifier, as described in Budd et al. (2022). In the following, we
are going to refer to submissions linked to a positive or negative COVID-19
PCR test result as COVID-19 positive and COVID-19 negative submissions. Also,
while technically the PCR test is an indicator of SARS-CoV-2 infection status,
in this paper we are going to refer to it as COVID-19 infection status
(positive or negative), since this terminology has become widespread in the
public health communication. Each submission consists of four audio modalities
(one and three successive forced coughs, three successive exhalation sounds,
and a full sentence read from text), and metadata including individual-level
information (age, gender, ethnicity, local authority, existing respiratory
health conditions, smoker status, first language, height, weight), PCR test
result (SARS-CoV-2+ or SARS-CoV-2-), self-reported symptoms as listed in
Figure 1, date of the PCR test, date of the audio submission and recruitment
source (NHS Test and Trace contact or REACT-1 survey). PCR cycle threshold and
resulting viral load estimation, as well as vaccination status information, is
available for some participants. Further details on the recruitment procedures
and the dataset are available in Budd et al. (2022).

For the analysis described in this paper, data was then removed for
participants if they did not satisfy all the following conditions:

  * •

Participant aged 18 or over.

  * •

Test results obtained via PCR.

  * •

Audio recordings submitted between the test result and 10 days after the test
result.

  * •

Test performed by a lab not under investigation by UK Health Security Agency
for inaccurate results.

  * •

No discrepancy in recording of symptoms (both symptoms and no symptoms
selected)

Submissions with missing data, either in terms of missing one or more of the
audio submissions or missing some of the meta-data, were removed. The
resulting dataset without missing data was comprised of 37018 submissions.

Figure 1 shows a breakdown of the meta-data for the missing data submissions
and for the final dataset (disclosure control measures have been implemented
where categories with less than 5 participants have been shown, see Budd et
al. (2022) for more details). Since this does not highlight any systematic
patterns and the overall number of instances with missing data is modest,
submissions with missing data were removed from the analysis.

| Missing audio (1213) | Missing meta-data (162) | Final dataset (37018)
---|---|---|---
Gender |  |  |
Test result |  |  |
Height by gender |  |  |
Age by test result |  |  |
Symptoms |  |  |
First Language |  |  |
Weight by gender |  |  |
Figure 1:  Breakdown of the available meta-data for the portions of the
dataset with missing audio features, missing meta-data and complete data,
i.e., the final dataset used in the following analysis. We cannot see any
problematic imbalances that may suggest that audio-feature or meta-data are
not missing at random.

##  4 Analysis of Meta-data

We carried out a descriptive analysis of the meta-data to explore potential
biases in the data collection and to design a test set which aims to provide a
reliable estimate of the performance in the general population.

### Self-reported symptoms.

The distribution of the self-reported symptoms varies dramatically between the
submissions with COVID-19 positive status and those with COVID-19 negative
status (see Figure 2), thus raising the question of potential confounding
between symptoms and COVID-19 status. We can see that for COVID-19 negative
submissions, there is a dominance of no-symptoms, and very few symptomatic
negative cases. For COVID-19 positive cases, there are only 446 asymptomatic
submissions – a key group for understanding the efficacy of models in
detecting COVID-19 infection status. It is also worth noting the dominance of
the cough symptom as the most common symptom in both positive and negative
cases.

### Other respiratory conditions and smoker status.

Information related to other respiratory conditions and smoker status was
captured in the data collection due to possible effects on the nature of
someone’s cough, e.g., if a person is asthmatic, or an ex-smoker. Stacked bar
plots and cross tabulations of these are included in Figure 3. Neither of
these appear to be associated with COVID-19 infection status in our dataset.

Figure 2: Distributions of self-reported symptoms for the submissions with
positive PCR results (top) and negative PCR results (bottom). Each symptom is
listed on the left, ordered by the frequency of appearance of each symptom.
The combinations of each pair of symptoms is represented in the vertical
stripes within each figure, ordered by the frequency of the symptom
combinations shown at the top of each plot. Frequency cut offs of 80 and 30
for symptom combinations are used for the positive PCR and negative PCR plots
respectively.

### Age, Gender, Height and Weight.

There is some evidence supporting the hypothesis that age might act as a
confounder of COVID-19 status in the dataset, with a dominance of older
negatives and younger positives. This could be caused by any number of
behavioural, societal and economic variations between younger and older
people, as well as increased vaccination levels in the older population – this
tendency is clearly seen in Figure 4. There is a difference of 12 years
between the median ages of those who tested positive and those who tested
negative. Note that the median age for those testing positive without symptoms
was 47 years. There does not appear to be any confounding between gender and
COVID-19 status (see Figure 4), although there are 6,426 more females than
males.

Figure 3: Left: Barplots of absolute frequencies of smoker status in the final
dataset by COVID-19 status. Right: Barplots of absolute frequencies of
respiratory conditions in the final dataset by COVID-19 status.

Figure 4: Left: Distribution of Age by COVID-19 status. Right: Frequency of
Gender by COVID-19 status.

Figure 5: Left: Subjects’ height broken down by COVID-19 status in the final
dataset. Right: Subjects’ weight broken down by COVID-19 status in the final
dataset.

Since weight and height were recorded only as classes, to analyse the height
and weight data, the midpoints of the bins were taken and then converted to
centimetres or kilograms for height and weight respectively. Height and weight
do not appear to be confounded with COVID-19 status (see Figure 5), and the
distributions of heights and weights appear reasonable, the only exception
being a proportion of participants who selected the lowest height and weight
option, probably due to these being the first options offered in the form (see
Budd et al., 2022, for more details).

### Recruitment source.

Members of the public were invited to take part in the study after undergoing
a test through two existing initiatives: NHS Test and Trace community testing
programme (Pillar 2) and REACT-1 (community prevalence survey). The difference
in these two testing initiatives meant that the NHS Test and Trace was a large
recruiter of COVID-19 positive participants, since it was aimed at subjects
suspected of having been infected and initially people were only contacted
after a positive test result was confirmed (see Budd et al., 2022). On the
other hand, REACT-1 survey was aimed to randomly sample the population and
hence, recruited a large number of negative cases with a few more positives
coming through in periods of higher prevalence. Because of the recruitment
structure, the recruitment channel is almost entirely confounded with COVID-19
infection status, as can be seen in Figure 6 and Table 1, which show the
recruitment pattern over time and cross-tables between these variables
respectively. Each rapid increase in REACT-1 submissions over time corresponds
to the different REACT-1 recruitment rounds.

Table 1:  Absolute frequency tables of COVID-19 status vs recruitment source,
COVID-19 status vs symptoms and symptoms vs recruitment source. |  | COV+ | COV-
---|---|---
T&T | 13035 | 962
REACT | 164 | 22857
|  | COV+ | COV-
---|---|---
Symp | 12753 | 5548
No Symp | 446 | 18271
|  | T&T | REACT
---|---|---
Symp | 13256 | 5045
No Symp | 741 | 17976
Figure 6:  Submission count over time for both the Test and Trace and REACT
recruitment channels.

### Geographic dispersion.

Figure 7: Geographic distribution of the submissions by recruitment source
(Left: NHS Test $\&$ TRACE, Right: REACT-1). Top: Total number of submissions.
Bottom: Submissions as proportions of the total population.

The geographic origin of the submissions for both recruitment sources can be
seen in Figure 7. The geographic distribution of the submissions appears to be
broadly similar for the two recruitment sources (and therefore for the two
COVID-19 statuses, due to the confounding discussed above). On the other hand,
not all regions contributed to the data collection in the same way and this is
therefore identified as a potential source of confounding. It should also be
noticed that the vast majority of the submissions come from England, with only
a very small number of submissions coming from Scotland and Wales via the NHS
Test and Trace route and none from Northern Ireland.

##  5 Choice of Challenging Test Sets

Having performed the exploratory analysis of the metadata, our aim was to
choose a split between train and test sets such that performance on the test
set mimicked the performance on the general population as well as possible.
For this purpose, variables which showed large imbalances between positive and
negative cases were considered as potential confounders for audio-based
classifiers. We considered, for example:

  * •

Age

  * •

Recruitment Source

  * •

Symptoms (Cough (any), Fatigue, Headache, A change to sense of smell or taste,
Runny or blocked nose, Fever (a high temperature), Loss of taste, Shortness of
breath, Sore throat, A new continuous cough, Diarrhoea, Abdominal pain, Other
symptom(s) new to you in the last 2 weeks, No symptoms, Prefer not to say.)
All are binary variables – either present (True) or not (False).

  * •

Geographic location.

Secondly, the test set was chosen to systematically over-represent category
combinations that are sparse in the overall data set. These are submissions
with:

  * •

First language different from English.

  * •

Ethnicity different from White British.

  * •

Older individual with positive tests results.

  * •

Younger individual with negative test results.

We then proceed in defining a split between training and test sets of 25,897
and 11,121 submissions respectively, a $70\%$-$30\%$ train-test split, which
was not uniformly randomly selected but based instead on the information
coming from the meta-data analysis. The detailed procedure to sample the test
set can be found in Algorithm 1, and we refer to the test set produced by this
procedure as the _designed_ test set.

Algorithm 1 Test set construction. The sample sizes in parentheses reflect the
test set that was generated for the methods’ assessment.

The following steps have been applied in this order to construct the test set
from the submission meta-data:

  1. 1.

Select all records from 5 randomly selected languages (excluding English)
(n=370). (to test out of sample performance for unseen first languages.)

  2. 2.

Select all records from 5 randomly selected ethnic or nationality groups
(excluding British) (n=857). (to test out of sample performance for unseen
ethnic or nationality groups.)

  3. 3.

Select all negative cases from Leeds and Cornwall (n=547) (due to large
numbers of positive submissions, to test out of sample performance for unseen
locations)

  4. 4.

Select all positive cases form Birmingham and Sheffield (n=388) (due to large
numbers of negative submissions, to test out of sample performance for unseen
locations)

  5. 5.

Select all records from 4 other randomly selected local authorities (n=390)
(to test for geographic and dialectal confounding.)

  6. 6.

Select all asymptomatic cases (positive test result with ‘no symptoms’
selected) (n=439)
(to test if the audio-based method can be extended to asymptomatic positive
cases, which are rare in the dataset.)

  7. 7.

Of those whose age is above the median by gender and tested positive, $50\%$
of records are selected (n=1299).

  8. 8.

Of those whose age is below the median by gender and tested negative, $50\%$
of records are selected (n=3032).
(to understand if Age could be acting as a confounding variable).

  9. 9.

Select all REACT positives (n=79)

  10. 10.

Select all Test and Trace negatives (n=962)
(to understand if recruitment channel could be acting as a confounding
variable.)

  11. 11.

Sample without replacement to ensure there is an even distribution of viral
load categories (n=598 high, 598 medium and 598 low) (in case the analysis of
accuracy of the models by viral load is required.)

  12. 12.

Fill test set to a 70-30 split by sampling without replacement from the
remaining data randomly from records where viral load is not recorded (n=2932)

Note that the groups listed above in (a)-(l) are not mutually exclusive.

However, in addition to the out-of-sample performance, there were also some
concerns that the method could act as a symptoms (or more generally metadata
variable) detector and any positive performance could be entirely due to the
confounding, which was only partially accounted for in the designed test set
and it would limit the usefulness of method as diagnostic tool.

Therefore, it is also appropriate to test the performance of the method on a
subset of the designed test set, which has been constructed to control for all
possible confounders in the collected data. We refer to this test set as the
1:1 _matched_ test set and it is constructed by exactly balancing the numbers
of COVID-19 positive and COVID-19 negative individuals in each stratum, where
to be in the same stratum individuals must be matched on all of the following
variables:

  * •

Recruitment channel (Test and Trace or REACT).

  * •

Age (binned in 10 years intervals).

  * •

Gender.

  * •

Cough (TRUE or FALSE).

  * •

Sore throat (TRUE or FALSE).

  * •

Asthma (TRUE or FALSE).

  * •

Shortness of breath (TRUE or FALSE).

  * •

Runny/blocked nose (TRUE or FALSE).

  * •

“At least one symptom” (TRUE or FALSE).

The matched test set has 984 COVID-19 positive and 984 COVID-19 negative
participants. This reduces the size of the test set, resulting in a more noisy
estimation of the prediction performance, but it also should remove the bias
introduced by the known (recorded) confounders.

##  6 Results of application of machine learning and statistical methods

The detailed description of the AI methods used to predict COVID-19 status
based on the audio submissions is beyond the scope of this paper and can be
found in Coppock et al. (2022), together with a full discussion of the results
on the designed and matched test sets, based on multiple indicators. This
paper reproduces a summary of the results of the analysis for one of the
methods used, i.e., a support vector machine classifier based only on audio
features extracted with open source speech and music interpretation by large-
space extraction (openSMILE) (see Eyben et al., 2013, for more details). We
also compare it with a logistic regression classifier based only on the
metadata. This is to have a benchmark of what would be the predictive accuracy
of methods based on the meta-data instead of the acoustic features. By
construction, this logistic regression will not be able to perform well on the
matched test set. A random split between training and test sets is also
considered for comparison, to see what we could have concluded in absence of
the careful analysis we carried out.

The area under the ROC curve index (AU-ROC) computed for the prediction on the
designed and matched test set can be found in Table 2. It can be seen that for
both methods the performance is significantly worse for the matched test set,
thus giving credence to the hypothesis that any signal in the data is due to
the confounding variables. Furthermore, even on the designed test set, the
performance of the method based on audio submission is worse than that of the
logistic regression based on metadata, thus supporting the hypothesis that
this method is not able to find additional information in the audio signals
with respect to the metadata. Both methods perform well on the randomly
selected test set, with performances in line with previously published studies
discussed in Section 2.

Table 2:  Area under the ROC Curve for the SVM classifier based on audio samples and for the logistic regression based on the metadata, when applied to a randomised train-test split, the designed train-test split obtained with Algorithm 1, and a matched test set obtained from the designed test-train split as described in Section 5, respectively. | Randomised | Designed | Matched
---|---|---|---
SVM - openSMILE | 0.80 | 0.73 | 0.60
Logistic regression | 0.95 | 0.89 | 0.59

##  7 Discussion

Recently, there has been much interest in using machine learning methods to
predict COVID-19 status based on audio signals, for example coughs. While
early results seemed promising, they were affected by limitations in the data
quality and their out-of-sample performance was an open issue. The ‘Speak up
to help beat coronavirus’ study, carried out by UKHSA, collected one of the
largest and most comprehensive datasets available to date to explore this
scientific question (the UK COVID-19 Vocal Audio Dataset). However, despite
the quality and size of the collected data, biases are still present due to
the inherent limitations in the data collection procedure, and particular care
needs to be applied when designing a study to assess the out-of-sample
performance of machine learning techniques. When confounding is accounted for
with a matching strategy, the predictive power of the acoustic features
becomes negligible in our model. This does not necessarily imply that it is
not possible to use acoustic features to predict COVID-19 infection status,
but we think we convincingly showed that without exceptional care in designing
the assessment procedure the risk of reporting inflated performance results is
very real, even with a large and well-curated dataset.

This study highlights important lessons on the use of machine learning
techniques in observational studies in health sciences, that go beyond the
immediate application to audio-based classifiers. Even with very large
datasets and multiple data sources, biases and confounding variables can be
present and dramatically overstate the prediction accuracy of machine learning
methods. The estimated prediction accuracy can largely be corrected by using a
matching strategy that approximates a case-control study, but this comes at
the cost of a much reduced size for the test set (and potentially the training
set, see Coppock et al. (2022) for more details on how this strategy can be
applied to the training as well).

It is therefore important to design the data collection procedure to record
participant information that can be associated with the outcome to be
predicted. However, in the case of the “Speak up and help beat Coronavirus”
study, some of the confounding was clearly due to the data collection
procedure. In particular, the much lower participation rate in the study for
symptomatic individuals who tested negative via NHS Test and Trace, in
comparison with the symptomatic individuals who tested positive. For future
studies, this suggests the need to monitor at least the most obvious potential
imbalances during the data collection and to intervene by making additional
effort in recruiting underrepresented subgroups into the study.

## Ethics

The study described in this paper has been approved by The National
Statistician’s Data Ethics Advisory Committee (reference NSDEC(21)01) and the
Cambridge South NHS Research Ethics Committee (reference 21/EE/0036) and
Nottingham NHS Research Ethics Committee (reference 21/EM/0067).

## Data and code availability statement

The data used in this paper are not publicly available. Access to the UK
COVID-19 Vocal Audio Dataset may be requested from UKHSA
(DataAccess@ukhsa.gov.uk), and will be granted subject to approval and a data
sharing contract. To learn about how to apply for UKHSA data, visit:
https://www.gov.uk/government/publications/accessing-ukhsa-protected-
data/accessing-ukhsa-protected-data, see Budd et al. (2022) for more details.
The code used to analyse the data and generate the train/test splits will be
made available on the Alan Turing Institute GitHub repository.

## Acknowledgments

Authors in The Alan Turing Institute and Royal Statistical Society Health Data
Lab gratefully acknowledge funding from Data, Analytics and Surveillance
Group, a part of the UKHSA. This work was funded by The Department for Health
and Social Care (Grant ref: 2020/045) with support from The Alan Turing
Institute (EP/W037211/1) and in-kind support from The Royal Statistical
Society.

## References

  * Babic et al. (2021) Babic, B., Gerke, S., Evgeniou, T. and Cohen, I. G. (2021) Beware explanations from AI in health care.  Science, 373, 284–286.
  * Brown et al. (2020) Brown, C., Chauhan, J., Grammenos, A., Han, J., Hasthanasombat, A., Spathis, D., Xia, T., Cicuta, P. and Mascolo, C. (2020) Exploring automatic diagnosis of COVID-19 from crowdsourced respiratory sound data.  arXiv preprint arXiv:2006.05919.
  * Budd et al. (2022) Budd, J., Baker, K., Karoune, E., Coppock, H., Patel, S., Tendero Cañadas, A., Titcomb, A., Payne, R., Hurley, D., Egglestone, S., Butler, L., Mellor, J., Nicholson, G., Kiskin, I., Koutra, V., Jersakova, R., McKendry, R., Diggle, P., Richardson, S., Schuller, B., Gilmour, S., Pigoli, D., Roberts, S., Packham, J., Thornley, T. and Holmes, C. (2022) A large-scale and pcr-referenced vocal audio dataset for COVID-19.  arXiv preprint.
  * Coppock et al. (2021) Coppock, H., Jones, L., Kiskin, I. and Schuller, B. (2021) Covid-19 detection from audio: seven grains of salt.  The Lancet Digital Health, 3, e537–e538.
  * Coppock et al. (2022) Coppock, H., Nicholson, G., Kiskin, I., Koutra, V., Baker, K., Budd, J., Payne, R., Karoune, E., Hurley, D., Titcomb, A., Egglestone, S., Tendero Cañadas, A., Butler, L., Jersakova, R., Patel, S., Thornley, T., Mellor, J., Diggle, P., Richardson, S., Packham, J., Schuller, B., Gilmour, S., Pigoli, D., Roberts, S. and Holmes, C. (2022) Audio-based AI classifiers show no evidence of improved COVID-19 screening over simple symptoms checkers.  arXiv preprint.
  * Eyben et al. (2013) Eyben, F., Weninger, F., Gross, F. and Schuller, B. (2013) Recent developments in opensmile, the munich open-source multimedia feature extractor.  In Proceedings of the 21st ACM international conference on Multimedia, 835–838.
  * Han et al. (2021) Han, J., Brown, C., Chauhan, J., Grammenos, A., Hasthanasombat, A., Spathis, D., Xia, T., Cicuta, P. and Mascolo, C. (2021) Exploring automatic COVID-19 diagnosis via voice and symptoms from crowdsourced data.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 8328–8332. IEEE.
  * Han et al. (2022) Han, J., Xia, T., Spathis, D., Bondareva, E., Brown, C., Chauhan, J., Dang, T., Grammenos, A., Hasthanasombat, A., Floto, A. et al. (2022) Sounds of COVID-19: exploring realistic performance of audio-based digital testing.  NPJ digital medicine, 5, 1–9.
  * Laguarta et al. (2020) Laguarta, J., Hueto, F. and Subirana, B. (2020) COVID-19 artificial intelligence diagnosis using only cough recordings.  IEEE Open Journal of Engineering in Medicine and Biology, 1, 275–281.
  * Riley et al. (2020) Riley, S., Atchison, C., Ashby, D., Donnelly, C. A., Barclay, W., Cooke, G. S., Ward, H., Darzi, A., Elliott, P., Group, R. S. et al. (2020) Real-time assessment of community transmission (REACT) of SARS-CoV-2 virus: study protocol.  Wellcome Open Research, 5.
  * Rudin (2019) Rudin, C. (2019) Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead.  Nature Machine Intelligence, 1, 206–215.
  * UK Health Security Agency (2021) UK Health Security Agency (2021) Speak up and help beat coronavirus.  https://www.gov.uk/government/news/speak-up-and-help-beat-coronavirus-covid-19.
  * Watson (2022) Watson, D. S. (2022) Conceptual challenges for interpretable machine learning.  Synthese, 200, 1–33.
  * World Health Organization (2020) World Health Organization (2020) WHO Director-General’s opening remarks at the media briefing on COVID-19 - 16 March 2020.  https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---16-march-2020.  Accessed: 15 September 2022.