# Generative AI is already widespread in the Public Sector

Jonathan Bright1 Florence E. Enock1 Saba Esnaashari1 John Francis1 Youmna
Hashem1 Deborah Morgan1

( 1Alan Turing Institute
<EMAIL_ADDRESS>
)

###### Abstract

Generative AI has the potential to transform how public services are delivered
by enhancing productivity and reducing time spent on bureaucracy. Furthermore,
unlike other types of artificial intelligence, it is a technology that has
quickly become widely available for bottom-up adoption: essentially anyone can
decide to make use of it in their day to day work. But to what extent is
generative AI already in use in the public sector? Our survey of 938 public
service professionals within the UK (covering education, health, social work
and emergency services) seeks to answer this question. We find that use of
generative AI systems is already widespread: 45% of respondents were aware of
generative AI usage within their area of work, while 22% actively use a
generative AI system. Public sector professionals were positive about both
current use of the technology and its potential to enhance their efficiency
and reduce bureaucratic workload in the future. For example, those working in
the NHS thought that time spent on bureaucracy could drop from 50% to 30% if
generative AI was properly exploited, an equivalent of one day per week (an
enormous potential impact). Our survey also found a high amount of trust (61%)
around generative AI outputs, and a low fear of replacement (16%). While
respondents were optimistic overall, areas of concern included feeling like
the UK is missing out on opportunities to use AI to improve public services
(76%), and only a minority of respondents (32%) felt like there was clear
guidance on generative AI usage in their workplaces. In other words, it is
clear that generative AI is already transforming the public sector, but uptake
is happening in a disorganised fashion without clear guidelines. The UK’s
public sector urgently needs to develop more systematic methods for taking
advantage of the technology.

Keywords: Generative AI, public services, productivity

##  1 Introduction

The economic and societal impacts of generative artificial intelligence
(GenAI) systems have received significant attention [13], but few studies have
examined how such systems are used in the public sector, and how they are
impacting operational public sector workers, or ‘street-level bureaucrats’
[15]. Given their accessibility and rapid proliferation, answers to these
questions will be vital in helping us to safely and robustly actualise the
potential of GenAI technologies in the public sector. We seek to tackle these
questions and in this article.

GenAI systems have been made easily accessible, with features often available
for free or included within corporate software packages. Interfaces such as
ChatGPT or Microsoft Copilot represent tools that potentially any public
sector worker can potentially use in their work however they wish, only
needing an internet connection (though of course paid for versions of these
technologies are emerging - however even then subscription prices are modest).
This is a mode of AI deployment in the public sector which is completely
different from the usual centralised corporate procurement process, where
technologies are designed and deployed in a top down fashion. As such, wide
use of these systems and integration into existing workflows may present a
significant ‘bottom up’ transition in the nature of public sector work at the
micro-level, driven by the discretionary needs of street-level bureaucrats
[15]. Discretion has been found to have positive effects for service users,
and is inherently valued by bureaucrats [22], meaning this mode of technology
adoption might be more positive than that focussed on the top-down imposition
of systems. Given the rapid speed of growth over the past year, the extent of
GenAI use by public sector workers has understandably not yet been the subject
of considerable research. We therefore seek to understand why and how public
sector workers are using these systems, and more specifically, the impact of
these systems upon their work and their own perceptions of productivity.

Our approach is based on a survey of public sector workers in the fields of
healthcare, education, social care, and the emergency services within the UK.
Respondents were asked about their use of different types of AI, their trust
and understanding of these systems, as well as their concerns and optimism
about this technology. We find considerable use of GenAI within the public
sector, and high levels of optimism about the effect this technology might
have on the future of public service delivery. Despite these findings,
respondents also reported a lack of clarity regarding oversight, as well as
low awareness of guidance on appropriate use. Importantly, despite there being
wide variation in use cases for GenAI across professions, public service
professionals were positive about the ability for AI to enhance productivity,
and reported little worry that AI would eventually replace their current job.

##  2 Background: AI and productivity in the public sector

AI systems within public services seem to have the potential to increase
public sector capacity in a range of areas [17, 16]; and these systems are
already in increasingly widespread use within the public sector. Such systems
include, for example, the use of predictive analytic systems to support the
allocation of resources in the area of health and social care, or the use of
spatial analysis to create heat maps of residents’ proximity to services in
the area of planning and development [7]. Many of these technologies have a
productivity focus, with the aim being to free up the time of skilled public
sector workers. However, despite a number of promising, large-scale public
sector ‘digital transformation’ initiatives and policies across government,
recent UK Office for National Statistics (ONS) data has highlighted that total
public service productivity only grew by an average of 0.2% per year between
1997 and 2019, with several service areas completely static or seeing
diminishing growth [19]. In other words, up until today, the public sector has
not taken much advantage of the major changes in digital technology that have
been seen in the last two decades, or at least such changes have not been
translated into enhanced productivity.

The emergence of GenAI seems to create the opportunity to change this dynamic.
The widespread deployment of freely available generative systems with readily
accessible user interfaces places significant AI resources in the hands of
operational - increasingly time pressed - public sector workers, who often
spend a large amount of their day on bureaucracy; for instance Viechnicki and
Eggers found that the average civil servant spends up to 30% of their time on
documenting information and other basic administrative tasks [23]. Clearly,
automated support with such activities could support enhanced delivery of
services and potentially re-orient civil servants work around tasks that
require human intelligence (perhaps also resulting in higher job satisfaction
[6]). This is supported by work in the private sector: research carried out by
Brynjolfsson et al., has found that GenAI systems can positively impact
productivity, particularly for novice and low-skilled workers [9].

The potential for GenAI is increased, as we highlighted above, by the ease
with which it can be adopted. A necessary dimension of the majority of
existing and prior digital transformation projects is that they are
conceptualised, procured, and implemented by centralised public sector bodies
to address a defined problem. GenAI completely reverses this paradigm: its
emergence and accessibility means that anyone with an internet connection can
theoretically start to integrate it into their day to day working life. This
bottom-up deployment creates huge potential for rapid (though likely uneven)
adoption.

However, thus far we have only limited knowledge about the actual extent of
this uptake in the public sector. Some statistics from early 2023 indicated
that 8.2% of employees at global companies had used ChatGPT [2], while
statistics from the UK indicated that more than a quarter of adults had used
the tool [18]. But these statistics do not address the public sector. In terms
of public sector more specifically, a recent report from the Department for
Education showed that teachers were using it in a variety of capacities [3]
(though without measuring the actual extent of usage), and recent survey work
with the Canadian Federal Public Service found that 11.2% had used generative
systems for work purposes, with positive perceptions of the use of AI systems
for data processing [14]. However, in large part due to the rapid deployment
of such systems within the public realm, few studies have surveyed operational
public sector workers across sectors regarding their use of systems to
understand impacts upon perceptions of discretion, responsibility, and
productivity. Such comparative work across sectors [21] can support more
nuanced and robust understandings of the impacts of GenAI as a professional
tool.

In addition to understanding levels of use of the technology, it is also worth
understanding the extent to which people using the technology feel like they
have clear guidance around it. One of the critical potential disadvantages of
a ‘bottom-up’ technology is that its adoption becomes somewhat chaotic, with
different people using it in different ways, potentially undermining things
such as the duty towards public sector equality. Of course, some guidance does
exist. Earlier this year, the UK government published guidance to civil
servants on the use of GenAI [10] noting the risks around inputting sensitive
and personal data alongside the risks of bias and misinformation from these
systems. However, they also note that such systems can be helpful and assist
with the work of civil servants, underscoring that they ‘should be inquisitive
about new technologies, including generative AI tools.’ [10]. This government-
wide guidance has been followed by sector specific guidance from individual
departments (see e.g. [1]). However questions remain about whether
professionals are even aware this guidance exists, or the extent to which it
is useful in their day-to-day engagements with the technology. Again, this is
something we will seek to tackle in the present article.

##  3 Methods

Data collection for this work took place online in November 2023 through a
survey that was created and administered using
Qualtrics111https://www.qualtrics.com. Participants were recruited through
Prolific222https://www.prolific.com, and had to be based in the UK and over
the age of 18 to take part. Furthermore, anyone taking part in the survey had
to be working in one of five key public sector areas: the NHS, the emergency
services, social work, schools or universities (it is worth noting of course
that in the UK, universities are not entirely ‘public sector’ in the sense
that they are not owned by government, but they receive considerable amounts
of funding from the public sector). Participants were paid at an equivalent
rate of approximately £20 per hour for completing the survey. Participants
were not allowed to complete the survey more than once. An attention check was
included within the survey, with responses not recorded for those who failed
the check.

A total of 938 individuals completed the survey. Of the final 938 respondents,
a majority (67%) were female and most respondents reside within England (84%).
Respondents were fairly balanced across professions, with the largest number
of responses received from NHS workers (24%) and the fewest from emergency
service workers (15%). Time spent in the profession ranged from less than a
year to 48 years, with a mean of about 11 years (SD=9.1), meaning that we
capture the views of both junior and senior professionals. A full list of
demographics can be seen in Table 1.

Table 1: Demographics of Survey Respondents Category | n | frequency
---|---|---
Use Any AI | 306 | 32.62%
Use Generative AI | 209 | 22.28%
Use Decision-Suppport AI | 92 | 9.81%
Use Perceptive AI | 80 | 8.53%
Female | 634 | 67.59%
Age - Under 30 | 180 | 19.19%
Age - 30-34 | 174 | 18.55%
Age - 35-39 | 165 | 17.59%
Age - 40-44 | 129 | 13.75%
Age - 45-49 | 81 | 8.64%
Age - 50-59 | 151 | 16.1%
Age - 60 years and over | 58 | 6.18%
Profession - NHS | 228 | 24.31%
Profession - Schools | 220 | 23.45%
Profession - Universities | 187 | 19.94%
Profession - Social Care | 160 | 17.06%
Profession - Emergency Services | 143 | 15.25%
Country - England | 788 | 84.01%
Country - Scotland | 85 | 9.06%
Country - Northern Ireland | 22 | 2.35%
Country - Wales | 43 | 4.58%
Years in Profession - 0-5 | 262 | 27.93%
Years in Profession - 5-10 | 235 | 25.05%
Years in Profession - 10-20 | 262 | 27.93%
Years in Profession - 20+ | 179 | 19.08%

It is worth highlighting two key limitations of the survey. First, the sample
is not likely to be fully representative of the population of public sector
works in the UK. Although, as seen in Table 1, we have a good range of ages,
genders, and different types of seniority, we would of course expect there to
be bias in our sample towards those who make use of paid survey platforms.
Furthermore, although we select people working in different professions, we do
not know the professional role of people in our sample: for example someone
working in a school might be in an HR or IT function, rather than working
directly with children.

Respondents were asked about three different types of AI systems which they
may have encountered within their work, defined in Table 2. Asking about a
variety of systems allows us to contrast uptake levels between traditionally
top-down technologies and bottom-up ones. For each AI system, participants
were asked about their general knowledge and use of these systems in their
professions. If the participant indicated they used any of them, they were
then asked about their trust and understanding of these systems. For the
purpose of our research, we primarily focused on GenAI: tools such as ChatGPT
which generate text or images based on prompts from a user. We therefore
included some additional targeted questions around GenAI implementation in the
workplace. All participants were then asked about their perceptions of AI use
in the public sector, including areas of concern and optimism around what AI
will portend for the future of public services. A full breakdown of the survey
procedure can be found in the Appendix.

Table 2: AI Definitions Type of AI |  Definition |  Example
---|---|---
Generative AI |  These are systems which can create text or images on your behalf, often on the basis of specific prompts. |  ChatGPT, which can be used to help draft responses to emails or summarise text.
Decision-Support AI |  These are systems where machines are able to support decision-making by predicting, recommending, or prioritising outputs based on a set of inputs. |  The use of resource allocation systems in hospitals that help triage patients arriving in the emergency room based on the symptoms they present.
Perceptive AI |  These are systems where machines are able to process sensory information such as visual and auditory inputs. |  The computer vision technology used for facial recognition to assess whether an applicant’s passport photo meets the criteria for a passport, or to authenticate an individual’s identity.

The survey results were exported to R, where we examined participants’
responses using descriptive statistics. We conducted chi-square tests of
independence and one-way ANOVA tests to examine if there were significant
differences in survey responses based on participants’ characteristics.

##  4 Results

Figure 1: AI in the public sector

From Figure 1 we can see that GenAI use is already more widespread than
predictive and perceptive AI applications in every profession apart form the
emergency services, despite its nascent status as a consumer facing tool.
Compared to other forms of AI with a longer history, such as facial
recognition tools, GenAI has largely outstripped these other use cases in
terms of uptake by the surveyed professionals. Over one fifth of respondents
reported that they are currently using GenAI in their work, with respondents
from Universities (36.4%) and Schools (30.5%) reporting the highest uptake.
Respondents from the NHS (17%), Emergency Services (12%) and Social Care (11%)
reported lower uptake levels. The findings around awareness of other
colleagues using the technology are also striking: 70% of people in
universities know at least one colleague using the technology, with high
numbers for most of the other professions as well.

In terms of demographic correlates of use, men are around 50% more likely to
use the technology than women, while increasing age lowers the likelihood of
use (someone under 30 is about three times more likely to be using it than
someone over 60). People who have been in their job a long time are, however,
are just as likely to use GenAI as people who have only been there a short
amount of time, once age differences are taken into account. Furthermore,
GenAI is used frequently by those who have taken it up. Of those using the
technology, almost 60% are using it either on a daily or weekly basis in their
work.

Of the 209 respondents who reported that they use GenAI in their work, 132
(63%) specified ChatGPT as the system they use, while the next highest
specified system was Bard at 5%. Respondents from Universities (26.7%) and
Schools (19.6%) were most likely to have used ChatGPT in their work, while
respondents from Emergency Services (5.6%), NHS (8.8%), and Social Care (6.9%)
were significantly less likely ($F=13.4,p<0.001$) to have mentioned use of
ChatGPT. Notably, there was not a significant difference in uptake of ChatGPT
by age or number of years in profession, although there was a significant
difference by gender ($F=10.5,p<0.001$) , with male respondents (19.6%) more
likely to report using ChatGPT than female respondents (11.7%).

Figure 2: AI and productivity

Respondents were positive about the use of AI for enhancing productivity. For
example, more than 80% of respondents in the social care profession said it
enhanced their productivity, with other professions at similar levels. Figure
2 shows that people are especially positive about the potential of AI to
reduce bureaucracy in their work. The figure shows, on average, how much time
people spend on bureaucracy, versus how much time they believe they would be
spending if current AI technology was deployed effectively. All categories
show a considerable decrease: for example, respondents working for the NHS
indicated they spent almost 50% of their time on bureaucracy, but thought AI
use could reduce this to less than 30% (the equivalent of saving an entire day
of work every week). Respondents who reported that they use GenAI were 15%
more likely to agree that GenAI could reduce their time on bureaucracy.
Additionally, the expected reduction in time spent on administrative tasks
from GenAI tools was over 5% higher on average (an additional two hours of
time saved per week) from those who actively use these tools in their work. In
other words, those using the tools can really see a benefit to them, and also
can see potential to do more than is currently being done.

When asked how they were using generative systems, respondent comments
highlighted the utility of such systems to enhance productivity and
creativity, and to reduce the time taken to complete tasks. Some ways
respondents reported using GenAI were to draft emails, create examples and
lesson materials, write up reports, aid in writing code, and generating
synthetic data. Respondents specified a variety of ways in which GenAI was
able to increase their productivity and was particularly useful to managing
their workloads. One such response noted that, ‘I use ChatGPT, when I am
overwhelmed with work’, while another respondent mentioned that ‘as an IT
technician, I utilize ChatGPT to streamline customer support, providing quick
responses to common queries’. In terms of more creative tasks, one respondent
noted that ChatGPT was useful ‘to generate ideas for marketing content, blog
posts etc’, while a teacher reported that ChatGPT and Bard were useful to
‘generate ideas for improving my teaching and supporting students’. The
variety of responses received demonstrate the complexity and breadth of use of
GenAI systems as an individual tool across professions in the public sector.

Figure 3: Understanding of Generative AI

Respondents that use GenAI were overall quite trusting of AI technologies in
the workplace (see Figure 3). For all professions, more than 60% of GenAI
users said that they understood how these AI systems worked. A majority of
these respondents (over 60% in all cases, except for Universities) also
trusted the output of GenAI technology. Additionally, a large majority of
respondents agreed that they understand the risks associated with using GenAI
in their workplaces. Despite their use of GenAI, fewer respondents (53%) said
that they understand who is held accountable for the outputs produced by
systems like ChatGPT. Furthermore, most professionals felt that the guidance
around the use of GenAI in their workplace was not clear.

Concern about respondents’ jobs being replaced by AI was not widespread. Only
about 16% of respondents were worried that AI would eventually replace their
current job. This number was highest among people working at a university
(25%), although interestingly there was no significant difference in responses
between people that use or don’t use GenAI in their work. Overall, public
service professionals responded positively about the potential benefits of AI
use, with over 75% of respondents optimistic about how AI will improve public
services in the future. Respondents that use GenAI were significantly more
likely to be optimistic ($t=4.42,p<.001$) than other respondents, with 86%
reporting optimism, versus only 72% of non GenAI users. Despite this optimism,
most respondents (76%) agreed that the UK is missing out on opportunities to
use AI to improve public services. This number was significantly higher
($t=3.65,p<0.001$) for users of GenAI (85%) than for those who do not use AI
in their work (73%).

##  5 Discussion

Through a preliminary survey of public professionals in the UK, we find that
use of GenAI already prevalent within the public sector. Respondents had high
levels of optimism about the potential for GenAI to reduce the amount of time
they spend on bureaucracy, and they believe this technology can improve public
service delivery in the future. Despite these anticipated improvements to
productivity, public sector workers are not particularly worried that this
technology will eventually replace their jobs. Respondents also noted that
despite their usage of GenAI tools, there is a general lack of guidance on
appropriate use provided by their employers.

The results presented in this survey suggest that uptake of GenAI in the
public sector is proceeding relatively rapidly. Despite only being available
as a consumer facing product for around a year, GenAI use is more widespread
than other types of AI that have been around comparatively longer. However,
while uptake of GenAI is relatively high given the recent proliferation of
this technology, our results also demonstrate that not everyone is using the
technology. Indeed, in some public service professions, directly reported
uptake remains only around 10%. Considering the positive results reported by
those who are using the technology, this seems like an opportunity for the
public sector to push further, and ensure that it is rolled out more widely.

Of course, there may be barriers to more widespread uptake. First, it is
likely that not all jobs support easy implementation of GenAI tools, as
suggested by Halal et al. [24]. One area where further research is needed is
understanding the proportion of public sector workers who could benefit from
using these tools. Furthermore, there may also be reluctance to implement
GenAI in some areas of work. A recent survey of public attitudes to AI in the
UK found that the public holds nuanced views on what it regards as proper use
cases of AI[4]. The same survey found that while the general public views
efficiency and improved accessibility as a main advantage of AI, there is
worry about AI being used to replace professional judgements, such as within
hiring [4]. While uptake of GenAI is high, it could also be that public
attitudes towards this burgeoning technology are causing some public
professionals to be a bit more cautious about integrating GenAI tools into
their daily practice. Furthermore, we find that responsibility for the outputs
of GenAI is currently unclear to public professionals, which echoes
conclusions drawn from Brown, who charts the complications created by GenAI in
AI supply chains, and discusses potential frameworks for addressing this issue
[8]. Notably, issues of responsibility in workflows are not specific to AI, as
other work has noted the difficulties in attributing responsibility or legal
liability to other forms of complex supply chains [12]. Lack of clear lines of
responsibility, and a perception of a lack of clear guidance, could be a
further factor blocking uptake.

Despite these barriers, the levels of optimism we found around the technology
was striking. One reason for this could be precisely the bottom-up way in
which it has been adopted. GenAI provides enhanced personal agency in how it
is used, with the ability to tailor and personalise use cases to meet distinct
needs, rather than forcing users to make use of a system in a specific way.
Our findings around time spent on administrative tasks suggest that there is
widespread hope from professionals that AI will reduce the amount of time
spent on this type of work. Despite this, there is little worry that
professionals’ jobs will be on the line. The future imagined by these
respondent beliefs is one where AI will not eliminate human activity in the
public sector workforce, but instead free up workers to spend more of their
time on non-routine tasks. The high levels of reported optimism around the
technology’s potential to alleviate bureaucratic workloads echo similar
sentiments in the United States, where research found that the majority of
public sector professionals were optimistic about the role that AI can play in
‘improving bureaucratic efficiency’ [5]. Alternatively, if, as the NHS workers
in our sample predict, a full day of work currently taken up by administration
can be eliminated, current productivity levels could be maintained while
making a powerful case for moving to a four-day work week, the benefits and
risks of which have long been debated [11].

The widespread use of generative applications such as ChatGPT suggests that
public sector workers may have already started becoming reliant on these
technologies to automate common administrative tasks. This does raise the
question of the business model behind these technologies, and who will provide
them in the future. There remains a potential danger of a few key companies
with the resources to create powerful GenAI tools having a stranglehold over
the productivity of governments, and the workforce more broadly. However,
GenAI technology is also becoming more widely distributed and accessible to
create, with projects such as BLOOM [20] attempting to democratise access to
this technology. One key question for government moving forward will be around
whether it wishes to invest in creating its own language models, supported by
open source technology, or whether it will focus on procurement from
technology companies. Resolving this question will be crucial to the future of
generative AI uptake in the public sector.

## References

  * noa [a] Generative artificial intelligence (AI) in education - GOV.UK, a.  URL https://www.gov.uk/government/publications/generative-artificial-intelligence-in-education/generative-artificial-intelligence-ai-in-education.
  * noa [b] Statistics of ChatGPT & Generative AI in business: 2024 Report, b.  URL https://masterofcode.com/blog/statistics-of-chatgpt-generative-ai-in-business-2023-report.
  * noa [2023] AI helps out time-strapped teachers, says report.  _BBC News_ , November 2023.  URL https://www.bbc.com/news/education-67433036.
  * [4] Ada Lovelace Institute and The Alan Turing Institute.  How do people feel about AI? a nationally representative survey of public attitudes to artificial intelligence in britain.  URL https://www.adalovelaceinstitute.org/report/public-attitudes-ai/.
  * [5] Mike Alva.  Impact of AI on decision-makers.  URL https://info.kpmg.us/news-perspectives/technology-innovation/thriving-in-an-ai-world/ai-adoption-government.html.
  * [6] Jamie Berryhill, Kévin Kok Heang, Rob Clogher, and Keegan McBride.  Hello, world: Artificial intelligence and its use in the public sector.  URL https://www.oecd-ilibrary.org/content/paper/726fd39d-en.
  * [7] Jonathan Bright, Bharath Ganesh, Cathrine Seidelin, and Thomas M. Vogl.  Data science for local government.  ISSN 1556-5068.  doi:10.2139/ssrn.3370217.  URL https://www.ssrn.com/abstract=3370217.
  * [8] Ian Brown.  Expert explainer: Allocating accountability in AI supply chains.  URL https://www.adalovelaceinstitute.org/resource/ai-supply-chains/.
  * [9] Erik Brynjolfsson, Danielle Li, and Lindsey Raymond.  Generative AI at work.  URL http://arxiv.org/abs/2304.11771.
  * [10] Cabinet Office and Central Digital and Data Office.  Guidance to civil servants on use of generative AI.  URL https://www.gov.uk/government/publications/guidance-to-civil-servants-on-use-of-generative-ai/guidance-to-civil-servants-on-use-of-generative-ai.
  * [11] Timothy T. Campbell.  The four-day work week: a chronological, systematic review of the academic literature.  ISSN 2198-1639.  doi:10.1007/s11301-023-00347-3.  URL https://doi.org/10.1007/s11301-023-00347-3.
  * [12] Jennifer Cobbe, Michael Veale, and Jatinder Singh.  Understanding accountability in the algorithmic supply chain.  In _2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’23)_. ACM.  doi:10.1145/3593013.3594073.  URL http://dx.doi.org/10.1145/3593013.3594073.
  * [13] Yogesh K. Dwivedi, Laurie Hughes, Elvira Ismagilova, Gert Aarts, Crispin Coombs, Tom Crick, Yanqing Duan, Rohita Dwivedi, John Edwards, Aled Eirug, Vassilis Galanos, P. Vigneswara Ilavarasan, Marijn Janssen, Paul Jones, Arpan Kumar Kar, Hatice Kizgin, Bianca Kronemann, Banita Lal, Biagio Lucini, Rony Medaglia, Kenneth Le Meunier-FitzHugh, Leslie Caroline Le Meunier-FitzHugh, Santosh Misra, Emmanuel Mogaji, Sujeet Kumar Sharma, Jang Bahadur Singh, Vishnupriya Raghavan, Ramakrishnan Raman, Nripendra P. Rana, Spyridon Samothrakis, Jak Spencer, Kuttimani Tamilmani, Annie Tubadji, Paul Walton, and Michael D. Williams.  Artificial intelligence (AI): Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy.  57:101994.  ISSN 02684012.  doi:10.1016/j.ijinfomgt.2019.08.002.  URL https://linkinghub.elsevier.com/retrieve/pii/S026840121930917X.
  * [14] Global Government Forum.  Attitudes to AI in the canada public service: Headline survey results.  URL https://www.globalgovernmentforum.com/wp-content/uploads/Attitudes-to-AI-Canada-Public-Service-Survey-2023.pdf?swcfpc=1.
  * [15] Michael Lipsky.  Street-level bureaucracy: dilemmas of the individual in public services.  OCLC: ocn503596064.
  * [16] Helen Margetts.  Rethinking AI for good governance.  151(2):360--371.  ISSN 0011-5266, 1548-6192.  doi:10.1162/daed_a_01922.  URL https://direct.mit.edu/daed/article/151/2/360/110613/Rethinking-AI-for-Good-Governance.
  * [17] Helen Margetts and Cosmina Dorobantu.  Rethink government with AI.  568(7751):163--165.  doi:10.1038/d41586-019-01099-5.  URL https://www.nature.com/articles/d41586-019-01099-5.
  * Milmo and editor [2023] Dan Milmo and Dan Milmo Global technology editor.  More than a quarter of UK adults have used generative AI, survey suggests.  _The Guardian_ , July 2023.  ISSN 0261-3077.  URL https://www.theguardian.com/technology/2023/jul/14/more-than-quarter-of-adults-have-used-generative-ai-artifical-intelligence-survey-suggests.
  * [19] Office for National Statistics.  Public services productivity.  URL https://www.ons.gov.uk/economy/economicoutputandproductivity/publicservicesproductivity/articles/publicserviceproductivityuk/1997to2022.
  * [20] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, and et al.  BLOOM: A 176b-parameter open-access multilingual language model.  URL http://arxiv.org/abs/2211.05100.
  * [21] Friso Selten, Marcel Robeer, and Stephan Grimmelikhuijsen.  ‘just like i thought’: Street-level bureaucrats trust AI recommendations if they confirm their professional judgment.  83(2):263--278.  ISSN 0033-3352, 1540-6210.  doi:10.1111/puar.13602.  URL https://onlinelibrary.wiley.com/doi/10.1111/puar.13602.
  * [22] Lars Tummers and V.J.J.M. Bekkers.  Policy implementation, street-level bureaucracy and the importance of discretion.  16(4):527--547.  URL https://ssrn.com/abstract=2289338.
  * [23] Peter Viechnicki and William D. Eggers.  How much time and money can AI save government?  URL https://www2.deloitte.com/content/dam/insights/us/articles/3834_How-much-time-and-money-can-AI-save-government/DUP_How-much-time-and-money-can-AI-save-government.pdf.
  * [24] William Halal, Jonathan Kolber, and Owen Davies.  Forecasts of AI and future jobs in 2030: Muddling through likely, with two alternative scenarios.  21(2):83--96.  doi:DOI:10.6531/JFS.2016.21(2).R83.  URL https://jfsdigital.org/wp-content/uploads/2017/01/JFS212Final（已拖移）-6.pdf.

##  Appendix A Appendix

###  A.1 Survey Procedure

For each participant, we first collected standard information about age and
gender, along with public service profession, time spent in profession and
time spent in current role. Participants next read a short definition of
perceptive AI systems and were then asked if they were aware of such systems
in use in their area of work (Yes/No/I don’t know). If participants responded
‘Yes’ to awareness, they were then asked whether they make use of any of these
perceptive AI systems (Yes/No/I don’t know). If they responded ‘Yes’ to using
such systems, they were asked to provide the name(s) or a brief description of
the system(s) they use in a few words using a free text box, and they were
then asked how often they use such systems (Daily/Weekly/Every month/Only once
or twice before). In the next block, participants read a short definition of
decision-support AI systems and then responded to the same questions about
awareness, use and frequency of use as described for perceptive AI. Following
these questions, participants read a short definition of generative AI systems
and again responded to the same questions about awareness, use and frequency
of use.

If participants indicated that they use generative AI systems, they were then
asked a follow up set of questions in which they were asked to indicate
agreement with the following statements: ‘My workplace has clear guidance on
how to use generative AI systems’; ‘I trust the outputs given by the
generative AI systems I use’; and ‘Using generative AI has changed the scope
of my day-to-day work’ (Strongly agree/Agree/Neither agree nor
disagree/Disagree/Strongly disagree/I don’t know).

Next, if participants had indicated that they use more than one type of the
three AI systems asked about, they were then asked to indicate which one they
use most frequently (Perceptive/Decision support/Generative).

If participants indicated that they had used at least one of the three system
types before, they were presented with another set of follow up questions
about their use and understanding of AI used in their current role. If they
had indicated they use more than one system, they were asked to consider the
one they use most often. Participants were asked to indicate agreement with
the following four statements: ‘I am confident I understand how the AI systems
I use work’; ‘I feel AI systems enhance my productivity’; ‘I understand the
risks associated with using this AI system in my day-to-day work’; ‘I
understand who is held accountable for outputs made by AI systems used in my
work’ (Strongly agree/Agree/Neither agree nor disagree/Disagree/Strongly
disagree/I don’t know).

All participants were then asked to indicate approximately what percentage of
their working week they typically spend on bureaucracy and administration, on
a sliding scale from 0 to 100, and whether they think AI has the potential to
decrease the amount of time spent on such tasks (Yes/No/Not sure). If they
answered Yes, they were asked to indicate what percentage of the time spent on
administrative tasks they think AI can help reduce on a sliding scale from 0
to 100.

All participants were then asked a final set of questions relating to their
general perceptions of AI use in the public sector. Participants were asked to
indicate level of agreement with the following three statements: ‘I am
concerned that AI will eventually replace my current job’; ‘I think we are
missing out on the opportunities to use AI to improve public services’; ‘I am
optimistic about how AI will improve public services in the future’ (Strongly
agree/Agree/Neither agree nor disagree/Disagree/Strongly disagree/I don’t
know).

Participants then responded to a simple attention check at the end of the
survey and were then thanked and redirected back to Prolific for payment.

###  A.2 Funding

This work was supported by Towards Turing 2.0 under the EPSRC Grant
EP/W037211/1 and The Alan Turing Institute.


# Seamless prediction of high-impact weather events:
a comparison of actionable forecasts

Zied Ben Bouallègue  European Centre for Medium-Range Weather Forecasts

###### Abstract

A new index for high-impact weather forecasting is introduced and assessed in
comparison with the well-established extreme forecast index (EFI). Two other
ensemble summary statistics are also included in this comparison study: the
shift-of-tail and a standardised ensemble mean anomaly. All these forecasts
are based on the same ingredients: the ensemble forecast run at the European
Centre for Medium-Range Weather Forecasts and the corresponding model
climatology derived from a set of reforecasts. The new index emerges from
recent developments in forecast verification of extreme events: it is derived
as a consistent forecast with the diagonal score, a weighted version of the
continuous ranked probability score targetting high-impact events. In this
study, we emphasise the importance of forecast discretisation for
communication purposes and decision-making. A forecast is actionable in the
situation where a user can decide to take action when a threshold is exceeded
by the forecast. Forecast verification is performed to assess both the
potential skill of the different indices as well as their specific skill as
actionable forecasts. Among the investigated actionable forecasts, the new
proposed index demonstrates the strongest discrimination power, in particular
at longer lead times, paving the way for seamless predictions of high-impact
weather across time ranges.

##  1 Introduction

Ensemble weather forecasts consist of a set of weather scenarios that capture
a range of equally probable (or at least possible) outcomes. Ensembles serve
as a basis for quantifying forecast uncertainty and generating probabilistic
forecasts. Our ability to predict high-impact weather events using ensemble
forecasts has improved over the past decades (Ben Bouallègue _et al._ , 2019).
However, the use and communication of probabilistic forecasts for effective
decision-making is still in its infancy, in particular when it comes to “low-
probability, high-impact” events (Fundel _et al._ , 2019).

At the beginning of the century, the extreme forecast index (EFI) was
introduced by Lalaurette (2003) to provide support for generating early
warnings of extreme events. This index highlights situations related to
“unusual” forecasts by comparing an ensemble forecast distribution with a
model climate distribution. The EFI is an indicator of the likelihood of an
extreme event but doesn’t provide information about the intensity of the
event. For this reason, the EFI is often complemented by the shift-of-tail
(SOT), a normalised quantile forecast using the climatological quantiles for
normalisation (Zsoter, 2006).

Several studies have illustrated that EFI has predictive skill not only for
raising early awareness of extreme precipitation (Lavers _et al._ , 2016,
2017, 2018, in particular in case of severe convection, Tsonevsky _et al._
2018), but also for the early warning of extreme winds and extreme windstorms
(Petroliagis and Pinson, 2014; Boisserie _et al._ , 2016). SOT predictive
performance is less explored in the literature but seems to show a comparable
level of skill as the EFI (Boisserie _et al._ , 2016; Raynaud _et al._ ,
2018). Another ensemble summary statistic has also been suggested for the
early detection of extreme weather: the ensemble mean anomaly forecast (ANF,
Guan and Zhu, 2017). ANF appears highly correlated to the EFI for temperature
with slightly higher skill in predicting extreme cold events.

In this study, we introduced and assessed a new index for high-impact weather
forecasting. We are building on the concept of crossing-point forecast (CPF)
that has emerged from recent developments in forecast verification of extreme
events (Ben Bouallègue, 2021). A CPF is a consistent forecast with the
diagonal score which is a weighted version of the continuous ranked
probability score (Ben Bouallègue _et al._ , 2018). The diagonal score is
routinely used at ECMWF for assessing the performance of the ensemble
forecasting system (Haiden _et al._ , 2021), but the underlying actionable
forecast has not been exploited so far.

An actionable forecast is defined as a forecast discretised with a set of
decision thresholds such that a user can decide to take action when a
threshold is exceeded by the forecast. Seamless forecasting across lead times
would rely on actionable forecasts that could be used across time ranges,
without the need to change the forecast definition or the set of decision
thresholds with the forecast lead time. In the following, the focus is on
actionable forecasts derived from an ensemble prediction system.

The CPF, like the EFI, is a summary statistic of both the ensemble forecast
and its model climatology. In the CPF case, the focus is set on the
intersection point between the two cumulative probability distributions. This
intersection marks the limit between the situation where the risk of an event
is higher in general (in the climatology, with no specific information about
the current weather situation) or in particular for a given day (in the
forecast, given the information available at the start of the prediction). The
CPF is a probabilistic forecast which takes value between 0 and 1 so one can
use it as a forecast index. Here, we explore the CPF performance as a forecast
index for extreme daily precipitation events.

This paper is organised as follows: a definition of the different types of
forecasts compared in this study is provided in Section 2, two case studies
are shown and commented on in Section 3, forecast intrinsic characteristics
and their interpretation are discussed in Section 4, general verification
results are presented in Section 5 followed by a focus on actionable forecasts
for seamless prediction of extreme events in Section 6 before to conclude in
Section 7.

##  2 From an ensemble forecast to actionable forecasts

For high-impact weather forecasting, a set of ensemble-based indicators can be
generated by comparing an ensemble forecast with a local model climatology. A
link between forecast intensity and potential impact is made by interpreting a
forecast in the context of a local climate: the climatological frequency of an
event serves as a measure of its rareness and its potential impact. Also, the
local variability of the weather can serve to contextualise the uncertainty of
a forecast (as measured by the ensemble spread, for example). Ensemble-derived
indicators become actionable forecasts when discretised with a set of decision
thresholds for visualisation and decision-making purposes as discussed in
Section 6.

In the following, forecast and climatology take the form of cumulative
probability distributions denoted $F$ and $G$, respectively. The forecast
empirical distribution is derived from the 50-member ensemble forecast run
operationally at ECMWF while the climate empirical distribution is derived
from reforecasts which consist of 10 members run twice weekly over the past 20
years (Lalaurette, 2003). A local climatology is built using all reforecasts
available within a time window of $\pm$2 weeks with respect to the validity
date of the forecast.

The climate distribution that reflects the model climatology is referred to as
the M-climate in opposition to the observed climate that is estimated based on
observations (and used for verification). In any case, climatology is meant
here in a local sense both in space and time: a climatology distribution is
associated with a given grid point (or point observation) and a given day of
the year. In practice, the empirical climate distributions are described by
percentiles from 0% (minimum) to 100% (maximum) with 1% intervals.

In this study, we compare 4 different types of forecasts, all derived from
both $F$ and $G$. CPF is based on the intersection point between $F$ and $G$,
EFI is an integral of $F$ with respect to $G$, SOT focuses on one percentile
of the tail of $F$ compared with the tail of $G$, while ANF is a standardised
mean of $F$ using the first and second moments of $G$. We provide below a
formal definition of each of these ensemble-derived forecasts.

###  2.1 Crossing-point forecast

Consider the single crossing condition: $F$ and $G$ satisfy the single
crossing condition if there exists a $y^{\star}$such that

$\forall x,x\geq y^{\star}\Longrightarrow F(x)\geq G(x)$ (1)

and

$\forall x,x\leq y^{\star}\Longrightarrow F(x)\leq G(x).$ (2)

When $y^{\star}$ exists, the crossing-point forecast is defined as:

$\text{CPF}:=G^{-1}(y^{\star})$ (3)

and is interpreted as a probabilistic worst-case scenario (Ben Bouallègue,
2021). CPF corresponds to the most extreme forecast event such that the risk
in the forecast is larger than the climate risk. This event is expressed in
terms of a quantile of the climate distribution. This quantile $q$ can easily
be translated in terms of a local return period $r$. For example, if $q=0.95$,
$r=20$ years, if $q=0.99$, $r=100$ years, and so on.

We use synthetic data to illustrate the concept of CPF in Fig. 1. While the
plot on the left represents a canonical example, the plot on the right
illustrates the difficulty of dealing with bounded variables. For
precipitation (or wind), forecast and climate cdfs overlap at the point-mass
0. In that case, in practice, CPF is derived by scanning $F(x)$ and $G(x)$
with increasing $x>0$ (from left to right) until an intersection point is
found (such that $F$ crosses $G$ from F<G to F>G, and not the other way
around). When no intersection exists, CPF takes the value 0 or 1 depending on
whether the $F$ is always above or below $G$, respectively.

Figure 1:  Synthetic examples of CPF in the case of (a) temperature-like
distributions and (b) precipitation-like distributions with $F$ in red and $G$
in blue.

###  2.2 Extreme forecast index

In Zsoter (2006) and Tsonevsky _et al._ (2018), the ensemble forecast index
EFI is defined as:

$\text{EFI}:=\frac{2}{\pi}\int_{0}^{1}\frac{p-F(p)}{\sqrt{p(1-p)}}\text{d}p$
(4)

where $F(p)$ is the proportion of the ensemble members lying below the
$p^{th}$-percentile of the climate distribution. The denominator helps provide
more weight to the tails of the distribution. The index takes values in
[-1,1]. The closer the EFI values to -1 or 1, the more abnormal the ensemble
forecast and the more likely an extreme event.

###  2.3 Shift of tails

The shift of tails (SOT) is designed to compare the tails of $F$ and $G$. For
the upper tail, the SOT is defined as:

$\text{SOT}:=-\frac{G^{-1}(0.99)-F^{-1}(0.9)}{G^{-1}(0.99)-G^{-1}(0.9)}$ (5)

with $G^{-1}(\tau)$ and $F^{-1}(\tau)$ the $\tau$%-quantile of the climate and
forecast distribution, respectively. When SOT is greater than 0, it means that
at least 10% of the ensemble members are beyond the 99${}^{\text{th}}$
M-climate percentile. The larger the SOT, the further these 10% are from the
M-climate. SOT was designed as a complementary forecast to the EFI, conveying
a piece of information about the intensity of the expected extreme event.

###  2.4 Anomaly forecast

Anomaly forecasting focuses on the departure of the ensemble mean with respect
to the climatology (Guan and Zhu, 2017). The anomaly forecast (ANF) is here
defined as the standardised ensemble mean anomaly with respect to the
M-climate mean. The standardisation is based on the standard deviation of the
M-climate distribution. ANF is computed as follows:

$\text{ANF}:=\frac{\mu(F)-\mu(G)}{\sigma(G)+k}$ (6)

where $\mu$ and $\sigma$ are the mean and standard deviation functions,
respectively, and $k$ is a constant set to 1 for precipitation (to avoid
numerical issues with ANF computation for arid areas), 0 for temperature for
example.

##  3 Case studies

Let’s illustrate similarities and specificities between these difference
forecasts with 2 case studies. We compare forecasts available 6 days and 1 day
ahead of the event (lead time day 6 and day 1, respectively) but valid for the
same dates. A subjective assessment of the forecast performance at day 6 can
be achieved by using the forecast at day 1 as a proxy of the ‘truth’. The 2
case studies are related to high-impact events that have affected Europe over
Summer 2021.

###  3.1 Example 1: heavy precipitation over Belgium and Germany in July 2021

Fig. 2 shows CFP, EFI, SOT, and ANC of daily precipitation valid on July 14,
2021. Extreme rainfall on that day led to flash floods and large-scale
flooding that had a catastrophic impact on parts of Belgium and western
Germany.

Forecasts 6 days ahead of the event are shown in the left panels: CPF has a
large-scale signal with high values over the Atlantic and Western/Central
Europe in Fig. 2(a) while EFI has a weak signal (if any) over the same areas
in Fig. 2(c). Forecasts one day before the event are presented in the right
panels. We see a convergence of CPF and EFI towards a similar solution with
the strongest signal over the impacted areas in Figs 2(b) and 2(d).

In this first example, the following general pattern emerges when comparing
CPF and EFI:

  1. 1.

CPF displays a strong signal at a longer lead time but the signal is large-
scale and noisy,

  2. 2.

EFI presents a weak signal or no signal at all at longer lead times,

  3. 3.

both forecasts converge at shorter lead times.

Regarding SOT and ANC, they have similar characteristics as EFI. Their signal
appears smooth and/or weak at longer lead times in Figs 2(e) and 2(g),
respectively, while a larger and sharper signal emerges closer to the event in
Figs 2(f) and 2 (h).

Figure 2:  Daily precipitation forecasts valid on July 14, 2021: (a,b) CPF,
(c,d) EFI, (e,f) SOT, and (g,h) ANC generated 6 days ahead of the event (left)
and 1 day ahead of the event (right).

###  3.2 Example 2: Convective precipitation over London in July 2021

Fig. 3 compares actionable forecasts for precipitation on July 25, 2021. On
that date, flash floods were reported in the London area after intense
precipitation in South-East England. With respect to the previous example,
this example deals with a more localised event, driven by convective activity.

Similarly to the previous example, we observe the following development with
the forecast lead time as we approach the validity time: a noisy and large-
scale CPF signal becomes more accurate while a smooth and light EFI signal
evolves into a stronger and more focused signal. As shown in Figs 3(a) and
3(c), the main differences between CPF and EFI are at longer lead times when
the uncertainty in the ensemble forecast is large and the ensemble mean has
not significantly departed from the climatological mean.

In this example, we can also notice differences between CPF and EFI at day 1,
in particular over England, by comparing Figs 3(b) and 3(d). Due to the
convective nature of the intense precipitation over South-East England on that
day, the predictability of the event was quite low. This low level of
predictability was appropriately captured by the ensemble: the ensemble spread
was large (not shown) with only some members predicting intense rain. As the
uncertainty remains high even at a shorter lead time, the EFI never reached
high values (greater than 50%) even at day 1. The CPF signal over England was
strong already at day 6 but much more refined over the impacted area at day 1.

Figure 3:  Same as Fig. 2 but for forecasts valid on July 25, 2021.

##  4 Forecasts intrinsic characteristics and interpretation

###  4.1 Correlation between forecasts

Following the visual inspection and comparison of CPF, EFI, SOT, and ANC in
Figs 2 and 3, the forecasts are now compared quantitatively in terms of
correlation. The level of similarity between the different forecasts is
measured with the rank correlation coefficient (also known as Kendall-$\tau$
correlation coefficient). The results are shown in Fig. 4.

The correlation between CPF and EFI is 0.94 at day 1 but decreases almost
linearly with the forecast lead time to reach 0.86 at day 6. Also, the
correlation of CPF with ANF and SOT as shown in Fig. 4(a) is much smaller than
the correlation of EFI with respect to the same quantities as shown in Fig.
4(b). These results suggest that the signal captured by CPF has different
characteristics than the one exhibited by EFI, SOT, and ANF.

Figure 4:  Averaged rank correlation coefficients between daily precipitation
forecasts over Europe during Summer 2021. Correlation of (a) CPF and (b) EFI
with respect to one another, SOT, and ANF. The grey lines showing the
correlation between CPF and EFI are identical in both panels.

The correlation of EFI with ANF is close to 0.9 for all lead times. Because
ANF is independent of the ensemble spread, this result confirms that EFI is
mainly driven by changes in the ensemble mean with occasional modulations by
the ensemble spread. EFI sensitivity to changes in the ensemble mean and the
ensemble spread is discussed more thoroughly in Dutra _et al._ (2013). The
correlation of EFI with SOT is also high and slightly increases with lead
time. SOT and ANF are themselves highly correlated (not shown) which can be
explained by the heteroscedastic nature of the precipitation distributions:
variations in the ensemble mean and in quantiles tend to be strongly linked.

###  4.2 Forecast distribution and forecast lead time

One key difference between the 4 types of forecasts investigated here is their
evolution with forecast lead times as illustrated in the 2 case studies in
Figs 2 and 3. To support a discussion on this point, we now analyse the
forecasts’ overall distributions during Summer 2021 for the European domain.
Fig. 5 compares the distributions at day 1 and day 6 for each forecast type.

CPF exhibits, in general, larger values at longer lead times as shown in Fig.
5(a). Indeed, the category closer to 1 is the most populated at day 6. At day
1, the distribution of CPF values tends to resemble a uniform distribution:
the strong signal at longer lead times in CPF are tempered as we approach to
the forecast validity time.

In contrast, large values of EFI, SOT, and ANC are issued predominantly at
shorter lead times as shown in Figs 5(b,c,d), respectively. The forecast
uncertainty has a smoothing effect on these high-impact weather indicators
such that large values are enabled only when the ensemble spread is small.
This characteristic is a limiting factor to communicating effectively low
probability, high-impact events as discussed in Section 6.

###  4.3 Forecasts interpretation

CPF and EFI are both bounded forecasts taking values in [0,1] and [-1,1]
respectively but they follow different interpretations. EFI is an index, a
number between -1 and 1 with no known statistical meaning attached to a single
EFI value. Its interpretation is the following: the larger the EFI values the
higher the risk of a high-impact event. The absence of a stringent statistical
interpretation is an asset for use by a wider audience. However, one important
limitation of the EFI is that no information is conveyed with this index about
the event itself, its intensity, or its uncertainty as already pointed out in
previous studies (Neal _et al._ , 2014; Boisserie _et al._ , 2016).

In contrast, the CPF can be interpreted as an index but has also a meaning in
statistical terms: it is a quantile level of a climate distribution.
Therefore, CPF values are directly related to local return periods (while EFI
is somehow indirectly related to local return periods). For example, a CPF
value of 0.95 corresponds to an event with a return period of 20 years with
respect to the local climate. By expressing the forecast in return periods (or
quantile level), systematic model errors are also automatically corrected
while bypassing the need for a post-processing step involving observations
(Fundel _et al._ , 2010; Prates and Buizza, 2011). In other words, a 20-year
return event derived from a model climatology can be interpreted by the users
based on an observed climatology.

We should also emphasise that a finer discretisation and a more accurate
representation of the distribution tails would be necessary for a focus on
very extreme events. So far, we have used a climatology described with 1%
quantile intervals and a forecast empirical distribution based on 50 ensemble
members. The relatively small ensemble size hinders an accurate estimation of
extreme quantiles of the forecast distribution (Leutbecher, 2019; Tempest _et
al._ , 2023). A finer discretisation of the tails could be achieved with the
help of statistical methods based on extreme value theory and a more accurate
representation of the forecast distribution tail by increasing the number of
ensemble members by leveraging, for example, the potential to generate large
ensembles with data-driven models based on machine learning (Ben Bouallègue
_et al._ , 2023).

Figure 5:  Forecast distribution at day 1 (red) and day 6 (blue) over Europe
for Summer 2021: (a) CPF, (b) EFI, (c) SOT and (d) ANC of daily precipitation.
Note the logarithmic scale of the y-axis in (c) and (d). CPF is the only type
of forecasts exhibiting stronger values at longer lead times.

##  5 General performance characteristics

Assessing forecast performance for high-impact events is challenging because,
by definition, these events are extreme and rare111Exposure and vulnerability
are not considered in this study.. Here, we consider relative rather than
absolute extreme events: we focus on events with a local return period of 20
years for a given point in space and time. In other words, we focus on
threshold-exceedance events where a threshold is defined as the
95${}^{\text{th}}$ percentile of the local climate distribution. Following
this approach and using synoptic observations as a reference, we mimic the
standard EFI verification setup used to assess and communicate EFI performance
as an ECMWF headline score (Tsonevsky _et al._ , 2018; Haiden _et al._ ,
2021). Here, the verification sample covers Europe over Summer 2021.

The forecasts that we compare here differ by their very own nature. For
example, CPF is a summary statistic consistent with a proper score while EFI
is an index with no statistical meaning. Aiming at a fair comparison, CPF and
EFI are both considered actionable forecasts in the form of an index for the
prediction of high-impact weather events. Both indices follow the same
interpretation: a larger CPF (or EFI) value indicates a higher risk of a high-
impact event materializing. Similarly, for SOT and ANC, a larger value in the
forecast is interpreted as a higher risk of a precipitation event without
directly taking into account the actual numerical value of the forecast.

Forecast performance is explored using contingency tables. A decision
threshold is applied to the forecast to transform it into a binary (yes/no)
forecast while an observation is transformed into binary observation using a
climate quantile as an event-threshold. For the discretisation of the
forecast, a set of decision thresholds needs to be defined for each forecast
type. It follows that contingency tables are populated for each decision
threshold of each forecast type. Eventually, hit rate ($H$) and false alarm
rate ($F$) are estimated from these contingency tables to derive diagnostic
plots and summary verification metrics. In particular, Relative Operating
Characteristics (ROC) curves and economic value plots are generated and
commented on below.

###  5.1 Potential discrimination ability

Discrimination is assessed within the ROC framework. A ROC curve is a set of
points ($F$,$H$) for increasing decision-thresholds (see Ben Bouallègue and
Richardson, 2022, and references within). Potential discrimination is assessed
by using a large number of decision thresholds in order to span all possible
forecast values222Alternatively, one could also examine distribution-fitted
ROC curves (see for example Ben Bouallègue and Richardson, 2022), but this
approach is not explored here.. Here we use 500 different decision thresholds
equally spaced between the minimum and the maximum possible value of each
forecast type. We talk about “potential” discrimination because, in practice,
500 different forecast values could not be distinguished on a map for
decision-making.

ROC curves based on 500 decision thresholds are shown in Fig. 6 for CPF, EFI,
SOT, and ANF at 3 different lead times. The ROC curves are similar for all 4
types of forecasts indicating similar levels of potential discrimination
ability, noting that CPF is slightly better and SOT slightly worse at the
shorter lead time. Recalling the correlation coefficients in Fig. 4, one could
infer some level of complementarity between the different types of forecasts
that could be further exploited in a post-processing framework.

Figure 6:  ROC curves assessing the potential discrimination ability of CPF,
EFI, ANF, and SOT for daily precipitation exceeding the 95% climate
percentile. Resulst for forecast lead times (a) day1 , (b) day 3, and (c) day
6.

###  5.2 Discriminating between moderate and heavy precipitation events

We propose now to compare the potential ability of the 4 different types of
forecasts to distinguish between “moderate” and “heavy” precipitation events.
We consider the situation where “wet” areas (with observed precipitation) can
be identified in advance and assess the forecast’s ability to pinpoint
subareas with heavy precipitation.

In a ROC framework, the forecast is conditioned on the observation. For
example, $H$, the hit rate, is an estimate of the probability of a correct
forecast given that the event actually materialised. From a theoretical point
of view, discrimination is assessed based on a likelihood/base-rate
factorization of the forecast/observation joint probability distribution
(Murphy and Winkler, 1987). For this reason, the verification dataset can be
stratified as a function of the observations while preserving the
interpretability of the results333 Note that a stratification as a function of
the observation only would violate the necessary condition for proper score
estimations..

Here, we assess whether the forecast is able to discriminate between extreme
precipitation events (observations exceeding the 95% percentile) among lighter
precipitation events (observation exceeding the 70% percentile of the local
climate). For this purpose, we build contingency tables from a verification
dataset that contains only cases where the observation exceeds the 70%
percentile of the climate distribution.

The results of this conditional verification exercise are shown in Fig. 7. CPF
and SOT appear better than ANF and EFI in this verification setting
particularly at lead times day 3 and day 6. EFI is the type of forecast
showing the smallest ability to discriminate between extreme and less extreme
precipitation events, at these time ranges.

Figure 7:  Potential discrimination ability to distinguish between “moderate”
and “heavy” precipitation events. Same as Fig. 6 but for the discrimination
among the observed cases where precipitation exceed the 70% climate
percentile.

###  5.3 Potential economic value

The economic value of a forecast is estimated with the help of a standard
cost-loss model. We consider the case where a forecast user can mitigate the
loss associated with the occurrence of a weather event by taking preventive
action. The user decides to take action or not based on an actionable
forecast. The key parameters of this model are 1) the cost of taking action
and 2) the loss encountered if no protective action is taken but the event
occurs. The cost-loss ratio characterises a user in the sense that it reflects
a specific user-defined application. Building on this model, the so-called
potential economic value is derived from the elements of the contingency table
(Richardson, 2000, 2011).

When considering extreme events, we can hypothesise that their monetary impact
is generally larger than the cost of preventive action. In this context, we
can recall that, by design, the diagonal score (the score consistent with CPF)
links the cost-loss ratio of a user with the rarity of the corresponding
event: the rarer the event, the smaller the cost-loss ratio. More precisely,
the CPF is designed as an optimal forecast for a user whose cost-lost ratio
varies linearly with the event base rate. This relationship is purely
theoretical but helped derive a score with the useful meta-property of
equitability (see Ben Bouallègue _et al._ , 2018, for more details).

Fig. 8 shows the potential economic value of CPF, EFI, SOT, and ANC as a
function of the user’s cost-loss ratio. Here again, we use 500 decision
thresholds and results are drawn from the same contingency tables as for the
plots in Fig. 6 A log scale is used on the X-axis to emphasize forecast
performance for users with a small cost-loss ratio. The user with a cost-loss
ratio equal to the event base rate is indicated with a vertical line. By
construction, the maximum forecast value is always reached precisely for this
cost-loss ratio.

CPF has a similar value as EFI for most users. Users that should prefer EFI to
CPF for decision-making are users with large cost-loss ratios, that is users
with a potential loss close to the cost for preventive action. This type of
user would take preventive action only when the event likelihood is high (the
uncertainty is low). EFI appears more appropriate than CPF in that case. For
situations where the user’s cost-loss is close to the event base rate, CPF has
more value than EFI, SOT, or ANC at day 1 and day 3. With larger score
uncertainty at longer time ranges, day 6 results show that all forecasts have
an estimated economic value in the same ballpark at this time range.

Figure 8:  Potential economic value of daily precipitation forecasts as a
function of a user’s cost-loss ratio. Results at (a) day 1, (b) day 3, and (c)
day 6 for Europe over Summer 2021. Note that the x-axis is distorted (using a
log-scale).

###  5.4 CPF reliability

So far, CPF has been interpreted and assessed as an index, that is with no
statistical meaning attached to its values. However, CPF is an ensemble
summary statistic which is also interpretable as a probabilistic forecast. So
we propose to assess the CPF reliability because reliability is, along
discrimination, a key attribute of a probabilistic forecast. An example of
what reliability means for CPF is the following: for all cases where CPF takes
value 0.95, it is expected that the observations exceed the 95% quantile of
the climatology in 5% of the time. This attribute can be checked with the help
of a so-called reliability diagram.

The reliability diagrams for CPF of daily precipitation are shown in Fig. 9.
In these plots, we show only cases where CPF is greater than 0.725, for
categories centred around 0.75, 0.8, 0.85, 0.9, 0.95, and 0.99. The bottom
panels show the reliability curves that ideally should be close to the
diagonal (dashed line). The upper panel shows the percentage of cases when CPF
falls within each of these categories.

In terms of consistency with the observations, precipitation CPFs exhibit
better reliability at longer lead times than at shorter ones. The ensemble
forecast is notably under-spread over the first days of the forecast (day 1
and 3). This characteristic is reflected in the CFP reliability plot in the
lower panel of Figs 9(a,b). In the upper panel, we see that high values of CPF
are more frequent at longer lead times, as already discussed in Section 4.2.

Reliability is an important forecast attribute for users who take forecasts at
face value. As the CPF has an interpretation in probabilistic terms, forecast
reliability ensures that the attached meaning to a forecast value and its
actual meaning are aligned. Reliability is a necessary condition for optimal
decision making (Wilks, 2018).

Figure 9:  Reliability diagram for CPF of daily precipitation. Results are
shown for forecasts at lead times (a) day 1, (b) day 3, and (c) day 7. In the
lower panel, the observation relative frequency is aggregated for different
CPF categories and perfect reliability is indicated with a dashed line. The
upper panel shows the distribution of CPF in each category.

##  6 Actionable forecasts for seamless predictions of extreme events

An actionable forecast is defined as a single value forecast444sometime also
referred to as a point forecast, by opposition to an ensemble forecast.
discretised with a set of decision thresholds such that a user can decide to
take action when a threshold is exceeded by the forecast. Seamless forecasting
across lead times would rely on actionable forecasts that could be used across
time ranges, without the need to change the forecast definition or the set of
decision thresholds with the forecast lead time.

Potential discrimination was examined in the previous section. Now, we assess
the forecast information content communicated to the end user for decision-
making. Forecast visualisation plays a key role in translating potential into
actual discrimination ability. The success of this translation is measured
using a verification setting consistent with the forecast visualisation: the
same set of decision thresholds is used for both applications. The rationale
is that a piece of information that exists but is not communicated is not
useful and has no value. This is typically the case when small probability
values are attached to high-impact weather events, especially in the medium-
range.

###  6.1 Performance of actionable forecasts

The forecasts are now discretised using a small set of decision thresholds
that can be used for plotting, communication, and eventually decision-making
purposes. For consistency, performance assessment relies on the same set of
thresholds to build contingency tables. The CPF decision thresholds are 0.85,
0.95, 0.98, 0.99, and 0.999, and the EFI ones are 0.3, 0.5, 0.6, 0.7, 0.8, and
0.9, and can be visualised in Figs 2 and 3. In these illustrations, the first
decision threshold is represented by a contour line while the colour coding is
used to show the other decision thresholds on the map. Note that the EFI
visualisation in this study is identical to the one used on the ECMWF official
web charts. For SOT, the thresholds used for plotting SOT contours on the same
charts are 0, 1, 2, 5, and 8. ANF is not included in this analysis because
this forecast type does not belong to the current ECMWF portfolio for medium-
range weather forecasts.

ROC curves built using these sets of decision thresholds are plotted in Fig.
10. Results are shown for CPF and EFI at 3 different lead times, day 1, day 3,
and day 6. For each curve, one point is highlighted. This point corresponds to
the combination ($H$,$F$) for a given decision threshold: the decision
thresholds selected for illustration are 0.95 for CPF and 0.7 for EFI. In
these plots, the ROC curves for SOT are also displayed for reference.

Figure 10:  Discrimination ability of the actionable forecasts as plotted on
ECMWF charts. Same as Fig. 6 but when considering a fixed set of decision
thresholds for each type of forecast.

Two key messages emerge from the inspection of Fig. 10. First, the empirical
ROC curves are “incomplete” for EFI (and SOT) while they are almost complete
for CPF. Second, the general performance associated with the selected decision
thresholds (indicated with $\mathbin{\ooalign{$\circledcirc$\cr$\bullet$}}$)
varies with the forecast lead time for EFI but not for CPF. These 2 aspects
are closely related and contribute to CPF being an appropriate actionable
forecast for high-impact weather, in particular at longer lead times.

The EFI curves are not complete because the set of decision thresholds does
not include values smaller than 0.5, but at day 6 for example, most of the EFI
values are below 0.5 as illustrated in Fig. 5(b). By changing the EFI
discretisation and adding smaller decision thresholds, more points would
contribute to the ROC curve leading to a higher AUC. A finer discretisation of
low EFI values would be particularly beneficial for forecasts at longer lead
times while the current discretisation seems more appropriate for forecasts at
shorter lead times.

So, for EFI, an ideal set of decision thresholds would vary with forecast lead
time. To improve the EFI actual discrimination power, one would recommend a
finer discretisation of low EFI values at longer lead times, when the
uncertainty is large and EFI generally small. By contrast, with CPF, a
consistent set of decision thresholds can be used seamlessly throughout all
lead times.

###  6.2 Seamless prediction of extreme events

Seamless forecasting is possible with CPF because the set of decision
thresholds maintains their performance characteristics and meaning with lead
time.

The highlighted points on the CPF curves (points associated with a given CPF
decision threshold) do not dramatically change position on the curves with the
forecast lead time. A direct link exists between the position on a ROC curve
and the optimal decision threshold for a given application. Indeed, a given
combination of false alarm rate $F$ and hit rate $H$ is more appropriate for
some users than for others. In other words, for an application with a given
cost-loss ratio, a forecaster can focus on a single decision threshold when
using CPF. By contrast, the optimal decision threshold would vary with
forecast lead time when using EFI (or SOT).

The optimal decision threshold is known apriori for CPF when CPF is well
calibrated555at longer lead times, CPF is not far from calibration as
illustrated in Fig. 9(c). For example, a user with a cost-loss ratio of 0.05
should use the CPF with a decision threshold of 0.95. More generally, a more
complete ROC curve means more users would benefit from the forecast,
especially users with a small cost-loss ratio. This is reflected in the
economic value plots in Fig. 11. Here, the economic value of CPF, EFI, and SOT
is estimated with the same contingency tables as for the ROC curves in Fig.
10.

Figure 11:  Economic value of daily precipitation actionable forecast. Same as
Fig. 8 but when considering the same set of decision-threolds as in Fig. 10,
that is the thresolds used for the forecast visualisation in Figs 2 and 3.

Differences between potential and actual discrimination are visible when
comparing ROC curves in Fig. 6 and Fig. 10, respectively. We can also directly
compare these differences by computing the area under the ROC curve (AUC).
Relative differences in discrimination ability are assessed by computing the
corresponding skill score measuring here the relative performance of CPF with
respect to EFI:

$\frac{\text{AUC}_{CPF}-\text{AUC}_{EFI}}{1-\text{AUC}_{EFI}}.$ (7)

AUC results as a function of the forecast lead time are presented in Fig. 12.

In Fig. 12(a), a large difference between potential discrimination (solid
lines) and actual discrimination (dashed line) is an indication that the
current way of communicating a forecast could be improved. A clear gap exists
for EFI and this gap increases with lead time. As already noted by Raynaud _et
al._ (2018), "[…] a more accurate calibration of optimal thresholds would be
necessary in order to improve the EFI utilisation." The difference between
potential and actual discrimination is much smaller for CPF and is constant
with lead time.

In Fig. 12(b), CPF appears significantly better at actually discriminating
than EFI at all lead times. The superiority of CPF in terms of actual
discrimination increases with the forecast lead time reaching a skill score of
almost 40% at day 6. With CPF, a forecaster is able to actually discriminate
the occurence of extreme precipitation events 7 days ahead with the same
ability as 4 days ahead when using EFI.

Figure 12:  (a) Area under the ROC curve as a function of the forecast lead
time and (b) corresponding skill score of CPF using EFI as a reference
forecast. The solid lines indicate the potential discrimination ability while
the dashed lines indicate the actual discrimination ability when considering a
limited set of decision thresholds. In (b), the shade represents the 5% and
95% confidence intervals as estimated with 5-day block boot-strapping.
Forecast discretisation has a negative impact on EFI discrimination while CPF
discrimination is preserved at all lead times.

##  7 Conclusion

This study presents the singularity and performance of a new index for the
early detection of high-impact weather events. Case studies and results focus
on daily precipitation, but the concept can be applied to any other weather
variable. We show that :

  1. 1.

the new index has a statistical meaning related to the concept of return-
period and is directly linked to a proper score,

  2. 2.

the new index has similar potential discrimination ability but a different
evolution with lead time compared to all the other key indicators of extreme
events investigated here,

  3. 3.

the new index can display a strong signal at longer lead times pointing to a
potential high-impact weather event leading to an enhanced actual
discrimination power on visualised maps,

The new proposed index is derived from the crossing-point forecast (CPF) which
is a consistent forecast with the diagonal score used for assessing ensemble
forecasts with a focus on extreme events. The CPF can be interpreted as a
"worst-case scenario in a probabilistic sense" for which the concept of
reliability holds and can be further assessed. CPF encompasses a duality
risk/intensity in its interpretation: its value refers at the same time to a
risk level and to an event intensity that can be expressed in terms of a
return period. Conveniently, the index takes a value between 0 and 1, so it
can be used and visualised as an index. But more importantly, its
(statistical) meaning and characteristics do not vary with lead time.

A comparison with the well-established extreme forecast index EFI reveals that
the new index has a similar potential discrimination ability of extreme
precipitation events and a lower correlation with EFI than the shift of tail
SOT or the ensemble mean standardised anomaly ANC. The new index is the only
forecast among those analysed here that exhibits a stronger signal for extreme
events at longer rather than shorter lead times. This new type of forecast is
particularly well-suited for users sensitive to missed events but could accept
a higher level of false alarms. Typically, the users who should most benefit
from the new index have a small cost-loss ratio, in other words, their cost of
taking a preventive action is rather lower compared to the potential loss in
case of a missed event. The new index is also better at distinguishing the
most extreme events from lower-intensity precipitation events.

Close to the event, CPF and EFI converge to similar “solutions”, but at longer
lead times, we observe a complementarity between CPF and EFI: EFI tends to be
muted which reduces false alarms while CPF tends to have a large-scale signal
which reduces missed events. In most cases, EFI does not exceed 0.5 at day 6
because it is highly correlated with the ensemble mean anomaly and dampened as
the ensemble spread increases. While it might be difficult to trigger
forecasters’ attention on an event with a small probability of occurrence in
the forecast or an EFI close to 0, with CPF, low-probability events are put
forward when the forecast risk exceeds the climatological risk. As a result, a
signal for a rare event can be present even when the forecast uncertainty is
large, but with the caveat that this signal requires confirmation with the
following forecast runs to limit false alarms. As illustrated with two case
studies, precipitation events with low probability and potentially high impact
can be seamlessly captured and communicated at the medium-range time scale
with the help of the new index.

## References

  * Ben Bouallègue (2021) Ben Bouallègue Z. 2021. On the verification of the crossing-point forecast. _Tellus A: Dynamic Meteorology and Oceanography_ 73(1): 1–10, doi:10.1080/16000870.2021.1913007.
  * Ben Bouallègue _et al._ (2023) Ben Bouallègue Z, Clare MCA, Magnusson L, Gascon E, Maier-Gerber M, Janousek M, Rodwell M, Pinault F, Dramsch JS, Lang STK, Raoult B, Rabier F, Chevallier M, Sandu I, Dueben P, Chantry M, Pappenberger F. 2023. The rise of data-driven weather forecasting. _arXiv_ doi:10.48550/arXiv.2307.10128.
  * Ben Bouallègue _et al._ (2018) Ben Bouallègue Z, Haiden T, Richardson DS. 2018. The diagonal score: definition, properties, and interpretations. _Quart. J. Roy. Meteor. Soc._ 144(714): 1463–1473, doi:10.1002/qj.3293.
  * Ben Bouallègue _et al._ (2019) Ben Bouallègue Z, Magnusson L, Haiden T, Richardson DS. 2019. Monitoring trends in ensemble forecast performance focusing on surface variables and high-impact events. _Quart. J. Roy. Meteor. Soc._ 145(721): 1741–1755, doi:10.1002/qj.3523.
  * Ben Bouallègue and Richardson (2022) Ben Bouallègue Z, Richardson DS. 2022. On the roc area of ensemble forecasts for rare events. _Weather and Forecasting_ 37(5): 787 – 796, doi:10.1175/WAF-D-21-0195.1.
  * Boisserie _et al._ (2016) Boisserie M, Descamps L, Arbogast P. 2016. Calibrated forecasts of extreme windstorms using the extreme forecast index (efi) and shift of tails (sot). _Weather and Forecasting_ 31(5), doi:10.1175/WAF-D-15-0027.1.
  * Dutra _et al._ (2013) Dutra E, Diamantakis M, Tsonevsky I, Zsoter E, Wetterhall F, Stockdale T, Richardson D, Pappenberger F. 2013. The extreme forecast index at the seasonal scale. _Atmospheric Science Letters_ 14(4): 256–262, doi:10.1002/asl2.448.
  * Fundel _et al._ (2010) Fundel F, Walser A, Liniger MA, Frei C, Appenzeller C. 2010. Calibrated precipitation forecasts for a limited-area ensemble forecast system using reforecasts. _Monthly Weather Review_ 138(1), doi:10.1175/2009MWR2977.1.
  * Fundel _et al._ (2019) Fundel VJ, Fleischhut N, Herzog SM, Göber M, Hagedorn R. 2019. Promoting the use of probabilistic weather forecasts through a dialogue between scientists, developers and end-users. _Quarterly Journal of the Royal Meteorological Society_ 145(S1): 210–231, doi:10.1002/qj.3482.
  * Guan and Zhu (2017) Guan H, Zhu Y. 2017. Development of verification methodology for extreme weather forecasts. _Weather and Forecasting_ 32(2): 479 – 491, doi:10.1175/WAF-D-16-0123.1.
  * Haiden _et al._ (2021) Haiden T, Janousek M, Vitart F, Ben Bouaégue Z, Ferranti L, Prates F. 2021. Evaluation of ECMWF forecasts, includuing 2021 upgrade. _ECMWF Technical Memorandum_ 884.
  * Lalaurette (2003) Lalaurette F. 2003. Early detection of abnormal weather conditions using a probabilistic extreme forecast index. _Quart. J. Roy. Meteor. Soc._ 129: 3037–3057, doi:doi: 10.1256/qj.02.152.
  * Lavers _et al._ (2016) Lavers DA, Pappenberger F, Richardson DS, Zsoter E. 2016. ECMWF Extreme Forecast Index for water vapor transport: A forecast tool for atmospheric rivers and extreme precipitation. _Geophysical Research Letters_ 43(22): 11,852–11,858, doi:10.1002/2016GL071320.
  * Lavers _et al._ (2018) Lavers DA, Richardson DS, Ramos AM, Zsoter E, Pappenberger F, Trigo RM. 2018. Earlier awareness of extreme winter precipitation across the western Iberian Peninsula. _Meteorological Applications_ 25(4): 622–628, doi:10.1002/met.1727.
  * Lavers _et al._ (2017) Lavers DA, Zsoter E, Richardson DS, Pappenberger F. 2017. An assessment of the ECMWF Extreme Forecast Index for water vapor transport during boreal winter. _Weather and Forecasting_ 32(4): 1667 – 1674, doi:10.1175/WAF-D-17-0073.1.
  * Leutbecher (2019) Leutbecher M. 2019. Ensemble size: How suboptimal is less than infinity? _Q. J. R. Meteorolog. Soc._ 145(Suppl. 1): 107–128, doi:10.1002/qj.3387.
  * Murphy and Winkler (1987) Murphy AH, Winkler RL. 1987. A general framework for forecast verification. _Mon. Wea. Rev._ 115: 1330–1338.
  * Neal _et al._ (2014) Neal RA, Boyle P, Grahame N, Mylne K, Sharpe M. 2014. Ensemble based first guess support towards a risk-based severe weather warning service. _Meteorological Applications_ 21(3): 563–577, doi:10.1002/met.1377.
  * Petroliagis and Pinson (2014) Petroliagis TI, Pinson P. 2014. Early warnings of extreme winds using the ECMWF Extreme Forecast Index. _Meteorological Applications_ 21(2): 171–185, doi:10.1002/met.1339.
  * Prates and Buizza (2011) Prates F, Buizza R. 2011. PRET, the Probability of RETurn: a new probabilistic product based on generalized extreme-value theory. _Quarterly Journal of the Royal Meteorological Society_ 137(655): 521–537, doi:10.1002/qj.759.
  * Raynaud _et al._ (2018) Raynaud L, Touzé B, Arbogast P. 2018. Detection of severe weather events in a high-resolution ensemble prediction system using the extreme forecast index (EFI) and shift of tails (SOT). _Weather and Forecasting_ 33(4), doi:10.1175/WAF-D-17-0183.1.
  * Richardson (2000) Richardson DS. 2000. Skill and relative economic value of the ECMWF ensemble prediction system. _Q. J. R. Meteorol. Soc._ 126: 649–667, doi:10.1002/qj.49712656313.
  * Richardson (2011) Richardson DS. 2011. Economic value and skill. In: _Forecast Verification: A Practitioner’s Guide in Atmospheric Science_ , Jolliffe IT, Stephenson DB (eds), John Wiley and Sons, pp. 167–184.
  * Tempest _et al._ (2023) Tempest KI, Craig GC, Brehmer JR. 2023. Convergence of forecast distributions in a 100,000-member idealised convective-scale ensemble. _Quarterly Journal of the Royal Meteorological Society_ 149(752): 677–702, doi:10.1002/qj.4410.
  * Tsonevsky _et al._ (2018) Tsonevsky I, Doswell CA, Brooks HE. 2018. Early warnings of severe convection using the ECMWF Extreme Forecast Index. _Weather and Forecasting_ 33(3): 857 – 871, doi:10.1175/WAF-D-18-0030.1.
  * Wilks (2018) Wilks D. 2018. Enforcing calibration in ensemble postprocessing. _Quart. J. Roy. Meteor. Soc._ 144(710): 76–84, doi:10.1002/qj.3185.
  * Zsoter (2006) Zsoter E. 2006. Recent developments in extreme weather forecasting. _ECMWF Newsletter_ 107: 8–17.