# Evaluating the Retrieval Component in LLM-Based Question Answering Systems

Ashkan Alinejad  Thomson Reuters AI LabsTorontoCanada
<EMAIL_ADDRESS>,  Krtin Kumar  Thomson Reuters AI
LabsTorontoCanada<EMAIL_ADDRESS>and  Ali Vahdat  Thomson
Reuters AI LabsTorontoCanada<EMAIL_ADDRESS>
(2024; 20 February 2024; 12 March 2024; 5 June 2024)

###### Abstract.

Question answering systems (QA) utilizing Large Language Models (LLMs) heavily
depend on the retrieval component to provide them with domain-specific
information and reduce the risk of generating inaccurate responses or
hallucinations. Although the evaluation of retrievers dates back to the early
research in Information Retrieval, assessing their performance within LLM-
based chatbots remains a challenge.

This study proposes a straightforward baseline for evaluating retrievers in
Retrieval-Augmented Generation (RAG)-based chatbots. Our findings demonstrate
that this evaluation framework provides a better image of how the retriever
performs and is more aligned with the overall performance of the QA system.
Although conventional metrics such as precision, recall, and F1 score may not
fully capture LLMs’ capabilities – as they can yield accurate responses
despite imperfect retrievers – our method considers LLMs’ strengths to ignore
irrelevant contexts, as well as potential errors and hallucinations in their
responses.

Information Retrieval, Evaluation, RAG, LLM

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
email; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Recent advancements in Large Language Models (LLMs) (Brown et al., 2020) have
shown promising results across a wide range of Natural Language Processing
(NLP) tasks (Park, 2023), including Information Retrieval (IR), text
generation, and summarization. Particularly noteworthy is the significant
improvement observed in Question Answering (QA) tasks, where the goal is to
generate accurate responses given relevant document chunks containing the
answer.

To enhance the accuracy of QA systems (Izacard and Grave, 2021; Lewis et al.,
2020) and mitigate the risk of hallucinations (Agrawal et al., 2024; Shuster
et al., 2021) from LLMs, Retrieval-Augmented Generation (RAG) models have been
proved as a promising solution (Lewis et al., 2020). These models integrate a
retriever component, which retrieves relevant document chunks to provide the
LLM with the necessary context for generating responses. Evaluation of the
retriever component typically relies on two types of metrics: (a) Rank-
agnostic metrics, such as Precision and Recall, which compare retrieved chunks
with gold-labeled chunks, and (b) Rank-aware metrics, such as Normalized
Discounted Cumulative Gain (NDCG) (Järvelin and Kekäläinen, 2002) or Mean
Reciprocal Rank (MRR) (Craswell, 2009), which consider the order of retrieved
documents.

Recent studies have proposed using LLMs as judges to evaluate various NLP
tasks (Li et al., 2023; Zheng et al., 2023; Huang et al., 2024) by providing
them with an evaluation scheme in the form of a prompt. This approach enables
LLMs to follow instructions and reliably evaluate responses based on user-
defined metrics (Huang et al., 2024). However, while many studies have focused
on using LLMs to evaluate QA responses, there is a notable gap in research
concerning the evaluation of the retriever component (Salemi and Zamani,
2024).

In this paper, we introduce LLM-retEval, a framework designed to evaluate the
retriever component in RAG-based question answering models. Our primary
objective is to develop an evaluation metric for the retriever that considers
the strengths and weaknesses of LLMs and provides a clearer understanding of
its performance within an LLM-based QA system.

To achieve this goal, we first examine how conventional metrics for measuring
retriever performance can fall short. Our experiments on the NQ-open corpus
reveal that solely focusing on annotated data can significantly impact our
ability to accurately assess retriever behavior, particularly when annotators
fail to annotate all documents containing the answer. Additionally, evaluating
the retriever as an isolated component overlooks the downstream effects of its
results. This is crucial, as closely related but irrelevant retrieved chunks
can easily distract LLMs from generating accurate responses.

Instead of solely examining the retriever’s output, we propose measuring the
performance of the downstream QA task relative to a model with an ideal
retriever. By separately passing retrieved documents and gold documents to the
answer generation LLM and comparing the resulting responses, we gain valuable
insights into the retriever’s effectiveness. Our findings suggest that by
addressing the limitations of conventional metrics, they can become highly
correlated with LLM-retEval, demonstrating the robustness of our method in
capturing retriever performance.

##  2\. Evaluation Framework

In this paper, we focus on evaluating the retriever in the QA task and we rely
only on the correctness of the responses. The LLM-based QA system can be split
into two distinct components. The retriever searches the corpus and extracts
the documents that potentially contain the answer, and the generator receives
those documents and generates an answer. We will start this section by
formally describing the retriever and generator. Then we explain our
evaluation method in Section 2.2.

###  2.1. LLM-based Question Answering

A Question Answering (QA) system tries to provide an accurate response to a
natural language query from the user, based on relevant contexts from a
provided pool of knowledge. Formally, Given a user query $\mathnormal{q}$, we
try to extract the relevant information $\mathnormal{R}$ from the corpus of
documents $\mathcal{D}=\\{d_{1},\dots,d_{n}\\}$ to generate answer
$\mathnormal{A}$. In an LLM-based QA system, this process is typically divided
into two distinct components:

### Retriever

Extracts a subset of documents $\mathnormal{R^{\prime}}$ from $\mathnormal{D}$
which contains the answer to the query. Having an annotated set of relevant
documents $\mathnormal{R}$, we traditionally try to maintain
$\mathnormal{R^{\prime}}$ as close as possible to $\mathnormal{R}$. Dense
Passage Retrieval (DPR) (Karpukhin et al., 2020) is a common method for
extracting $R^{\prime}$ which encodes the query and documents into the same
vector space. The distance between the embeddings of the query and each
document will be used to select $R^{\prime}$.

### Generator

Attempts to generate an accurate response to the user query based on the
extracted relevant documents ($\mathnormal{R^{\prime}}$) by utilizing a
language model $\mathcal{M}$:

$A=\mathcal{M}(\mathnormal{q},\\{\mathnormal{r}\\}),\ \ \ \ \
\forall{r}\in\mathnormal{R^{\prime}}$

The generator can receive the documents in $\mathnormal{R^{\prime}}$ all at
once, or it can receive them one by one or in batches depending on the
document sizes, the complexity of the task, or the capabilities of the LLM.

###  2.2. Evaluating retrieval in QA context

In order to measure the performance of the retriever in the QA system, we pass
its output ($R^{\prime}$) to the generator LLM $\mathcal{M}$ to generate
answers. We also pass the gold relevant documents ($R$) to the same LLM to see
how the generator performs with an ideal retriever. By fixing the LLM
parameters and comparing the answers from the two configurations, we can get a
clear picture of how the retriever performs in an end-to-end question
answering system. There are multiple methods for automatic comparison of the
QA answers:

  * •

Exact Match (EM) compares strings directly to determine if they are exactly
equal. However, it may be overly strict due to the potential variability in
LLM outputs.

  * •

Token-based metrics such as ROUGE-1, BLEU, and METEOR quantify the deviation
between texts on a token/word level. Setting a threshold on these scores
enables acceptance of answers that are highly similar but not exact matches.

  * •

Embedding-based metrics vectorize answers and compute the cosine similarity
between the vectors. BERTScore (Zhang et al., 2020) is an example of such
metric that is based on pretrained BERT embeddings which can capture the
contextual information in answers.

  * •

LLM-based evaluation have been recently utilized to evaluate the QA systems
(Huang et al., 2024; Zheng et al., 2023). They have demonstrated a great
ability to capture the semantics of the answers while attending to their
nuance variances.

We focus primarily on LLM-based evaluation to measure answer variances in this
work. We can summarize our approach in three primary steps:

  1. (1)

Running the RAG-based QA system under evaluation. We use the retriever in our
RAG pipeline to extract the relevant data and pass them to the generator LLM.

  2. (2)

Passing the gold relevant passages to the generator LLM to generate semi-gold
responses.

  3. (3)

Comparing the responses generated by the RAG-based QA model with the semi-gold
responses in step 2 using an LLM-based evaluation method, which outputs ”Yes”
if the responses match and ”No” if they differ.

It is important to note that the decision to use a yes/no grading system is
based on the characteristics of the dataset. In the NQ-open dataset, questions
are typically broad and the answers are short (fewer than five tokens.)
However, when evaluating QA tasks in specialized domains such as legal or
medical, where nuances in the answers are crucial, a more granular grading
scale is recommended.

##  3\. Experimental setup

In this section, we will describe our experimental settings, beginning with an
explanation of the data used, followed by an outline of our evaluation
methodology.

###  3.1. Dataset

For our experiments, we used the NQ-open dataset (Kwiatkowski et al., 2019a;
Lee et al., 2019), which represents a subset of the Natural Questions (NQ)
corpus (Kwiatkowski et al., 2019b). Each sample in the NQ dataset includes a
single question, a tokenized representation of the question, a Wikipedia URL,
and the HTML representation of the corresponding Wikipedia page. The NQ-open
dataset enhances this information by providing one or multiple answers for
each question that are extracted directly from the associated Wikipedia
passages and are no longer than 5 tokens. Although our evaluation approach
doesn’t require the gold answers to the queries, having them allows us to
assess the model performance thoroughly and closely investigate how the
retriever is performing relative to the overall QA performance.

Given the focus of our paper on evaluation criteria and the utilization of a
basic embedding retrieval approach, we opt not to use the NQ-open training
set. Instead, we only rely on the 2,889 samples in the test set for evaluation
purposes. We use the English Wikipedia dump of 20 December 2018 for extracting
the relevant chunks in our RAG model and follow the steps in (Cuconasu et al.,
2024) to split the passages into smaller chunks and clean the dataset. This
gives us 21,035,236 documents to extract the relevant chunks.

###  3.2. Retrieval and generative models

In our experiments, we are utilizing a dense retrieval where we embed the
document chunks using the ”e5-large-v2”
model111https://huggingface.co/intfloat/e5-large-v2 (Wang et al., 2022). We
pick the top $k\in[1,5,10]$ documents based on the cosine similarity of the
embeddings of query and chunks.

For the generation component of our QA models, we utilize two state-of-the-art
language models: GPT-4 and ChatGPT-Turbo. These models have demonstrated
strong performance in various natural language processing tasks, including
question answering. Since for each question in NQ-open dataset, there might be
multiple correct answers, we generate the ground-truth responses 3 times with
a temperature of 0.5, to make sure that the variety of possible correct
answers is generated.

For comparing the QA answers with the ground-truth model, we adopt the
GPT4-Eval (Kamalloo et al., 2023) similar to the setup in (Adlakha et al.,
2023), which has been shown to be highly correlated with the human judgments.
we have made slight modifications to the prompt to better align with the
characteristics of our dataset. Details of these modifications are provided in
Appendix A.

##  4\. Results

Analyzing the performance of LLM-retEval starts by examining the failure cases
of conventional metrics and LLM-retEval. We investigate the statistics of
failure cases and assess the correlation between the two evaluation
approaches. Although our experiments include reporting Precision@k and F1@k,
our primary focus lies on comparing our method with Recall@k. This emphasis is
due to the fact that in the NQ-open dataset, only one paragraph is labeled for
each question. Consequently, the precision@k metric cannot exceed $1/k$
percent. In contrast, Recall@k indicates whether the correct answer appears
within the top K results.

###  4.1. Qualitative Analysis

By closely examining cases that result in discrepancies between the
conventional retriever metrics such as Precision or Recall and our LLM-retEval
model, we gain insight into the limitations of traditional metrics in LLM-
based QA models and how our proposed framework addresses these limitations. We
identify these shortcomings through our experiments on the NQ-open dataset.
It’s important to note that testing additional datasets could reveal more
failure cases.

### Failure cases of conventional metrics:

We categorize the sources of failure in traditional metrics into the following
categories:

  * •

Case A: Not annotating all the correct responses:
This occurs when an answer to a question can appear in multiple documents, but
only one of them is labeled. This limitation is common in many databases where
annotators are unable to search the entire corpus. As a result, traditional
metrics penalize the retriever for not retrieving the gold excerpt. However,
since the generator in LLM-based QA models can generate accurate responses
using retrieved chunks, our model does not consider this scenario a failure.

  * •

Case B: Discrepancy between the searching documents and labeled data:
For example, consider the case illustrated in Figure 1, where the gold
document is an older version of the same Wikipedia page being searched. In
such cases, traditional metrics penalize the retriever for not returning the
exact same chunk, even though its output is accurate and the generator can
answer the question correctly based on context.

  * •

Case C: The retriever returns close but irrelevant chunks alongside the gold
documents, distracting the generator: This scenario is more common in LLM-
based QA models. In such cases, the retriever receives a high score based on
traditional metrics because it returns the gold documents. However, the
presence of irrelevant chunks alongside the gold documents can lead the LLM to
generate incorrect responses.

In the first example in Figure 1, the LLM correctly generates the answer
Tulsa, Oklahoma using a document other than the gold-labeled document. Here,
the retriever effectively retrieves a document chunk containing the answer,
although not the one labeled as correct by annotators.

The retrieved and gold documents in the second example in Figure 1 come from
the same Wikipedia page but their versions are different. Despite both
containing the answer, the retriever is penalized by conventional metrics due
to the discrepancy in indexing within our corpus. However, the generator
successfully produces the correct response using the retrieved chunk.

In the final example illustrated in Figure 1, the retriever includes the
correct document in its top 5 results. However, the addition of a misleading
document about The Horn of Africa distracts the generator, leading to an
incorrect response (Nigeria, Horn of Africa). While the retriever achieves a
high Recall@5 by including the correct chunk, LLM-retEval penalizes it for
including the misleading document. Notably, the low Precision@5 of 20 is due
to the presence of only one correct chunk for each question, so the
Precision@5 cannot go above 20.

### Failure cases of LLM-retEval:

There are three main sources of errors that come from inaccuracies in LLM
responses:

  * •

Case D: LLM cannot generate ground-truth answer Generating an accurate
response from the gold context is a crucial step in evaluating the retriever,
and the LLMs can potentially fail to find the correct answer in the text. This
usually happens when the information lies within a mal-processed table or
text, or the answer to the question is not explicitly mentioned in the text.

  * •

Case E: The ground-truth LLM doesn’t generate all the correct answers: This
issue happens when there are multiple correct answers and the LLM returns a
subset of them. For example, the answer to the question ”who played scotty
baldwins father on general hospital” can be both ’Peter Hansen’, and ’Ross
Elliott’. If the ground-truth LLM returns only one of them, then the other
answer would be considered incorrect. This issue can be partly addressed by
generating ground-truth responses multiple times with a temperature above 0.

  * •

Case F: Inaccuracies in comparing two answers In this case the LLM is unable
to precisely compare the generated response $R^{\prime}$ with the ground-truth
answer $R$. This issue has been reported in previous studies (Kamalloo et al.,
2023; Adlakha et al., 2023). One possible solution to mitigate this problem is
to integrate the LLM’s judgments with alternative comparison techniques
discussed in Section 2.2.

Error Type: Not annotating all the correct responses
---
Question: where do the greasers live in the outsiders
Gold document ($R$): The story in the book takes place in Tulsa, Oklahoma, in
1965, but this is never explicitly stated in the book.
Retrieved document ($R^{\prime}$): In Tulsa, Oklahoma, greasers are a gang of
tough, low-income working - class teens. They include Ponyboy Curtis and his
two older brothers, …
Scores: Recall@1: 0, Precision@1: 0, F1@1: 0, GPT4-retEval: Yes
Error Type: Discrepancy between the searching documents and labeled data
Question: who got the first nobel prize in physics
Gold document ($R$): The first Nobel Prize in Physics was awarded in 1901 to
Wilhelm Conrad Röntgen … As of 2017 , the prize has been awarded to 206
individuals. There have been six years in which the Nobel Prize in Physics was
not awarded ( 1916 , 1931 , 1934 , 1940 – 1942 ) .
Retrieved document ($R^{\prime}$): The first Nobel Prize in Physics was
awarded in 1901 to Wilhelm Conrad Röntgen … As of 2016 , the prize has been
awarded to 203 individuals . There have been six years in which the Nobel
Prize in Physics was not awarded ( 1916 , 1931 , 1934 , 1940 – 1942 ) .
Scores: Recall@1: 0, Precision@1: 0, F1@1: 0, GPT4-retEval: Yes
Error Type: The retriever returns close but irrelevant chunks alongside the
gold documents, distracting the generator
Question: in which regions are most of africa petroleum and natural gas found
Gold document ($R$): Nigeria is the largest oil and gas producer in Africa.
Crude oil from the delta basin comes in …
Gold answer: Nigeria, delta basin
3rd retrieved document ($R^{\prime}$): The Horn of Africa is a peninsula in
Northeast Africa . It juts hundreds of kilometers into …
LLM answer based on top 5 docs: Nigeria, Horn of Africa
Scores: Recall@5: 100, Precision@5: 20, F1@5: 33.3, GPT4-retEval: No
Figure 1. Qualitative examples cases where conventional metrics fail, along with LLM-retEval scores. The text in green color is the correct answer to the question. | @1 results | @5 results | @10 results
---|---|---|---
Recall failures | 500 | 248 | 161
LLM-retEval Failures | 101 | 147 | 154
Recall@k | 0.498 | 0.792 | 0.865
Precision@k | 0.498 | 0.158 | 0.086
F1@k | 0.498 | 0.264 | 0.157
LLM-retEval | 0.648 | 0.767 | 0.786
GPT-Eval | 0.666 | 0.778 | 0.796
Table 1. The statistics of the failure cases along with the overall
performance of the retriever and the QA model. Each ”@k results” column
presents the results of the model when we use the top k outputs of the
retriever for generating answers.

###  4.2. Quantitative Analysis

Table 1 provides an overview of the failure cases, retriever performance, and
overall QA results. Each column in the table represents how well the model
performs when considering the top $k\in[1,5,10]$ results retrieved by the
retriever. When we extract more chunks from the RAG pipeline, the likelihood
of finding the correct answer in the retriever’s output increases, leading to
higher recall and a decrease in cases A and B. Although a higher value of $k$
may cause the LLM to become more easily distracted (case C), this increase is
smaller than the decrease observed in cases A and B. As a result, overall
failures in Recall decrease. However, increasing $k$ also raises the
likelihood of the RAG-based QA model generating a response that is close but
not entirely accurate. Thus, as we increase $k$ from 1 to 5, we observe Case F
more frequently, leading to an increase in LLM-retEval failures. Nonetheless,
overall, LLM-retEval maintains a consistently low failure rate across our
experiments.

The second part of Table 1 summarizes the retriever’s performance based on
conventional metrics and LLM-retEval. As $k$ increases, both precision and F1
scores drop significantly due to only one labeled chunk for each query.
Therefore, our focus will be on recall. The last row of the table assesses how
well the QA model performs based on the gold answers using the GPT-Eval model.
Comparing the Recall@k and GPT-Eval results reveals that for lower values of
$k$, the retriever’s recall penalizes it for not leveraging the LLM’s ability
to generate correct responses from non-labeled chunks, while also giving undue
credit to the retriever for not considering distractions from irrelevant
retrieved chunks. Meanwhile, LLM-retEval consistently aligns closely with the
overall QA performance across different numbers of retrieved chunks.

| @1 results | @5 results | @10 results
---|---|---|---
Dataset | ALL | Refined | ALL | Refined | ALL | Refined
GPT4 | 0.58 | 0.87 | 0.48 | 0.80 | 0.44 | 0.73
chatGPT-turbo | 0.55 | 0.80 | 0.48 | 0.78 | 0.43 | 0.71
Table 2. Spearman’s correlation of GPT models and Recall@$k$ when we retrieve
$k\in[1,5,10]$ documents. The ”ALL” dataset means NQ-open test set and
”Refined” means the NQ-open test set after removing the Failure cases of
conventional metrics.

We conducted Spearman’s correlation (Spearman, 1904) analysis between the LLM-
retEval model and Recall@k, and the results are summarized in Table 2.
Correlation was measured on two versions of the test set: (1) ”ALL,” which
includes all available data in our test set, and (2) ”Refined,” where failure
cases of the recall are excluded. Due to the significant number of failure
cases, Recall is not strongly correlated with LLM-retEval results when tested
on the entire test set. However, upon removing these failure cases, the two
evaluation approaches become highly correlated. With the increase in $k$, the
correlation on the Refined dataset slightly decreases. This is mainly due to
the fact that the Recall’s failure cases decrease, leaving harder samples in
the ”Refined” set.

Table 2 also presents a comparison of the results between GPT4 and chatGPT-
turbo. Both models exhibit very similar performance and are highly correlated
with the ”Refined” test set. However, GPT4 consistently maintains a closer
correlation with Recall across our experiments, indicating that it experiences
fewer failure cases compared to chatGPT-turbo. This suggests that GPT4 may be
more adept at handling challenging scenarios and maintaining accuracy in
retrieving relevant information.

##  5\. Related work

Most evaluations of the RAG systems rely on metrics such as Precision or
Recall (Baeza-Yates and Ribeiro-Neto, 1999; Alaofi et al., 2024). These
metrics treat the retriever as a standalone search engine and assess its
performance independently of the downstream task. For systems outputting a
ranked list, rank-aware metrics like Normalized Discounted Cumulative Gain
(NDCG), Mean Reciprocal Rank (MRR), or Mean Average Precision (MAP) are
commonly employed. While these metrics utilize human-annotated data to
evaluate retriever performance, they overlook how the retriever’s output
influences the performance of subsequent components in the QA system.

Alternative approaches to measuring retriever behavior without relying on gold
relevance judgments have been explored (Soboroff et al., 2001; Jin et al.,
2001; Li and Shang, 2000). These methods automate precision evaluation using
techniques such as the vector space model and statistical analysis of
documents. However, similar to conventional metrics like Precision or Recall,
they fail to account for the downstream components of the QA pipeline,
focusing solely on corpus statistics.

A related line of research investigates the use of Large Language Models
(LLMs) to evaluate various NLP tasks (Li et al., 2023; Zheng et al., 2023;
Huang et al., 2024). These studies emphasize end-to-end evaluation of LLM-
based systems but do not provide clear insights into the impact of the
underlying retriever component on performance.

A recent study by Salemi et al. (2024) (Salemi and Zamani, 2024) explores the
use of LLMs to measure retrieval quality in RAG-based models. Similar to our
approach, they consider the downstream task when evaluating the retriever.
However, their focus lies on enhancing the accuracy of conventional metrics
such as Precision and Recall by leveraging LLMs to annotate relevance
judgments for each question.

##  6\. Conclusion

This paper investigates methods for evaluating retrievers in RAG-based
question answering models and introduces a new metric for end-to-end
performance assessment of the retriever. Our findings suggest that a direct
comparison between the retrieved document set and the gold-relevant documents
may not fully illustrate the performance of the retriever in LLM-based QA
models. We discovered that generating answers based on gold-relevant documents
using the same generator LLM can provide a reliable indicator of the
retriever’s performance. Our results demonstrate that this method is less
susceptible to errors in LLM-based QA systems and aligns more closely with
conventional metrics when failure cases are not considered. Furthermore, our
observations reveal that while ChatGPT-Turbo demonstrates comparable
performance to the GPT-4 model, the GPT-4 model consistently outperforms other
models.

## References

  * (1)
  * Adlakha et al. (2023) Vaibhav Adlakha, Parishad BehnamGhader, Xing Han Lu, Nicholas Meade, and Siva Reddy. 2023.  Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering.  _arXiv:2307.16877_ (2023).
  * Agrawal et al. (2024) Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. 2024.  Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Survey.  arXiv:2311.07914 [cs.CL]
  * Alaofi et al. (2024) Marwah Alaofi, Negar Arabzadeh, Charles L. A. Clarke, and Mark Sanderson. 2024.  Generative Information Retrieval Evaluation.  arXiv:2404.08137 [cs.IR]
  * Baeza-Yates and Ribeiro-Neto (1999) Ricardo A. Baeza-Yates and Berthier A. Ribeiro-Neto. 1999.  _Modern Information Retrieval_.  ACM Press / Addison-Wesley.  http://www.ischool.berkeley.edu/~hearst/irbook/glossary.html
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Craswell (2009) Nick Craswell. 2009\.  _Mean Reciprocal Rank_.  Springer US, Boston, MA, 1703–1703.  https://doi.org/10.1007/978-0-387-39940-9_488
  * Cuconasu et al. (2024) Florin Cuconasu, Giovanni Trappolini, Federico Siciliano, Simone Filice, Cesare Campagnano, Yoelle Maarek, Nicola Tonellotto, and Fabrizio Silvestri. 2024.  The Power of Noise: Redefining Retrieval for RAG Systems.  arXiv:2401.14887 [cs.IR]
  * Huang et al. (2024) Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024.  An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Models are Task-specific Classifiers.  arXiv:2403.02839 [cs.CL]
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 874–880.  https://doi.org/10.18653/v1/2021.eacl-main.74
  * Järvelin and Kekäläinen (2002) Kalervo Järvelin and Jaana Kekäläinen. 2002.  Cumulated gain-based evaluation of IR techniques.  _ACM Trans. Inf. Syst._ 20, 4 (oct 2002), 422–446.  https://doi.org/10.1145/582415.582418
  * Jin et al. (2001) Rong Jin, Christos Falusos, and Alex G. Hauptmann. 2001\.  Meta-scoring: automatically evaluating term weighting schemes in IR without precision-recall. In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ (New Orleans, Louisiana, USA) _(SIGIR ’01)_. Association for Computing Machinery, New York, NY, USA, 83–89.  https://doi.org/10.1145/383952.383964
  * Kamalloo et al. (2023) Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023.  Evaluating Open-Domain Question Answering in the Era of Large Language Models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 5591–5606.  https://doi.org/10.18653/v1/2023.acl-long.307
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020\.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 6769–6781.  https://doi.org/10.18653/v1/2020.emnlp-main.550
  * Kwiatkowski et al. (2019a) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019a.  Natural Questions: A Benchmark for Question Answering Research.  _Transactions of the Association for Computational Linguistics_ 7 (08 2019), 453–466.  https://doi.org/10.1162/tacl_a_00276 arXiv:https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00276/1923288/tacl_a_00276.pdf
  * Kwiatkowski et al. (2019b) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019b.  Natural Questions: a Benchmark for Question Answering Research.  _Transactions of the Association of Computational Linguistics_ (2019).
  * Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019\.  Latent Retrieval for Weakly Supervised Open Domain Question Answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 6086–6096.  https://doi.org/10.18653/v1/P19-1612
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020\.  Retrieval-augmented generation for knowledge-intensive NLP tasks. In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ (¡conf-loc¿, ¡city¿Vancouver¡/city¿, ¡state¿BC¡/state¿, ¡country¿Canada¡/country¿, ¡/conf-loc¿) _(NIPS ’20)_. Curran Associates Inc., Red Hook, NY, USA, Article 793, 16 pages.
  * Li and Shang (2000) Longzhuang Li and Yi Shang. 2000.  A new method for automatic performance comparison of search engines.  _World Wide Web_ 3 (12 2000), 241–247.  https://doi.org/10.1023/A:1018790907285
  * Li et al. (2023) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  AlpacaEval: An Automatic Evaluator of Instruction-following Models.  https://github.com/tatsu-lab/alpaca_eval.
  * Park (2023) Daniel Park. 2023\.  _Open-LLM-Leaderboard-Report_.  https://github.com/dsdanielpark/Open-LLM-Leaderboard-Report
  * Salemi and Zamani (2024) Alireza Salemi and Hamed Zamani. 2024.  Evaluating Retrieval Quality in Retrieval-Augmented Generation.  arXiv:2404.13781 [cs.CL]
  * Shuster et al. (2021) Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021.  Retrieval Augmentation Reduces Hallucination in Conversation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Punta Cana, Dominican Republic, 3784–3803.  https://doi.org/10.18653/v1/2021.findings-emnlp.320
  * Soboroff et al. (2001) Ian Soboroff, Charles Nicholas, and Patrick Cahan. 2001\.  Ranking retrieval systems without relevance judgments. In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ (New Orleans, Louisiana, USA) _(SIGIR ’01)_. Association for Computing Machinery, New York, NY, USA, 66–73.  https://doi.org/10.1145/383952.383961
  * Spearman (1904) C. Spearman. 1904\.  The Proof and Measurement of Association between Two Things.  _The American Journal of Psychology_ 15, 1 (1904), 72–101.  http://www.jstor.org/stable/1412159
  * Wang et al. (2022) Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022\.  Text Embeddings by Weakly-Supervised Contrastive Pre-training.  _arXiv preprint arXiv:2212.03533_ (2022).
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.  https://openreview.net/forum?id=SkeHuCVFDr
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.  arXiv:2306.05685 [cs.CL]

##  Appendix A Prompts

⬇ Please read the question provided below and then review the accompanying
document excerpts. Your task is to answer the question using the information
from the documents: Question: {question} Relevant Document chunks: {context}
After considering the information in the documents, please provide an answer
(maximum 5 tokens) to the question: {question}. Answer: Figure 2. The prompt
template used for answer generation. ⬇ You are CompareGPT, a machine to verify
the correctness of predictions. Answer with only "Yes" or "No". You are given
a question, one or more corresponding ground-truth answers, and a prediction
from a model. Compare the "Ground-truth␣answers" and the "Prediction" to
determine whether the prediction correctly answers the question based on any
of the provided ground-truth answers. All information in at least one of the
ground-truth answers must be present in the prediction, including numbers and
dates. You must answer "No" if the prediction does not completely match at
least one set of specific details in the ground-truth answers. There should be
no contradicting statements in the prediction. The prediction may contain
extra information that does not contradict the ground-truth answers. Question:
{query} Ground-truth answers: {answer} Prediction: {result} Answer "Yes" if
the prediction correctly answers the question based on any of the Ground-truth
answers, otherwise answer "No".. Figure 3. The prompt template based on GPT-
Eval for comparing LLM responses.


# Uncovering the Spatial and Temporal Variability of Wind Resources in Europe:
A Web-Based Data-Mining Tool

Alban Puech
Institut Polytechnique de Paris
Palaiseau, France
<EMAIL_ADDRESS>
&Jesse Read
Laboratoire d’informatique de l’École Polytechnique
Palaiseau, France
<EMAIL_ADDRESS>

###### Abstract

We introduce REmap-eu.app, a web-based data-mining visualization tool of the
spatial and temporal variability of wind resources. It uses the latest open-
access dataset of the daily wind capacity factor in 28 European countries
between 1979 and 2019 and proposes several user-configurable visualizations of
the temporal and spatial variations of the wind power capacity factor. The
platform allows for a deep analysis of the distribution, the cross-country
correlation, and the drivers of low wind power events. It offers an easy-to-
use interface that makes it suitable for the needs of researchers and
stakeholders. The tool is expected to be useful in identifying areas of high
wind potential and possible challenges that may impact the large-scale
deployment of wind turbines in Europe. Particular importance is given to the
visualization of low wind power events and to the potential of cross-border
cooperations in mitigating the variability of wind in the context of
increasing reliance on weather-sensitive renewable energy sources.

##  1 Introduction

As the world is facing an unprecedented energy crisis, more and more
investments are going into intermittent renewable energy sources (IRES).
IRENA’s (2019) analysis forecasts wind power to represent 40 percent of the
EU-28 electricity needs in 2050. As energy production depends more and more on
weather conditions, there is a call for tools that could help in better
understanding and dealing with the risks that this can present. Wind
variability poses a significant challenge to energy production, as it can lead
to fluctuations in power output, making it difficult to manage energy supply
Staffell & Pfenninger (2018). Therefore, understanding the historical
variability of wind power is essential to make well-informed decisions
regarding the future deployment and management of wind turbines. One solution
to the problem of wind variability is cross-border cooperation, which allows
countries to share their wind resources and balance out variations in power
output. In January 2023, the European Commission called for cross-border
renewable energy projects European Climate Infrastructure and Environment
Executive Agency (2023), and more and more attention is given to the study of
the existing and the future necessary regulations, policies, and incentives
Aras (2021); Ecofys and eclareon (2023); EEA (2020); Kerres et al. (2020).
However, to effectively implement cross-border cooperation, a deep
understanding of the spatial and temporal distribution of wind energy in
different countries and regions is needed. This is where data-mining
visualization tools can play a crucial role by providing an intuitive and
interactive way to explore and analyze wind data from different regions,
helping to identify patterns, trends, and potential issues that would be
difficult to discern from raw data alone.

In this paper, we present a dashboard-like data-mining interface that aims at
making the historical spatial and temporal variability of wind power more
accessible to decision-makers. We also hope to bridge the gap between the
available climate data and the needs of the energy industry looking for a more
intelligible analysis tool that: (1) Uses the latest open-access dataset of
the daily wind capacity factor in 28 European countries between 1979 and 2019;
(2) Proposes more than 16 fully-configurable visualizations of the temporal
(intraday, intrayear, year-over-year), and spatial variations of the wind
power capacity factor; (3) Allows a deep analysis of the distribution, the
cross-country correlations of the temporal distribution, and the drivers of
low wind power events of different European countries.

We expect the platform to be useful to (1): Researchers seeking visual
representations of the historical variability of wind power, the possible
obstacles to the reliability of a power supply system highly relying on wind,
and more generally to the large-scale integration of wind power into the grid;
provide stakeholders (2) with valuable insights into the potential of
countries for wind energy development, as well as the potential for regional
cooperation and help them make well-informed decisions.

##  2 Dataset, development and deployment specifics

Dataset. The dataset used here is an ERA5-derived time series of European
country-aggregate electricity wind power generation Bloomfield et al. (2020).
The authors use the ERA5 reanalysis data Hersbach et al. (2020) to compute the
nationally aggregated hourly capacity factor of 28 European countries between
1979 and 2019. The capacity factor of a wind turbine is the ratio of the
electrical energy produced by the turbine for the period of time considered to
the total amount of electrical energy that could have been produced at full
power operation during the same period. Further details regarding the data are
given in appendix A1.

Software Framework. Our platform is built using Dash, a python library built
on top of React.js that allows building data applications with perfect
integration of Plotly.js. Dash was made popular for its use in bioinformatics
database visualization Shammamah Hossain (2019). Dash uses callbacks to modify
or add new HTML elements as a result of the user’s action. The platform is
accessible at: REmap-eu.app

Interactive Two-Card-Layout.  The layout of the web application is based on
two cards, placed side-by-side. Fig. 1 in the Appendix shows a screenshot of
the platform. Each card groups together elements that interact with each
other. Hence, each of the two cards is independent, with a unique exception:
The choropleth map of the left card is used to filter the countries plotted on
the right card. The left card displays ‘raw’ data, with little pre-processing
involved. It provides a simple yet clear description of the spatial and
temporal distribution of the capacity factor. The right card provides more
elaborated visualizations, that require more pre-processing. Those
visualizations are to be interpreted with the data shown on the left card. The
layout was designed to allow the two plots to be side by side, so that the
user does not need to switch between them.

##  3 Visualizations for the spatial and temporal analysis of wind power

In this section, we introduce the different visualizations describing the
spatial and temporal variability of wind energy. Appendix A2 gives a summary
of the proposed plots and their settings.

Analysing the spatial distribution of wind resources in Europe. European
countries are not equal in terms of wind resources Enevoldsen et al. (2019).
We use a choropleth map A.2 of the country-aggregated capacity factors to
highlight the spatial distribution of wind power. We aim at providing an easy
way to compare the average and the standard deviation of the capacity factor
of European countries, for different time resolutions and over different time
periods. The map shows the average and standard deviation of the capacity
factor of each European country. This type of visualization is often used to
show how different geographic entities compare. The settings available to the
user are summarized in Appendix A2 and Fig. 2 shows a possible configuration
of such a map.

Comparing the temporal distribution of wind resources across countries. We
display a line plot of the capacity factor A.2 below the choropleth map. The
role of the line plot is to show the intraday, intrayear, and yearly capacity
factor variations of European countries, depending on what resolution the user
chose. It also allows comparison of the capacity factor across countries, over
different time windows. The average capacity factor over all 28 European
countries available is displayed, and the line is labeled as “28C”. This
allows the user to compare a country with the Europe-aggregated data, or
compare multiple countries, as shown in Fig. 3.

Intrayear and intraday wind resources variability. The “Intrayear variation
range of the monthly capacity factor” bar plot A.2 displays the variation
range of the monthly capacity factor of each country, as shown in Fig. 4. The
same plot is provided for the intraday variation range of the hourly capacity
factor. Understanding seasonal patterns in historical data can help in
building more effective forecast models, which in turn allows for more
accurate planning and management of the grid. This knowledge can also be used
to develop more effective policies and strategies for integrating wind energy
into the grid Kaspar et al. (2019) and maximizing its potential benefits. For
example, if hybrid energy systems based on solar and wind are often assumed to
better deal with the variability of renewable energies, Mahmud et al. (2022)
showed that in some countries, solar and winter may not complement each other
as well. This is the case when wind production does not increase during the
winter, and as a consequence, does not compensate for the drop in solar energy
production.

Cumulative time above threshold comparison. One way to compare the capacity
factor of different regions is to look at their proportion of days that had a
capacity factor higher than some threshold. This is what can be done using the
“Cumulative days above threshold” plot A.2. The plot supports multi-country
selection, in which case a line corresponding to the data aggregated over the
entire selected region is added, as in Fig. 5. Again, this allows the user to
consider the selected countries as a single unique region in the context of
perfect grid interconnection.

Year-over-year monthly capacity factor comparison. So far, the visualizations
were focusing on the spatial distribution of wind power, and on the comparison
of capacity factor-derived features across countries. However, the increasing
investments into wind energy have pushed for more research on the year-over-
year country-scale evolution of wind energy resources Jung & Schindler (2022).
The long-term evolution of wind resources is important for the calculations
used in the preliminary assessment of energy-producing projects, such as the
levelized cost of energy (LCOE). In the “YoY (year-over-year) monthly capacity
factor comparison” plot A.2, we display the intra-year evolution of the
capacity factor for the selected country and the selected year. The lines
corresponding to the other years of the period 1979-2019 are displayed in
gray, allowing the user to compare the capacity factor of a given year to the
other years of the period. This is shown in Fig. 6.

##  4 Analyzing low wind power events: Frequency, drivers and mitigation
strategies

Unlike solar PV, which exhibits relatively predictable diurnal and seasonal
cycles, wind power has more complex and irregular variations in energy
generation, both at inter-annual and intra-annual scales. In particular, the
study of the temporal distribution of low-wind-power (LWP) events has gained
more attention in the literature Cai & Bréon (2021); Ohlendorf & Schill
(2020). LWP events are becoming a growing concern in countries where wind
power makes up a significant portion of the energy mix, as it raises questions
about energy security and stability. Studying the past occurrences of these
events can provide valuable insights into the drivers of variability and
inform the development of strategies to mitigate their impact. Although there
is not a single definition of low wind power events, they can be defined as an
uninterrupted period of time during which the capacity factor is below some
threshold Patlakas et al. (2017); Leahy & Mckeogh (2013); Ohlendorf & Schill
(2020). In our web app, we arbitrarily set this threshold to 10 percent. This
is a value that will be modifiable by the users in the next version of the
platform.

Comparing the number, the duration, and the distribution of low wind power
events. When the user selects “LP events” in the dropdown, two plots are
displayed. The first one is a bar plot of the number of occurrences of low
wind power events for different minimum durations. The second one is a
calendar plot that indicates the low wind power days in the selected region.
When the user selects multiple countries, the bar chart displays grouped bars
corresponding to each country. This allows for comparing the number of
occurrences of LWP events of each minimum duration across selected countries.
We also add the data corresponding to the selected-region-aggregated data.
This allows the user to see how grid interconnection mitigates the risk of
observing LWP events, as shown in Fig. 7. Indeed, a selected region often has
a lower number of LWP events than each of its constitutive countries, since
LWP events don’t necessarily happen at the same time in all constitutive
countries. The calendar plot indicates the low wind power days at the scale of
the selected region. This plot gives information on the temporal distribution
of those days within the considered year. An example of such plots is shown in
Fig. 8.

Cross-country correlation of the low wind power day temporal distribution. The
previous plots that we described allow the user to compare countries in terms
of their number of LWP events. However, understanding how the capacity factors
of neighboring countries are correlated is of major importance to determine
the interconnection level that could help in alleviating the spatial
variability of wind energy Olauson & Bergkvist (2016); Monforti et al. (2016);
Malvaldi et al. (2017). For example, by understanding the patterns and trends
of low wind power events in different countries, it may be possible to
identify the most appropriate market mechanisms, such as interconnection
capacities, pricing schemes, and balancing mechanisms, that can enable cross-
border cooperation. For this reason, we propose two different choropleth maps.
Both of them require one country to be selected on the left card. The first
plot shows the Pearson correlation coefficient between the selected country
and the other countries in terms of LWP day distribution, see Fig.9. The
second one shows the same statistics, but for the raw capacity factor values.
Only statistically significant (p-value$\geq$0.05) correlations are displayed.

Detecting the possible drivers of low wind power events.  Climate indices
provide a measure of large-scale atmospheric circulation and weather
conditions that impact wind resources. For instance, the North Atlantic
Oscillation (NAO) is one of the most commonly studied climate indices that has
a positive correlation with wind power in Europe Brayshaw et al. (2011); Ely
et al. (2013). We incorporate a plot of the historical climate indices during
low wind speed events, providing a valuable tool for examining the
relationship between climate indices and wind generation. It can help to
identify the indices that are most correlated with wind generation. This
historical information can be used to develop more accurate models for
predicting wind power generation. The plot shows the climate indices for the
selected year and highlights the values corresponding to low wind power days
in the selected country, as shown in Fig. 10. The user can select the climate
index to display among the North Atlantic Oscillation index (NAO), the Artic
Oscillation index (AO), the Madden-Julian Oscillation indices (MJO) for
different longitudes, and the El-Niño Southern Oscillation (NINO).

##  5 Additional features

Electricity prices. It is important to consider the interplay between wind
power, electricity prices, and other factors in order to develop a
comprehensive understanding of the energy market. Low wind power days can have
a significant impact on electricity prices. When wind power generation
decreases, other sources of electricity, such as fossil fuels or hydropower,
need to ramp up production to compensate. This can result in an increase in
electricity prices, as those energy sources typically have a variable cost,
contrary to wind and solar which have no fuel or variable O&M costs. We
propose a plot of the daily average day-ahead electricity prices, shown in Fig
11. We highlight the prices corresponding to low wind power days and display
the correlation between low wind power events and electricity prices, which is
found to be high for European countries that heavily rely on this energy
source. The price data is obtained from Ember Climate (2023).

Solar energy data. Combining wind and solar energy has gained interest as a
way to mitigate their intermittency and variability, creating a more reliable
and stable energy mix. The platform proposes the user to compare different
renewable energy mixes by choosing the weights given to solar and wind energy
in the computation of the capacity factor data displayed in the
visualizations. The solar capacity factor data is also obtained from
Bloomfield et al. (2020). By default, only the data corresponding to wind
energy (respective weights of 1 and 0 for wind and solar) is displayed, and we
only focused on the visualizations obtained using this setting in this paper.

##  6 Conclusion and Future Work

In this paper, we presented a new web platform that offers multiple
visualizations of the temporal and spatial variability of historical wind
energy resources over Europe. The tool proposes configurable plots that allow
the user to deeply analyze the ERA5-derived capacity factor dataset Bloomfield
et al. (2020). Although we primarily expect this platform to be useful to
climate researchers, the energy industry, and the decision-makers, we also
hope to serve the needs of machine learning engineers and scientists looking
for a better understanding of the wind energy resource assessment challenges.
We plan on continuously improving the platform based on the feedback that we
have already received from academics and stakeholders who were introduced to
the tool. Specifically, we will add the demand data. This will allow the study
of the relationship between electricity demand and wind production. We also
plan on adding the demand-net-wind, the electricity demand that needs to be
covered by another energy source than wind.

##  7 Acknowledgments

We thank Dr. Naveen Goutham for his help in correcting the last version of the
paper. We also thank Dr. Hannah Bloomfield and Prof. Emmanuel Pietriga for
their insights and suggestions to improve the platform.

## References

  * Aras (2021) Melis Aras.  Territorial Governance of EU Cross-Border Renewable Energy Cooperation: A Soluble or Turbulent Model in the Current Framework?  _Global Energy Law and Sustainability_ , February 2021.  doi: 10.3366/gels.2021.0048.
  * Bloomfield et al. (2020) Hannah Bloomfield, David Brayshaw, and Andrew Charlton-Perez.  Era5 derived time series of european country-aggregate electricity demand, wind power generation and solar power generation: hourly data from 1979-2019, 2020.
  * Bloomfield et al. (2021) Hannah Bloomfield, David Brayshaw, David Brayshaw, Paula Gonzalez, Paula Gonzalez, and Andrew Charlton-Perez.  Sub-seasonal forecasts of demand and wind power and solar power generation for 28 european countries.  _Earth System Science Data_ , 13:2259–2274, 05 2021.  doi: 10.5194/essd-13-2259-2021.
  * Brayshaw et al. (2011) David James Brayshaw, Alberto Troccoli, Rachael Fordham, and John Methven.  The impact of large scale atmospheric circulation patterns on wind power generation and its potential predictability: A case study over the uk.  _Renewable Energy_ , 36(8):2087–2096, 2011.  ISSN 0960-1481.  doi: https://doi.org/10.1016/j.renene.2011.01.025.  URL https://www.sciencedirect.com/science/article/pii/S0960148111000474.
  * Cai & Bréon (2021) Yiling Cai and François-Marie Bréon.  Wind power potential and intermittency issues in the context of climate change.  _Energy Conversion and Management_ , 240:114276, 2021.  ISSN 0196-8904.  doi: https://doi.org/10.1016/j.enconman.2021.114276.
  * Ecofys and eclareon (2023) Ecofys and eclareon.  Cross-border renewables cooperation, Jan 2023.  URL https://www.agora-energiewende.de/en/publications/cross-border-renewables-cooperation/.
  * EEA (2020) EEA.  Cross-border cooperation on renewable energy, Dec 2020.  URL https://www.eea.europa.eu/themes/energy/renewable-energy/cross-border-cooperation-on-renewable-energy.
  * Ely et al. (2013) Caroline R. Ely, David J. Brayshaw, John Methven, James Cox, and Oliver Pearce.  Implications of the north atlantic oscillation for a uk–norway renewable power system.  _Energy Policy_ , 62:1420–1427, 2013.  ISSN 0301-4215.  doi: https://doi.org/10.1016/j.enpol.2013.06.037.  URL https://www.sciencedirect.com/science/article/pii/S0301421513005223.
  * Ember Climate (2023) Ember Climate.  European wholesale electricity price data, Jan 2023.  URL https://ember-climate.org/data-catalogue/european-wholesale-electricity-price-data/.
  * Enevoldsen et al. (2019) Peter Enevoldsen, Finn-Hendrik Permien, Ines Bakhtaoui, Anna-Katharina von Krauland, Mark Z. Jacobson, George Xydis, Benjamin K. Sovacool, Scott V. Valentine, Daniel Luecht, and Gregory Oxley.  How much wind power potential does europe have? examining european wind power potential with an enhanced socio-technical atlas.  _Energy Policy_ , 132:1092–1100, 2019.  ISSN 0301-4215.  doi: https://doi.org/10.1016/j.enpol.2019.06.064.
  * European Climate Infrastructure and Environment Executive Agency (2023) European Climate Infrastructure and Environment Executive Agency.  2023 call for cross-border renewable energy projects (cb res) to obtain the status, Jan 2023.  URL https://cinea.ec.europa.eu/news-events/news/cef-energy-2023-call-cross-border-renewable-energy-projects-cb-res-obtain-status-2023-01-10_en.
  * Hersbach et al. (2020) Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons, Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Healy, Robin J. Hogan, Elías Hólm, Marta Janisková, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume, and Jean-Noël Thépaut.  The era5 global reanalysis.  _Quarterly Journal of the Royal Meteorological Society_ , 146(730):1999–2049, 2020.  doi: https://doi.org/10.1002/qj.3803.
  * IRENA (2019) IRENA.  Future of wind: Deployment, investment, technology, grid integration and socio-economic aspects (A Global Energy Transformation paper).  Technical report, International Renewable Energy Agency, 2019.
  * Jung & Schindler (2022) Christopher Jung and Dirk Schindler.  Development of onshore wind turbine fleet counteracts climate change-induced reduction in global capacity factor.  _Nature Energy_ , 7, 07 2022.  doi: 10.1038/s41560-022-01056-z.
  * Kaspar et al. (2019) Frank Kaspar, Michael Borsche, Uwe Pfeifroth, Jörg Trentmann, Jaqueline Drücke, and Paul Becker.  A climatological assessment of balancing effects and shortfall risks of photovoltaics and wind energy in germany and europe.  _Advances in Science and Research_ , 16:119–128, 07 2019\.  doi: 10.5194/asr-16-119-2019.
  * Kerres et al. (2020) Pia Kerres, Bastian Lotz, Felix von Blücher, Fabian Wigand, Naida Taso, Jasmina Trhulj, and Dirk Buschle.  Policy brief: Renewables cross-border cooperation in the energy community, Nov 2020.  URL http://aures2project.eu/2020/11/25/policy-brief-renewables-cross-border-cooperation-in-the-energy-community/.
  * Leahy & Mckeogh (2013) Paul Leahy and Eamon Mckeogh.  Persistence of low wind speed conditions and implications for wind power variability.  _Wind Energy_ , 16, 05 2013.  doi: 10.1002/we.1509.
  * Mahmud et al. (2022) Zabir Mahmud, Kenji Shiraishi, Mahmoud Youssef Abido, Dev millstein, Pedro Perez, and Sarah Kurtz.  Geographical variability of summer- and winter-dominant onshore wind.  _Journal of Renewable and Sustainable Energy_ , 14, 03 2022.  doi: 10.1063/5.0070430.
  * Malvaldi et al. (2017) A. Malvaldi, S. Weiss, D. Infield, J. Browell, P. Leahy, and A. M. Foley.  A spatial and temporal correlation analysis of aggregate wind power in an ideally interconnected europe.  _Wind Energy_ , 20(8):1315–1329, 2017.  doi: https://doi.org/10.1002/we.2095.
  * Monforti et al. (2016) F. Monforti, M. Gaetani, and E. Vignati.  How synchronous is wind energy production among european countries?  _Renewable and Sustainable Energy Reviews_ , 59:1622–1638, 2016.  ISSN 1364-0321.  doi: https://doi.org/10.1016/j.rser.2015.12.318.
  * Ohlendorf & Schill (2020) Nils Ohlendorf and Wolf-Peter Schill.  Frequency and duration of low-wind-power events in germany.  _Environmental Research Letters_ , 15, 08 2020.  doi: 10.1088/1748-9326/ab91e9.
  * Olauson & Bergkvist (2016) Jon Olauson and Mikael Bergkvist.  Correlation between wind power generation in the european countries.  _Energy_ , 114:663–670, 11 2016.  doi: 10.1016/j.energy.2016.08.036.
  * Parker (2016) Wendy S. Parker.  Reanalyses and observations: What’s the difference?  _Bulletin of the American Meteorological Society_ , 97(9):1565 – 1572, 2016.  doi: 10.1175/BAMS-D-14-00226.1.
  * Patlakas et al. (2017) Platon Patlakas, George Galanis, Dimitris Diamantis, and George Kallos.  Low wind speed events: persistence and frequency.  _Wind Energy_ , 20, 06 2017.  doi: 10.1002/we.2078.
  * Shammamah Hossain (2019) Shammamah Hossain.  Visualization of Bioinformatics Data with Dash Bio.  In Chris Calloway, David Lippa, Dillon Niederhut, and David Shupe (eds.), _Proceedings of the 18th Python in Science Conference_ , pp. 126 – 133, 2019.  doi: 10.25080/Majora-7ddc1dd1-012.
  * Staffell & Pfenninger (2018) Iain Staffell and Stefan Pfenninger.  The increasing impact of weather on electricity supply and demand.  _Energy_ , 145:65–78, 2018.  ISSN 0360-5442.  doi: https://doi.org/10.1016/j.energy.2017.12.051.

##  Appendix A Appendix

###  A.1 A1 - Dataset

The authors of the dataset Bloomfield et al. (2020) used by the platform use
the ERA5 reanalysis data Hersbach et al. (2020) to compute the nationally
aggregated hourly capacity factor of 28 European countries between 1979 and
2019. Reanalysis combines past weather observations and current atmospheric
models to generate climate and weather historic data Parker (2016). It allows
getting a complete weather record from sparse - both in space and time - past
data. In addition to the wind speed data, the authors use the wind farm
spatial distribution of 2017, taken from https://thewindpower.net. However, it
is worth mentioning that, because the absolute wind power capacity is not used
to compute the capacity factors, only the relative spatial distribution of
wind turbines is assumed to be constant. The capacity factor of each country
is estimated by aggregating the capacity factor computed for each grid box,
weighted by its estimated installed capacity. The capacity factor in each grid
box is derived using the 100 m wind speeds and the power curve of the type of
wind turbine maximizing the energy produced during the entire period
(1979-2019), as indicated in Bloomfield et al. (2021). Although the authors
reported an average percentage error of 10 percent in the validation settings,
it is important to note that capacity factors indicated here may deviate from
the true values.

###  A.2 A2 - Summary of the visualizations described in Section 3

Description |  User interaction |  Comments
---|---|---
[1] Choropleth map of the average or standard deviation of the hourly, monthly or yearly capacity factor |  \- The right dropdown sets the data resolution (yearly, monthly, or hourly). \- The left dropdown allows switching between average and std of the capacity factor \- The slider filters the year, month or hours included in the data |  \- The Average and standard deviation are computed over the entire time period (1979-2019), except if the time resolution is set to yearly, in which case the user can decide what years to include
|  |
[2] Line plot of the intraday, intrayear or yearly average capacity factor |  \- The countries displayed on the plot are those selected when clicking on the choropleth map (holding shift allows multi-selection) \- The range slider can then be used to filter the time period that is displayed |  \- When no country is selected (initial state), the average capacity factor over all 28 European countries available is displayed, and the line is labelled as “28C”
|  |
[3] Bar plots of the Intrayear/intraday variation range of the monthly/ hourly capacity factor |  \- The dropdown located on the top part of the card is used to select the year of the data to be shown \- The countries displayed on the plot are those selected when clicking on the choropleth map (holding shift allows multi-selection) |  \- When no country is selected on the left card, all European countries are plotted, in ascending order of values. This is because the goal of this visualization is to provide a ranking of the countries based on their capacity factor variability \- The scale starts at 0 for the difference in bar heights to accurately represent the difference in capacity factor variation range. \- The color used for each country is consistent across all visualizations of the web app, allowing the user to quickly spot the selected countries on each plot
|  |
[4] Plot of the minimum, maximum and average intraday/ intrayear capacity factor |  as in 3. |  \- The scatter plot displays the mean with a cross and the min and max values with error bars \- The countries are ordered by mean values
|  |
[5] Line plot of the cumulative days above capacity factor threshold |  as in 3. Moreover : \- When multiple countries are selected, an additional line corresponding to the selected region (average over all selected countries) is displayed |  \- When no country is selected, the data for the 28 countries-aggregated capacity factor is displayed, and the lines corresponding to each European country are displayed in light gray. This allows the user to quickly see how each country compares to the other 27 European countries by hovering over its corresponding line
|  |
[6] YoY (year-over-year) monthly capacity factor plot |  as in 4. |  \- As the goal here is really to compare the intrayear capacity factor over the years for a single region, selecting multiple countries results in showing only one single line corresponding to the data aggregated over the entire selected region \- The lines corresponding to the other years of the period 1979-2019 are displayed in light gray, allowing the user to quickly figure out how the capacity factor of a given year compares to the other years of the period
|  |

###  A.3 A3 - figures and use-cases examples

Figure 1: Layout of the web app. The left card shows the choropleth map A.2
and the line plot A.2 of the capacity factor, while the right one offers more
advanced visualizations. The choropleth map of the left card is used to filter
the data plotted on the right card. Figure 2: Choropleth map A.2 of the
average yearly capacity factor over the period 1984-2019. The range slider and
the two dropdowns are used to parametrize the visualization. Figure 3: Line
plot A.2 showing the average monthly capacity factor of France, the UK, and
the average capacity factor over the 28 European countries (28C) for the
period 1979-2019. The scale starts at 0 to allow a better comparison between
countries. The user selects the countries to be displayed by clicking on the
choropleth map A.2 shown in Fig. 2 and sets the time resolution using the
dropdown shown in Fig. 1. Figure 4: Bar chart of the intrayear variation range
of the monthly capacity factor of the 28 countries in 1979. The bar
corresponding to the 28 countries-aggregated capacity factor range is shown in
red. This chart helps in identifying countries that suffer from a large
capacity factor gap between high and low wind power months. Figure 5: Line
plot of the cumulative days above thresholds for the year 1979. We display the
data for France and Denmark, and the data aggregated over the region
(France+Denmark) in blue. When neighboring countries are connected, excess
energy from windy regions can be transferred to regions with less wind,
reducing the need for reserve capacity. In practice, this can lead to an
increase in the overall capacity factor of wind power in the interconnected
system. Figure 6: Year-over-Year monthly capacity factor comparison for
France, The highlighted year is 2018. The gray lines correspond to the other
years of the period (1979-2019). 2018 showed a particularly low wind
production during summer (note how the height of the green line compares to
the gray lines).  Figure 7: Number of low wind power events in France,
Germany, and in the entire (France+Germany) region for different minimum LWP
events duration in 1979. Interconnected countries have a tendency to
experience fewer low wind power events due to the ability of one country to
compensate for the other’s low capacity factor, particularly when they have
distinct coastal regions and are exposed to varying wind patterns. Grid
interconnections generally enhance the reliability of energy supply.

Figure 8: Calendar plots of the low wind power days in France (top figure),
and in the region (France+Germany) in 2018. Low wind power days are displayed
in red. The observation is similar to the one made about Fig. 7. Figure 9:
Correlation between the LWP events distribution of France and the ones of the
28 other European countries. Figure 10: Line plot of the NAO climate index in
2018. The red dots correspond to low wind power days in the selected country
(here, France). After having identified a low wind power event using the
calendar plot shown in Fig. 8, a typical workflow would be to use this
visualization to investigate the climate indices preceding this event. Figure
11: Line plot of the electricity prices in Germany. The red dots correspond to
low wind power days. The correlation between wind capacity factor and
electricity prices is particularly high in countries that highly rely on wind,
as is the case of Germany.