# Denmark’s Participation in the Search Engine TREC COVID-19 Challenge:
Lessons Learned about Searching for Precise Biomedical Scientific Information
on COVID-19

Lucas Chaves Lima The first and second author contributed equally. Department
of Computer Science, University of Copenhagen, Denmark  Casper
Hansen††footnotemark:  Department of Computer Science, University of
Copenhagen, Denmark  Christian Hansen  Department of Computer Science,
University of Copenhagen, Denmark  Dongsheng Wang  Department of Computer
Science, University of Copenhagen, Denmark  Maria Maistro  Department of
Computer Science, University of Copenhagen, Denmark  Birger Larsen  Department
of Communication and Psychology, Aalborg University, Denmark  Jakob Grue
Simonsen  Department of Computer Science, University of Copenhagen, Denmark
Christina Lioma  Department of Computer Science, University of Copenhagen,
Denmark

##  1 Introduction

This report describes the participation of two Danish universities, University
of Copenhagen and Aalborg University, in the international search engine
competition on COVID-19 (the _2020 TREC-COVID Challenge_ [25]) organised by
the U.S. National Institute of Standards and Technology (NIST) and its Text
Retrieval Conference (TREC) division. The aim of the competition was to find
the best search engine strategy for retrieving precise biomedical scientific
information on COVID-19 from the largest, at that point in time, dataset of
curated scientific literature on COVID-19—the COVID-19 Open Research Dataset
(CORD-19). CORD-19 was the result of a call to action to the tech community by
the U.S. White House in March 2020, and was shortly thereafter posted on
Kaggle as an AI competition by the Allen Institute for AI, the Chan Zuckerberg
Initiative, Georgetown University’s Center for Security and Emerging
Technology, Microsoft, and the National Library of Medicine at the US National
Institutes of Health111 https://www.whitehouse.gov/briefings-statements/call-
action-tech-community-new-machine-readable-covid-19-dataset/. CORD-19
contained over 200,000 scholarly articles (of which more than 100,000 were
with full text) about COVID-19, SARS-CoV-2, and related coronaviruses,
gathered from curated biomedical sources (listed in Table 2) [31]. The TREC-
COVID challenge asked for the best way to (a) retrieve accurate and precise
scientific information, in response to some queries formulated by biomedical
experts, and (b) rank this information decreasingly by its relevance to the
query.

In this document, we describe the TREC-COVID competition setup (Section 2),
our participation to it (Section 3), and our resulting reflections and lessons
learned about the state-of-art technology when faced with the acute task of
retrieving precise scientific information from a rapidly growing corpus of
literature, in response to highly specialised queries, in the middle of a
pandemic [13] (Section 4).

##  2 The TREC-COVID Competition Setup

The general idea of the TREC-COVID competition was similar to that of standard
ad-hoc TREC retrieval competitions: given some dataset and some queries, the
goal was to find the best way of retrieving the most precise and accurate
information from the dataset, and of ranking this information decreasingly by
its relevance to each query. At the beginning of the competition, no labels or
ground truth of any kind were provided, so the setup was unsupervised;
gradually, during the competition, a proportion of ground truth was released.
We explain this next.

The competition consisted of five rounds (see Figure 1). In the first round, a
version of the dataset was released, together with 30 queries [31].
Participating teams had approximately one week to submit rankings of maximum
1,000 results per query (also known as runs). At the end of round 1, all
participating runs were collected, pooled, and the top results in the whole
pool were manually assessed by expert biomedical assessors as relevant,
partially relevant, or not relevant. Using these assessments, all the
participating runs were evaluated, and the results of this evaluation
(together with the expert assessments) were released to the participants. Runs
were evaluated with respect to NDCG@K, Precision@K, Brepf, RBP ($p=0.5$), and
MAP, with a varying cut-off $K$, using
trec_eval222https://github.com/usnistgov/trec_eval. Rounds two to five of the
competition had the same format, with the difference that in each round, a new
version of the dataset was released, and 5 new queries were added to the set
of queries. The differences in the versions of the dataset between rounds one
to five resulted from either the addition of new scientific literature into
the dataset, or from the removal of scientific literature from the dataset
(for instance, in case of duplicates, or literature that was irrelevant or of
unreliable provenance)333https://ir.nist.gov/covidSubmit/data.html.

Figure 1: Overview of the TREC-COVID competition setup (image source: Roberts
et al. [25]).

The CORD-19 dataset [34]444https://www.semanticscholar.org/cord19 used in the
competition contained scientific literature on COVID-19 and coronaviruses
collected from PubMed, WHO, bioRxiv and medRxiv, and the list of metadata
shown in Table 1. The queries had the standard TREC format and the following
three fields: the succinct query text (query), a question describing the query
(question), and a slightly longer elaboration of the information need
(description). Figure 2 shows an example query.

<topic number="1"><query>coronavirus origin</query><question>what is the
origin of COVID-19</question><narrative>seeking range of information about the
SARS-CoV-2 virus’s origin, including \newlineits evolution, animal source, and
first transmission into humans</narrative></topic>

Figure 2: Example of query. Table 1: Metadata in CORD-19. Metadata | Description
---|---
cord_uid | Each unique article receives a corresponding ‘cord_uid’
title | Title of the article
doi | Digital object identifier
url | Source url of the article
pmcid | PubMed Central (PMC) identifier
pubmed_id | Pubmed Identifier
abstract | Full abstract
has_full_text | name of full text file of this article
publish_time | Time of publication
authors | Author list
Microsoft Academic Paper ID | Microsoft Academic ID mapping
WHO #Covidence | World Health Organization Covidence Identifier

##  3 Our Participation in TREC-COVID

Our participating runs can be grouped into three high-level categories: (1)
using standard, well-understood methods for retrieval; (2) using state-of-the-
art methods for data representation; (3) using meta-search heuristics. We
explain each of these below.

###  3.1 Standard, well-understood retrieval methods

We used the Best Match 25 (BM25) retrieval model on default parameters
($k_{1}=1.2$ and $b=0.75$) [26], as implemented in the Indri search
engine555https://www.lemurproject.org/indri/, and reranked its top 50 results
to prioritise articles published in 2020. The motivation behind this reranking
was to boost the retrieval of recent articles, to match the recency of
COVID-19.

###  3.2 State-of-the-art data representation methods

We used multiple techniques to boost the semantic representation of the data.
Our motivation was that the more refined and precise the semantic
representation of the articles and queries, the better their matching by the
retrieval model. Below, we first present methods applied only to queries, and
subsequently methods applied to both queries and documents.

#### Query expansion and entity detection in queries

Motivated by the fact that vagueness, ambiguity, and domain-specificity are
three well-known factors that render queries difficult for both machines and
humans to process [21], our first approach was to use query expansion as a
means of boosting the semantic representation of queries. Query expansion has
been found useful as a means for disambiguation, especially in scientific
domains [14, 20] or technical areas where metadata is available [17]. We
expanded queries with terms from the Human Disease Ontology
(HDO)666https://www.ebi.ac.uk/ols/ontologies/doid. The HDO contains
standardised descriptions of human disease terms that are hierarchically
organised as an ontology. For example, COVID-19 is a leaf concept, whose
parent concept is coronavirus infection, and whose further parent concept is
viral infectious disease. Similarly to prior work in expanding queries from
ontologies [33], we retrieved the parent-class and the sub-class disease
concepts and used them to expand the queries.

We also enhanced the semantic representation of queries by identifying
grammatical entities with Lexigram777https://www.lexigram.io, which is a
semantic analysis tool for healthcare data. We used it to extract entities and
the types of those entities from the queries. For example, in the text pain in
the stomach, the entities would be pain, stomach, and their types would be
condition for pain, and anatomy for stomach.

Having expanded the queries with terms from HDO and having identified query
entities and their types with Lexigram, we then reweighted query terms based
on the weighting schema of Faessler et al. [7]: original query terms with
weight $1.0$; alternative disease terms from HDO with weight $0.7$; entity
labels with weight $0.4$; and entity types with weight $0.1$. This was
implemented in Indri with the #weight operator.

#### BioBERT semantic embeddings for queries and documents

Motivated by the superior performance of word embeddings as semantic
representation across various tasks [24, 32], we generated semantic
representations for each query and for each article (separately for each
paragraph per article, concatenated with the title and abstract of the article
each time) using pre-trained embeddings with BioBERT [18], as implemented in
the HuggingFace Transformer library. We computed the similarity between each
query and article paragraph when represented like this, and used this
similarity to re-rank the rankings produced by BM25 as described in Section
3.1.

###  3.3 Meta-search heuristics

Our final strategy consisted of: (1) generating different versions of queries
and of indices representing the articles in the dataset; (2) producing
different rankings using combinations of (1); and (3) fusing the rankings in
(2) into a single fused ranking. We explain this next.

We created three indices:

  * •

An index based on the title and abstract of each document;

  * •

An index based on the full text of each document;

  * •

Each document was split into its paragraphs, such that from each paragraph a
new _artificial_ document was constructed based on the title, abstract, and
that paragraph. We built an index based on these constructed paragraph-level
documents.

To generate multiple queries, we use named entity
recognition888https://spacy.io/usage/linguistic-features#named-entities to
extract additional query terms. Let NE$(\cdot)$ denote the function that
extracts named entities from a query; then we created the following four
different query variations:

  * •

Query and NE(question);

  * •

Query, NE(question), and NE(narrative);

  * •

Question and NE(query);

  * •

Question, NE(query), and NE(narrative).

For each query variation we created four runs using BM25 with RM3 with default
parameters999https://github.com/castorini/pyserini:

  * •

We queried the index of title and abstract;

  * •

We queried the index of full text;

  * •

We queried the paragraph-level index;

  * •

We queried the paragraph-level index, where we only kept the first occurrence
of the original document id associated with each paragraph.

We applied Reciprocal Rank Fusion (RRF) [3] (using default $k=60$) to combine
the above four runs into one fused run. The precise details of our fused runs
are included in Appendix A.

The results of the competition are in the process of being published at:
https://ir.nist.gov/covidSubmit/data.html.

##  4 Lessons Learned

#### Non-expert assessments have 47% disagreement with expert assessments.

In the first round of the competition, we recruited a group 8 of non-experts
to assess the top retrieved results of the 30 queries of round 1. The non-
experts were highly educated computer scientists with no background in
biological, medical or health sciences. We compared their non-expert
assessments to the TREC assessments of those same 30 queries (which were
produced by expert bioscientists): the disagreement between the two sets of
assessments is approximately 47%. This is very close to random and indicates
that our non-expert assessments should not be used, even for weak supervision.
The cost of producing them was higher than any gain they can yield.

#### 87% of all relevant and partially relevant articles came from Elsevier,
medRxiv and PMC.

Table 2 displays the portion of partially relevant and relevant articles per
article source. We see that approximately 87% of all partially relevant and
relevant articles came from three sources: Elsevier, medRxiv and PMC. This
agrees with prior findings about the importance of the data source for
scholarly curated information retrieval [4, 5, 6].

#### BioBERT pre-trained embeddings were expensive to use but did not help.

The state-of-art in semantic representation, BERT embeddings, pretrained on
domain specific data, did not bring performance gains of any significance or
reliability, over standard word count based semantic representation (which are
also less computationally expensive than embeddings). This finding holds for
this specific dataset and retrieval setup, and we abstain from generalising
it. Detailed figures can be found in:
https://ir.nist.gov/covidSubmit/data.html. Dedicated research is needed to
investigate the extent to which retrieval performance improvements can come
from using advanced embedding representations, and to identify ways to
facilitate this (for instance, how to overcome the limitations in the length
of input accepted by BERT). Steps in this direction have already been taken
[15].

Another aspect to consider is the importance of fine-tuning BERT models. For
our submission we did not fine tune BioBERT and this likely affected the
performance. Other teams could effectively exploit BioBERT and SciBERT by
finetuning with MS-MARCO [23, 35] and other datasets [19].

#### There was not enough data to use advanced machine learning.

When working in a completely unsupervised setup (as in the first round of the
competition), or in a semi-supervised setup (as in rounds two to five), and
when working with relatively few data points (maximum 50 queries and maximum
200,000 documents), standard retrieval methods can give reliable and
satisfactory performance. This amount of data is too small to use advanced
machine learning, in the sense that the data would not be enough for the model
to learn anything that performs competitively enough and that can generalise
to out of sample data. As future direction we will investigate how to train
advanced models on different datasets and then exploit transfer learning
techniques [30].

Table 2: Number of unique documents that were assessed as partially relevant ($1$) and/or relevant ($2$) over sources. Statistics computed on the CORD-19 dataset released on April 10th, 2020. Source | Partially Relevant (1) | Relevant (2) | # documents per source
---|---|---|---
biorxiv | 45 (4.66%) | 62 (5.88%) | 764
CZI | 19 (1.97%) | 17 (1.61%) | 117
Elsevier | 368 (38.1%) | 374 (35.48%) | 19457
medrxiv | 178 (18.43%) | 348 (33.02%) | 1088
PMC | 312 (32.3%) | 183 (17.36%) | 28648
WHO | 44 (4.55%) | 70 (6.64%) | 1004
Total | 966 (100%) | 1054 (100%) | 51078

##  5 Conclusion

This document describes the participation of two Danish universities,
University of Copenhagen and Aalborg University, in the international search
engine competition on COVID-19 organised by the U.S. National Institute of
Standards and Technoloy (NIST) and its Text Retrieval Conference (TREC)
division. Our retrieval strategy combined (i) standard, well-understood
retrieval methods, (ii) state-of-the-art methods of semantic representation,
and (iii) meta-search heuristics. Our findings, and overall observations from
the competition are that: expert assessments could not be approximated to any
useful degree by non-experts; the source of the scientific literature is a
strong indicator of relevance; semantic term embeddings did not yield
performance improvements over traditional word count based representations;
and the data was generally too small for advanced machine learning methods.
While there is no doubt that information retrieval test collections are needed
for specialised domains like this [16], any use of advanced machine learning
that can generalise to out of sample data requires larger scale training data.

Moving forward, one future direction of work would be to combine automatic
query generation methods [29] together with standard, well-understood
retrieval methods, in order to increase significantly the number of query-
document pairs. The top positions of such rankings (if a critical mass of them
existed) could then be used as weak supervision [8, 10], or with autoencoder
architectures that learn semantic representations [9, 12, 11], or for learning
to rank with online learning [1, 2], for instance. Another future research
direction would be to develop more refined evaluation measures that assess not
only the relevance of the retrieved results to the query, but also qualitative
aspects of the results as a factor of their ranking position, such as
credibility, reliability, or usefulness [22].

## Acknowledgments

This research is partially supported by QUARTZ (721321, EU H2020 MSCA-ITN).

## References

  * Brost et al. [2016a] B. Brost, I. J. Cox, Y. Seldin, and C. Lioma.  An improved multileaving algorithm for online ranker evaluation.  In R. Perego, F. Sebastiani, J. A. Aslam, I. Ruthven, and J. Zobel, editors, _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016_ , pages 745–748. ACM, 2016a.  doi: 10.1145/2911451.2914706.  URL https://doi.org/10.1145/2911451.2914706.
  * Brost et al. [2016b] B. Brost, Y. Seldin, I. J. Cox, and C. Lioma.  Multi-dueling bandits and their application to online ranker evaluation.  In S. Mukhopadhyay, C. Zhai, E. Bertino, F. Crestani, J. Mostafa, J. Tang, L. Si, X. Zhou, Y. Chang, Y. Li, and P. Sondhi, editors, _Proceedings of the 25th ACM International Conference on Information and Knowledge Management, CIKM 2016, Indianapolis, IN, USA, October 24-28, 2016_ , pages 2161–2166. ACM, 2016b.  doi: 10.1145/2983323.2983659.  URL https://doi.org/10.1145/2983323.2983659.
  * Cormack et al. [2009] G. V. Cormack, C. L. Clarke, and S. Buettcher.  Reciprocal rank fusion outperforms condorcet and individual rank learning methods.  In _Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval_ , pages 758–759, 2009.
  * Dragusin et al. [2011] R. Dragusin, P. Petcu, C. Lioma, B. Larsen, H. Jørgensen, and O. Winther.  Rare disease diagnosis as an information retrieval task.  In G. Amati and F. Crestani, editors, _Advances in Information Retrieval Theory - Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011. Proceedings_ , volume 6931 of _Lecture Notes in Computer Science_ , pages 356–359. Springer, 2011.  doi: 10.1007/978-3-642-23318-0“˙38.  URL https://doi.org/10.1007/978-3-642-23318-0_38.
  * Dragusin et al. [2013a] R. Dragusin, P. Petcu, C. Lioma, B. Larsen, H. Jørgensen, I. Cox, L. Hansen, P. Ingwersen, and O. Winther.  Findzebra: A search engine for rare diseases.  _International Journal of Medical Informatics_ , 82(6):528–538, 2013a.  ISSN 1386-5056.  doi: 10.1016/j.ijmedinf.2013.01.005.
  * Dragusin et al. [2013b] R. Dragusin, P. Petcu, C. Lioma, B. Larsen, H. L. Jørgensen, I. J. Cox, L. K. Hansen, P. Ingwersen, and O. Winther.  Specialized tools are needed when searching the web for rare disease diagnoses.  _Rare Diseases_ , 1(1):e25001, 2013b.  doi: 10.4161/rdis.25001.  URL https://doi.org/10.4161/rdis.25001.  PMID: 25002998.
  * [7] E. Faessler, M. Oleynik, and U. Hahn.  Julie lab & med uni graz@ trec 2019 precision medicine track.
  * Hansen et al. [2019a] C. Hansen, C. Hansen, S. Alstrup, J. G. Simonsen, and C. Lioma.  Neural check-worthiness ranking with weak supervision: Finding sentences for fact-checking.  In S. Amer-Yahia, M. Mahdian, A. Goel, G. Houben, K. Lerman, J. J. McAuley, R. Baeza-Yates, and L. Zia, editors, _Companion of The 2019 World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_ , pages 994–1000. ACM, 2019a.  doi: 10.1145/3308560.3316736.  URL https://doi.org/10.1145/3308560.3316736.
  * Hansen et al. [2019b] C. Hansen, C. Hansen, J. G. Simonsen, S. Alstrup, and C. Lioma.  Unsupervised neural generative semantic hashing.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 735–744, New York, NY, USA, 2019b. Association for Computing Machinery.  ISBN 9781450361729.  doi: 10.1145/3331184.3331255.  URL https://doi.org/10.1145/3331184.3331255.
  * Hansen et al. [2019c] C. Hansen, C. Hansen, J. G. Simonsen, and C. Lioma.  Neural weakly supervised fact check-worthiness detection with contrastive sampling-based ranking loss.  In L. Cappellato, N. Ferro, D. E. Losada, and H. Müller, editors, _Working Notes of CLEF 2019 - Conference and Labs of the Evaluation Forum, Lugano, Switzerland, September 9-12, 2019_ , volume 2380 of _CEUR Workshop Proceedings_. CEUR-WS.org, 2019c.  URL http://ceur-ws.org/Vol-2380/paper_56.pdf.
  * Hansen et al. [2020a] C. Hansen, C. Hansen, J. G. Simonsen, S. Alstrup, and C. Lioma.  Content-aware neural hashing for cold-start recommendation.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’20, page 971–980, New York, NY, USA, 2020a. Association for Computing Machinery.  ISBN 9781450380164.  doi: 10.1145/3397271.3401060.  URL https://doi.org/10.1145/3397271.3401060.
  * Hansen et al. [2020b] C. Hansen, C. Hansen, J. G. Simonsen, S. Alstrup, and C. Lioma.  Unsupervised semantic hashing with pairwise reconstruction.  In J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu, editors, _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , pages 2009–2012. ACM, 2020b.  doi: 10.1145/3397271.3401220.  URL https://doi.org/10.1145/3397271.3401220.
  * Jimenez-Solem et al. [2020] E. Jimenez-Solem, T. Petersen, C. Hansen, C. Hansen, C. Lioma, C. Igel, W. Boomsma, O. Krause, S. Lorenzen, R. Selvan, J. Petersen, M. E. Nyeland, M. Zöllner Ankarfeldt, G. M. Virenfeldt, M. Winther-Jensen, A. Linneberg, M. M. Ghazi, N. Detlefsen, A. Lauritzen, A. G. Smith, M. de Bruijne, B. Ibragimov, J. Petersen, M. Lillholm, J. Middleton, S. H. Mogensen, H.-C. Thorsen-Meyer, A. Perner, M. Helleberg, B. S. Kaas-Hansen, M. Bonde, A. Bonde, A. Pai, M. Nielsen, and M. Sillesen.  Developing and validating covid-19 adverse outcome risk prediction models from a bi-national european cohort of 5594 patients, 2020.  URL https://europepmc.org/article/PPR/PPR224419.
  * Jochim et al. [2011] C. Jochim, C. Lioma, and H. Schütze.  Expanding queries with term and phrase translations in patent retrieval.  In _Proceedings of the Second International Conference on Multidisciplinary Information Retrieval Facility_ , IRFC’11, page 16–29, Berlin, Heidelberg, 2011. Springer-Verlag.  ISBN 9783642213526.
  * Khattab and Zaharia [2020] O. Khattab and M. Zaharia.  Colbert: Efficient and effective passage search via contextualized late interaction over BERT.  In J. Huang, Y. Chang, X. Cheng, J. Kamps, V. Murdock, J. Wen, and Y. Liu, editors, _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , pages 39–48. ACM, 2020.  doi: 10.1145/3397271.3401075.  URL https://doi.org/10.1145/3397271.3401075.
  * Larsen and Lioma [2016] B. Larsen and C. Lioma.  On the need for and provision of an ’ideal’ scholarly information retrieval test collection.  In P. Mayr, I. Frommholz, and G. Cabanac, editors, _Proceedings of the Third Workshop on Bibliometric-enhanced Information Retrieval (BIR 2016), Padova, Italy, March 20, 2016_ , CEUR Workshop Proceedings, pages 73–81. CEUR-WS.org, 2016.  null ; Conference date: 20-03-2016 Through 20-03-2016.
  * Larsen et al. [2012] B. Larsen, C. Lioma, I. Frommholz, and H. Schütze.  Preliminary study of technical terminology for the retrieval of scientific book metadata records.  In _Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’12, page 1131–1132, New York, NY, USA, 2012. Association for Computing Machinery.  ISBN 9781450314725.  doi: 10.1145/2348283.2348504.  URL https://doi.org/10.1145/2348283.2348504.
  * Lee et al. [2020] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinform._ , 36(4):1234–1240, 2020.  doi: 10.1093/bioinformatics/btz682.  URL https://doi.org/10.1093/bioinformatics/btz682.
  * Li et al. [2020] C. Li, A. Yates, S. MacAvaney, B. He, and Y. Sun.  PARADE: Passage Representation Aggregation for Document Reranking.  2020\.  URL https://arxiv.org/abs/2008.09093.
  * Lioma et al. [2011a] C. Lioma, A. Kothari, and H. Schuetze.  Sense discrimination for physics retrieval.  In _Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’11, page 1101–1102, New York, NY, USA, 2011a. Association for Computing Machinery.  ISBN 9781450307574.  doi: 10.1145/2009916.2010069.  URL https://doi.org/10.1145/2009916.2010069.
  * Lioma et al. [2011b] C. Lioma, B. Larsen, and H. Schütze.  User perspectives on query difficulty.  In G. Amati and F. Crestani, editors, _Advances in Information Retrieval Theory - Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011. Proceedings_ , volume 6931 of _Lecture Notes in Computer Science_ , pages 3–14. Springer, 2011b.  doi: 10.1007/978-3-642-23318-0“˙3.  URL https://doi.org/10.1007/978-3-642-23318-0_3.
  * Lioma et al. [2017] C. Lioma, J. G. Simonsen, and B. Larsen.  Evaluation measures for relevance and credibility in ranked lists.  In J. Kamps, E. Kanoulas, M. de Rijke, H. Fang, and E. Yilmaz, editors, _Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2017, Amsterdam, The Netherlands, October 1-4, 2017_ , pages 91–98. ACM, 2017.  doi: 10.1145/3121050.3121072.  URL https://doi.org/10.1145/3121050.3121072.
  * MacAvaney et al. [2020] S. MacAvaney, A. Cohan, and N. Goharian.  SLEDGE: A Simple Yet Effective Baseline for COVID-19 Scientific Knowledge Search.  2020\.  URL https://arxiv.org/abs/2005.02365.
  * Naseem et al. [2020] U. Naseem, K. Musial, P. W. Eklund, and M. Prasad.  Biomedical named-entity recognition by hierarchically fusing biobert representations and deep contextual-level word-embedding.  In _2020 International Joint Conference on Neural Networks, IJCNN 2020, Glasgow, United Kingdom, July 19-24, 2020_ , pages 1–8. IEEE, 2020\.  doi: 10.1109/IJCNN48605.2020.9206808.  URL https://doi.org/10.1109/IJCNN48605.2020.9206808.
  * Roberts et al. [2020] K. Roberts, T. Alam, S. Bedrick, D. Demner-Fushman, K. Lo, I. Soboroff, E. Voorhees, L. L. Wang, and W. R. Hersh.  TREC-COVID: rationale and structure of an information retrieval shared task for COVID-19.  _Journal of the American Medical Informatics Association_ , 27(9):1431–1436, 07 2020.  ISSN 1527-974X.  doi: 10.1093/jamia/ocaa091.  URL https://doi.org/10.1093/jamia/ocaa091.
  * Robertson and Zaragoza [2009] S. E. Robertson and H. Zaragoza.  The probabilistic relevance framework: BM25 and beyond.  _Found. Trends Inf. Retr._ , 3(4):333–389, 2009\.  doi: 10.1561/1500000019.  URL https://doi.org/10.1561/1500000019.
  * Sakai and Lin [2010] T. Sakai and C. Lin.  Ranking retrieval systems without relevance assessments: Revisited.  In T. Sakai, M. Sanderson, and W. Webber, editors, _Proceedings of the 3rd International Workshop on Evaluating Information Access, EVIA 2010, National Center of Sciences, Tokyo, Japan, June 15, 2010_ , pages 25–33. National Institute of Informatics (NII), 2010.  URL http://research.nii.ac.jp/ntcir/workshop/OnlineProceedings8/EVIA/05-EVIA2010-SakaiT.pdf.
  * Soboroff et al. [2001] I. Soboroff, C. Nicholas, and P. Cahan.  Ranking retrieval systems without relevance judgments.  In _Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’01, page 66–73, New York, NY, USA, 2001. Association for Computing Machinery.  ISBN 1581133316.  doi: 10.1145/383952.383961.  URL https://doi.org/10.1145/383952.383961.
  * Sordoni et al. [2015] A. Sordoni, Y. Bengio, H. Vahabi, C. Lioma, J. Grue Simonsen, and J.-Y. Nie.  A hierarchical recurrent encoder-decoder for generative context-aware query suggestion.  In _Proceedings of the 24th ACM International on Conference on Information and Knowledge Management_ , CIKM ’15, page 553–562, New York, NY, USA, 2015. Association for Computing Machinery.  ISBN 9781450337946.  doi: 10.1145/2806416.2806493.  URL https://doi.org/10.1145/2806416.2806493.
  * Torrey and Shavlik [2010] L. Torrey and J. Shavlik.  Transfer learning.  In _Handbook of research on machine learning applications and trends: algorithms, methods, and techniques_ , pages 242–264. IGI global, 2010\.
  * Voorhees et al. [2020] E. M. Voorhees, T. Alam, S. Bedrick, D. Demner-Fushman, W. R. Hersh, K. Lo, K. Roberts, I. Soboroff, and L. L. Wang.  TREC-COVID: constructing a pandemic information retrieval test collection.  _CoRR_ , abs/2005.04474, 2020.  URL https://arxiv.org/abs/2005.04474.
  * Wang et al. [2020a] B. Wang, D. Zhao, C. Lioma, Q. Li, P. Zhang, and J. G. Simonsen.  Encoding word order in complex embeddings.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020a.  URL https://openreview.net/forum?id=Hke-WTVtwr.
  * Wang et al. [2019] D. Wang, Q. Li, L. Chaves Lima, J. Grue Simonsen, and C. Lioma.  Contextual compositionality detection with external knowledge bases and word embeddings.  In _Companion Proceedings of The 2019 World Wide Web Conference_ , WWW ’19, page 317–323, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450366755.  doi: 10.1145/3308560.3316584.  URL https://doi.org/10.1145/3308560.3316584.
  * Wang et al. [2020b] L. L. Wang, K. Lo, Y. Chandrasekhar, R. Reas, J. Yang, D. Eide, K. Funk, R. M. Kinney, Z. Liu, W. Merrill, P. Mooney, D. Murdick, D. Rishi, J. Sheehan, Z. Shen, B. Stilson, A. Wade, K. Wang, C. Wilhelm, B. Xie, D. Raymond, D. S. Weld, O. Etzioni, and S. Kohlmeier.  CORD-19: The Covid-19 Open Research Dataset.  _ArXiv_ , 2020b.
  * Zhang et al. [2020] E. Zhang, N. Gupta, R. Tang, X. Han, R. Pradeep, K. Lu, Y. Zhang, R. Nogueira, K. Cho, H. Fang, and J. Lin.  Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset.  2020\.  URL https://arxiv.org/abs/2007.07846.

##  Appendix A Fusion Run Details

Based on the indices and query variations outlined in Section 3.3 we created
different fused runs for both round 2 and 3 in the challenge:

Round 2 and 3 - RunID: fusionOfFusions

We obtain one fused run from each of the query variations (4 in total), which
we further combine using RRF into a final run denoted as fusion of fusions.

Round 3 - RunID: fusionOfRuns

We obtain four runs for each of the query variations (16 in total), which we
combine using RRF into a final run denoted as fusion of runs.

Round 2 - RunID: allFiltering

We apply RRF on all the automatic runs (27 in total), that did not contribute
to the pooling from round 1, combined with our fusionOfFusions and
fusionOfRuns runs.

Round 2 - RunID: soboroffFiltering

We apply RRF on the 9 automatic runs, that did not contribute to the pooling
and were chosen based on the 9 middle runs returned by applying Soboroff’s
method [28, 27] on the 27 runs that did not contribute to the pooling from
round 1. Additionally, we include our fusionOfFusions and fusionOfRuns runs.