# Solving Hard Analogy Questions with Relation Embedding Chains

Nitesh Kumar  Steven Schockaert
Cardiff NLP, School of Computer Science and Informatics
Cardiff University, United Kingdom
<EMAIL_ADDRESS>

###### Abstract

Modelling how concepts are related is a central topic in Lexical Semantics. A
common strategy is to rely on knowledge graphs (KGs) such as ConceptNet, and
to model the relation between two concepts as a set of paths. However, KGs are
limited to a fixed set of relation types, and they are incomplete and often
noisy. Another strategy is to distill relation embeddings from a fine-tuned
language model. However, this is less suitable for words that are only
indirectly related and it does not readily allow us to incorporate structured
domain knowledge. In this paper, we aim to combine the best of both worlds. We
model relations as paths but associate their edges with relation embeddings.
The paths are obtained by first identifying suitable intermediate words and
then selecting those words for which informative relation embeddings can be
obtained. We empirically show that our proposed representations are useful for
solving hard analogy questions.111Source code to reproduce our experimental
results and the model checkpoints are available in the following repository:
https://github.com/niteshroyal/SolvingHardAnalogyQuestions.

##  1 Introduction

Many applications rely on Knowledge graphs (KGs) to model the relationship
between concepts. For instance, KGs have been used to characterise how an
answer candidate is related to the concepts that appear in a question Yasunaga
et al. (2021); Zhang et al. (2022b) and to help interpret visual scenes Gu et
al. (2019). In such applications, relations are modelled as KG paths, which
has two key advantages: (i) we can easily inject domain-specific knowledge and
(ii) the interpretable nature of KG paths offers a degree of transparency. But
KGs also have important drawbacks. They use a fixed set of relation types,
which may not be fine-grained enough. KGs are also inevitably incomplete and,
especially in the case of commonsense KGs such as ConceptNet Speer et al.
(2017), noisy.

There is also a tradition in Natural Language Processing (NLP) to model
relations as embeddings. For instance, DIRT Lin and Pantel (2001) and LRA
Turney (2005) are early examples of methods which used vectors to represent
the relation between two words. More recently, relation embeddings have been
obtained using fine-tuned language models. For instance, RelBERT Ushio et al.
(2021a) has achieved state-of-the-art results on several analogy datasets
using a fine-tuned RoBERTa-large model Liu et al. (2019). The strengths and
weaknesses of relation embeddings are complementary to those of KGs: relation
embeddings lack transparency and cannot easily incorporate external knowledge,
but they are capable of capturing subtle differences, making it possible to
encode relational knowledge in a way that is considerably richer than what can
be achieved with KGs. A final drawback of relation embeddings is that they are
best suited for concept pairs that have a clear and direct relationship. For
instance, they can encode the relation between _umbrella_ and _rain_ , but are
less suitable for encoding the relation between _umbrella_ and _cloudy_ (e.g.
if it is cloudy, there is a chance of rain, which means that we might take an
umbrella).

In this paper, we propose a hybrid approach which combines the advantages of
KGs with those of relation embeddings. The idea is conceptually
straightforward. We represent relations as paths, where nodes correspond to
concepts, but rather than associating edges with discrete types, we label them
with relation embeddings. We will refer to such paths as _relation embedding
chains_. Clearly, this approach allows us to model indirect relationships,
while keeping the flexibility of embeddings, as well as some of the
interpretability of KG paths.

We are still faced with the challenge of selecting suitable paths. KGs are
insufficient for this purpose, given their noisy and incomplete nature. Our
solution relies on the following idea: to decide whether $a\rightarrow
x\rightarrow b$ is a suitable path for modelling the relationship between $a$
and $b$, what matters is whether we have access to an informative relation
embedding for the word pairs $(a,x)$ and $(x,b)$. Motivated by this, we first
develop a classifier to predict whether a given RelBERT embedding is
informative or not. We then generate possible relational paths, using
ConceptNet as well as standard word embeddings, and filter these paths based
on the informativeness classifier.

While relation embedding chains are expressive, we sometimes need a simpler
representation, especially in unsupervised settings. We therefore also study
how the information captured by relation embedding chains can be summarised as
a single vector, without relying on task-specific supervision.

To evaluate the usefulness of relation embedding chains, we focus on word
analogy questions. Given a query word pair (e.g.  _word_ :_language_) and a
set of candidate word pairs, the task is to select the most analogous
candidate (e.g.  _note_ ,_music_). We show that relation embedding chains are
well-suited for answering hard analogy questions.

##  2 Related Work

Modelling analogies has been a long-standing topic of interest in NLP Turney
(2005, 2012). Recent years have seen a significant increase in interest in
this topic, fuelled by the success of large language models (LLMs). For
instance, Bhavya et al. (2022) used LLMs for generating explanations of
scientific concepts involving analogies, while Sultan and Shahaf (2022) used
LLMs for identifying analogies between procedural texts. However, most work
has focused on modelling analogies between word pairs Ushio et al. (2021b, a);
Chen et al. (2022); Czinczoll et al. (2022a); Li et al. (2023); Yuan et al.
(2023), which is also the setting we consider in this paper. We focus in
particular on RelBERT Ushio et al. (2021a), which is the state-of-the-art on
several benchmarks. RelBERT is a RoBERTa-large model that was fine-tuned on
the relational similarity dataset from SemEval 2012 Task 2 Jurgens et al.
(2012). Given a word pair, RelBERT computes a relation embedding by feeding
that word pair as input to the fine-tuned RoBERTa model and averaging the
output embeddings.

The use of KG paths for modelling relations between concepts has also been
extensively studied. For instance, Boteanu and Chernova (2015) proposed the
use of ConceptNet paths for explaining why two word pairs are analogous. Zhou
et al. (2019) highlighted the noisy nature of many ConceptNet paths. To
address this issue, they trained a system to predict path quality based on
crowdsourced judgments of naturalness. More recently, ConceptNet paths have
been used to provide external knowledge to NLP systems, for instance in
question answering systems Lin et al. (2019); Yasunaga et al. (2021); Zhang et
al. (2022b); Jiang et al. (2022); Sun et al. (2022). In such cases, a Graph
Neural Network is typically used to process the paths, and the resulting
representations are then integrated with those obtained by a language model.
An important distinction with our work is that we focus on unsupervised
settings. KG paths are especially helpful for question answering over KGs.
However, most approaches rely on matching chains of discrete relation types
Das et al. (2022); Zhang et al. (2022a). Chains of relation embeddings, as we
study in this paper, make it possible to model finer relationships.

##  3 Scoring RelBERT Embeddings

Given a word pair $(a,b)$, we write $\mathbf{r_{ab}}\in\mathbb{R}^{d}$ for the
corresponding RelBERT embedding. While RelBERT can provide a relation
embedding for any pair of words, not all these embeddings are equally
informative. In particular, we would like to distinguish embeddings
$\mathbf{r_{ab}}$ which encode a specific relationship from embeddings that
rather reflect the lack of any (known) relationship between $a$ and $b$.
Unfortunately, the relation embeddings of different unrelated word pairs are
typically not similar.

To address this, we want to train a classifier to predict whether a given
relation embedding $\mathbf{r_{ab}}$ is informative or not. However, this
requires access to a set of related word pairs Pos and a set of unrelated word
pairs Neg. The set Neg can simply be constructed by choosing random word
pairs. While we may occasionally end up with word pairs that are actually
related in some way, the probability that this is the case for randomly chosen
words is sufficiently low for this not to be problematic. However,
constructing the set Pos is less straightforward, since we are not looking for
words that have a particular relation, but rather for words that have _any_
kind of (clear) relationship. If the examples are not sufficiently diverse,
then our classifier will simply detect whether $\mathbf{r_{ab}}$ corresponds
to one of the relations that were considered during training. One of the few
relevant large-scale resources we have at our disposal is ConceptNet. However,
during initial experiments, we quickly found ConceptNet to be too noisy for
this purpose. We therefore instead relied on
GPT-4222https://openai.com/research/gpt-4 to generate a diverse set of around
11,000 positive examples. The prompt we used to obtain a sufficiently diverse
set of examples is shown in the Appendix D.

To create a set of negative examples, of the same size, we corrupted the
positive examples. We then trained a logistic regression classifier on the
resulting training set. Throughout this paper, we will write
$\textit{inf}(\mathbf{r_{ab}})$ for the probability that $\mathbf{r_{ab}}$
expresses an informative relationship, according to this classifier. We will
refer to this value as the informativeness of the embedding $\mathbf{r_{ab}}$.

##  4 Connecting Concepts

The relationship between two concepts $a$ and $b$ is sometimes easiest to
explain using an intermediate concept $x$. For instance, _umbrella_ is related
to _cloudy_ because (i) an _umbrella_ protects against _rain_ and (ii) _rain_
is only possible if it is _cloudy_. A natural strategy for identifying such
intermediate concepts is to find paths of length two connecting the target
words $a$ and $b$ in ConceptNet. However, the coverage of ConceptNet is
limited, and many relevant intermediate concepts might not be found in this
way. One possible solution would be to consider a sequence of intermediate
concepts, and model the relation between $a$ and $b$ in terms of a path
$a\rightarrow x_{1}\rightarrow...\rightarrow x_{n}\rightarrow b$. However,
longer ConceptNet paths are often too noisy to be useful. Moreover, in
practice a single intermediate concept is usually sufficient, so rather than
considering longer paths, we propose two strategies to find additional links
between concepts based on word embeddings. They are illustrated in Figure 1.

#### Missing Link Prediction

We add missing links based on the informativeness classifier from Section 3.
Specifically, we first use a standard word embedding to find the top-$k$ most
similar words to $a$, with e.g. $k=500$. For each of these words $y$, we add a
link between $a$ and $y$ if $\textit{inf}(\mathbf{r_{ay}})>0.75$.333Since we
focused on unsupervised settings in this paper, we cannot tune this value.
Compared to a threshold of 0.5, the choice of 0.75 reflects the idea that we
want to be selective: if in doubt it seems better not to add the link. We add
missing links from $b$ in the same way. We will refer to this strategy as
_missing link prediction_. Note that this strategy not only finds synonyms or
morphological variations of words but also often finds words that are related
in different ways. For instance, some links that are not present in ConceptNet
and have been added using this approach include: (dog, owners), (cashier,
grocery store), (helium, noble gases), (drug trafficking, illegal), and
(disinfectant, sterilization). Such links clearly enrich ConceptNet in a non-
trivial way.

#### Semantic Smoothing

For the second strategy, we only consider the top-5 neighbours, noting that
these are often either synonyms or morphological variations of the same word
(e.g.  _cloud_ and _cloudy_). Specifically, rather than only considering $x$
as an intermediate concept if $x$ is connected to both $a$ and $b$, we now
consider $x$ as an intermediate concept as soon as it is connected to one of
the 5 nearest neighbours of $a$ (or $a$ itself) and one of the 5 nearest
neighbours of $b$ (or $b$ itself). We will refer to this second strategy as
_semantic smoothing_.

Figure 1: For the concept pair (_blandishment_ , _coax_), the integration of
missing link prediction and semantic smoothing strategies identifies
_seduction_ , _inducement_ , _enticement_ , _persuade_ , _flattery_ ,
_persuasion_ , and _wooing_ as intermediate concepts. The concepts _entice_
and _cajole_ used for smoothing are not considered intermediate concepts; they
merely help in inducing additional intermediate concepts. If only ConceptNet
were used to determine intermediate concepts, flattery and persuade would be
the only ones identified.

##  5 Condensing Relation Embedding Chains

The strategy from Section 4 allows us to identify intermediate concepts
$x_{1},...,x_{n}$ that can help to describe the relation between two given
words $a$ and $b$. Each intermediate concept $x_{i}$ corresponds to a chain
$\mathbf{r_{ax_{i}}};\mathbf{r_{x_{i}b}}$ of two relation embeddings. These
_relation embedding chains_ can encode relationships in a fine-grained way,
but in practice we often need a more compact representation. We therefore
train a simple model to summarise a set of relation embedding chains
$\\{\mathbf{r_{ax_{1}}};\mathbf{r_{x_{1}b}},...,\mathbf{r_{ax_{n}}};\mathbf{r_{x_{n}b}}\\}$
for a given concept pair $(a,b)$ as a single vector. Our model has the
following form:

$\displaystyle\mathbf{s_{ab}}=\psi\Big{(}f\big{(}\phi(\mathbf{r_{ax_{1}}},\mathbf{r_{x_{1}b}}),...,\phi(\mathbf{r_{ax_{n}}},\mathbf{r_{x_{n}b}})\big{)}\Big{)}$

The function
$\phi:\mathbb{R}^{d}\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{m}$ is
intuitively doing a kind of relational composition: it aims to predict a
relation embedding for the pair $(a,b)$ from the relation embeddings
$\mathbf{r_{ax_{i}}}$ and $\mathbf{r_{x_{i}b}}$. These predicted embeddings
are then combined using a pooling function
$f:\mathbb{R}^{m}\times...\times\mathbb{R}^{m}\rightarrow\mathbb{R}^{m}$.
Finally, the resulting vector is mapped back to a $d$-dimensional embedding
using the decoder $\psi:\mathbb{R}^{m}\rightarrow\mathbb{R}^{d}$.

For the decoder $\psi$, we simply use a linear layer. For the pooling function
$f$, we experimented with sum-pooling and max-pooling. We found the result to
be largely insensitive to this choice. Throughout this paper, we will
therefore fix $f$ as summation. Finally, the composition function $\phi$ is
implemented as follows:

$\displaystyle\phi(\mathbf{r_{ax}},\mathbf{r_{xb}})=\mathsf{GeLU}(\mathbf{A}(\mathbf{r_{ax}}\oplus\mathbf{r_{xb}})+\mathbf{b})$
(1)

where we write $\oplus$ for vector concatenation, $\mathbf{A}$ is a matrix,
and $\mathbf{b}$ is a bias vector.

#### Training

Our focus is on solving analogy questions, which is an unsupervised problem.
In the absence of task-specific training data, we train the model to predict
the RelBERT embedding $\mathbf{r_{ab}}$, using the following loss:

$\displaystyle\mathcal{L}=-\sum_{(a,b)}\cos(\mathsf{s_{ab}},\mathsf{r_{ab}})$
(2)

where $\mathsf{s_{ab}}$ is the vector predicted from the relation embedding
chains, and $\mathsf{r_{ab}}$ is the RelBERT embedding. The sum in (2) ranges
over concept pairs $(a,b)$ from ConceptNet for which the informativeness
$\textit{inf}(\mathbf{r_{ab}})$ is sufficiently high. This is important to
ensure that the model is not trained on a noisy supervision signal.
Specifically, we only considered concept pairs $(a,b)$ for which
$\textit{inf}(\mathbf{r_{ab}})>0.75$.

Note that while we train the model on concept pairs $(a,b)$ which have an
informative RelBERT embedding, our aim is to use this model for pairs for
which this is not the case. We thus rely on the assumption that a composition
model which is trained on word pairs with informative RelBERT vectors will
generalise in a meaningful way to word pairs for which this is not the case.

##  6 Solving Analogy Questions

Analogy questions involve a query pair $(a,b)$ and a number of candidate
answers $(x_{1},y_{1}),...,(x_{k},y_{k})$. The aim is to select the candidate
$(x_{i},y_{i})$ whose relationship is most similar to that of the query pair.
When using RelBERT, we simply choose the pair $(x_{i},y_{i})$ for which
$\cos(\mathbf{r_{ab}},\mathbf{r_{x_{i}y_{i}}})$ is maximal.

#### Identifying Hard Analogy Questions

Our key hypothesis is that the informativeness of the RelBERT vectors, as
predicted by the classifier from Section 3, can tell us whether using RelBERT
vectors is a reliable strategy for a particular analogy question. The lower
the informativeness of $\mathbf{r_{ab}}$ or $\mathbf{r_{x_{i}y_{i}}}$, with
$(a,b)$ the query and $(x_{i},y_{i})$ the chosen answer candidate, the less
confident we can be about the chosen answer. We define our confidence in
RelBERT’s prediction as follows:

$\displaystyle\textit{conf}(a,b,x,y)=\textit{min}(\textit{inf}(\mathbf{r_{ab}}),\textit{inf}(\mathbf{r_{xy}}))$
(3)

with $(a,b)$ the query pair and $(x,y)$ the candidate selected by RelBERT. In
cases where the confidence is low, we aim to improve the prediction by taking
advantage of relation embedding chains.

#### Condensed Relation Chain Comparison

We can straightforwardly solve analogy questions using condensed relation
chains. In particular, we use the model from Section 5 to obtain relation
embeddings $\mathbf{s_{xy}}$ for the query and the candidate pairs. We then
proceed in the same way as with RelBERT, choosing the answer candidate
$(x_{i},y_{i})$ for which $\cos(\mathbf{s_{ab}},\mathbf{s_{x_{i}y_{i}}})$ is
maximal.

#### Direct Relation Chain Comparison

We can also solve analogy questions by using relation chains directly. Let
$c_{1},...,c_{u}$ be the intermediate concepts for the query pair $(a,b)$ and
let $z_{1},...,z_{v}$ be the intermediate concepts for an answer candidate
$(x,y)$. Then we evaluate the compatibility $\textit{comp}(a,b,x,y)$ of this
candidate as follows:

$\displaystyle\sum_{i=1}^{u}\max_{j=1}^{v}\textit{sim}(\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}},\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}})$

The idea is that when $(a,b)$ and $(x,y)$ are analogous, for most relation
chains $\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}}$ connecting $a$ and $b$ there
should exist a similar relation chain
$\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}}$ connecting $x$ and $y$. The
similarity between relation chains can be evaluated as follows:

$\displaystyle\textit{sim}_{1}(\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}},\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}})$
(4)
$\displaystyle\quad=\min(\cos(\mathbf{r_{ac_{i}}},\mathbf{r_{xz_{j}}}),\cos(\mathbf{r_{c_{i}b}},\mathbf{r_{z_{j}y}}))$

In other words, two relation chains are similar if their first component is
similar and their second component is also similar. Our default configuration
uses this approach. However, this may be too strict. For instance, suppose
that $\mathbf{r_{c_{i}b}}$ and $\mathbf{r_{z_{j}y}}$ capture the _located-at_
relation, while $\mathbf{r_{ac_{i}}}$ captures _part-of_ and
$\mathbf{r_{xz_{j}}}$ captures _is-a_. Then the relation chains
$\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}}$ and
$\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}}$ both essentially encode that $a$ is
located at $c$, but they would not be considered to be similar according to
(4). To allow such relation chains to be identified as being similar, we will
also consider the following alternative:

$\displaystyle\textit{sim}_{2}(\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}},\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}})$
(5)
$\displaystyle\quad=\cos(\psi(\phi(\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}})),\psi(\phi(\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}})))$

with $\phi$ and $\psi$ the composition function and decoder of the model for
condensing relation chains. Finally, we will also consider a baseline strategy
which ignores the order of the relation embeddings:

$\displaystyle\textit{sim}_{3}(\mathbf{r_{ac_{i}}};\mathbf{r_{c_{i}b}},\mathbf{r_{xz_{j}}};\mathbf{r_{z_{j}y}})$
(6)
$\displaystyle\quad=\cos(\mathbf{r_{ac_{i}}}+\mathbf{r_{c_{i}b}},\mathbf{r_{xz_{j}}}+\mathbf{r_{z_{j}y}})$

##  7 Experiments

We empirically analyse our proposed strategies for solving hard analogy
questions. We are specifically interested in the following research questions.
(i) How suitable is the confidence degree (3) as a proxy for estimating the
difficulty of an analogy question? (ii) How suitable are relation embedding
chains for answering difficult analogy questions? (iii) What are the best ways
for learning and using these relation embedding chains?

###  7.1 Experimental Setup

#### Datasets

We evaluate our models on a number of different analogy question datasets.
First, we use the 5 datasets that were used by Ushio et al. (2021b): the SAT
dataset proposed by Turney et al. (2003); the U2 and U4 datasets which were
obtained from an educational website; and reformulations of the Google Mikolov
et al. (2013) and BATS Gladkova et al. (2016) datasets into the multiple-
choice analogy question format. We obtained these datasets from the RelBERT
repository444https://huggingface.co/datasets/relbert/analogy_questions. We
also use a reformulation of SCAN Czinczoll et al. (2022b) into the multiple-
choice analogy question format, which is available from the same repository.
Finally, we include the English version of E-KAR555Available at https://ekar-
leaderboard.github.io Chen et al. (2022). E-KAR contains questions questions
involving word pairs and questions involving word triples. We only considered
the former for our experiments. We use accuracy as the evaluation metric.

There are some important differences between these datasets. For instance,
Google and BATS focus on morphological, encyclopedic, and in the case of BATS,
lexical relations. SCAN focuses on scientific and creative analogies,
requiring models to link relationships from one domain (e.g. the solar system)
to relationships to a completely different domain (e.g. atoms). The other
datasets focus on abstract conceptual relations, but they cover a range of
difficulty levels (for humans): SAT is obtained from college entrance tests;
U2 is aimed at children from primary and secondary school; U4 has difficult
levels ranging from college entrance tests to graduate school admission tests;
E-KAR was derived from Chinese civil service exams.

| SAT | U2 | U4 | BATS
---|---|---|---|---
Conf |  RelB |  Cond |  Dir |  GPT4 |  RelB |  Cond |  Dir |  GPT4 |  RelB |  Cond |  Dir |  GPT4 |  RelB |  Cond |  Dir |  GPT4
$[0.0,0.25)$ |  40.7 |  51.9 |  37.0 |  59.3 |  17.6 |  35.3 |  52.9 |  76.5 |  26.9 |  46.2 |  46.2 |  38.5 |  32.1 |  46.7 |  55.5 |  90.5
$[0.25,0.5)$ |  51.4 |  48.6 |  60.0 |  54.3 |  38.1 |  33.3 |  47.6 |  42.9 |  43.8 |  43.8 |  50.0 |  62.5 |  66.3 |  72.9 |  68.7 |  94.0
$[0.5,0.75)$ |  78.0 |  71.0 |  67.0 |  65.0 |  65.9 |  70.5 |  61.4 |  70.5 |  59.3 |  52.3 |  48.8 |  61.6 |  68.0 |  69.5 |  67.0 |  87.3
$[0.75,1.0]$ |  80.6 |  77.7 |  66.9 |  78.3 |  78.1 |  67.8 |  59.6 |  76.0 |  71.0 |  68.0 |  57.7 |  75.0 |  89.9 |  84.8 |  80.1 |  91.0
| Google | SCAN | E-KAR | Avg
Conf |  RelB |  Cond |  Dir |  GPT4 |  RelB |  Cond |  Dir |  GPT4 |  RelB |  Cond |  Dir |  GPT4 |  RelB |  Cond |  Dir |  GPT4
$[0.0,0.25)$ |  42.0 |  52.0 |  66.0 |  98.0 |  23.2 |  22.9 |  18.9 |  23.5 |  35.0 |  40.0 |  35.0 |  30.0 |  31.1 |  42.1 |  44.5 |  59.5
$[0.25,0.5)$ |  50.0 |  50.0 |  57.1 |  100.0 |  26.7 |  28.3 |  27.4 |  20.4 |  42.3 |  34.6 |  19.2 |  50.0 |  45.5 |  44.5 |  47.2 |  60.6
$[0.5,0.75)$ |  54.1 |  51.4 |  64.9 |  100.0 |  32.2 |  32.5 |  30.9 |  22.1 |  53.5 |  55.8 |  55.8 |  48.8 |  58.7 |  57.6 |  56.5 |  65.0
$[0.75,1.0]$ |  93.0 |  88.1 |  84.9 |  98.9 |  33.6 |  35.5 |  35.0 |  31.4 |  52.3 |  50.8 |  41.5 |  60.0 |  71.2 |  67.5 |  60.8 |  72.9
Table 1: Results on analogy questions of different difficulty levels, across
all datasets (accuracy). The best performing model per dataset and difficulty
level is shown in bold.

#### Methods

The RelBERT repository on Huggingface contains a number of different RelBERT
variants. We have used the model666Available from
https://huggingface.co/relbert/relbert-roberta-large-nce-semeval2012-0-400
trained using Noise Contrastive Estimation on SemEval 2012 Task 2 Jurgens et
al. (2012), as this variant outperforms the original variant from Ushio et al.
(2021a).

For our implementation of _missing link prediction_ we combined the top-250
neighbours (in terms of cosine similarity) according to the ConceptNet
Numberbatch word embedding Speer et al. (2017) with the top-250 neighbours
according to GloVe Pennington et al. (2014). For _semantic smoothing_ we used
to top-5 neighbours from GloVe.

In terms of baselines, our primary emphasis is on comparing the proposed
methods with RelBERT, which is the state-of-the-art relation embedding
model777While Yuan et al. (2023) have reported higher results on some datasets
using another fine-tuned RoBERTa model, their method does not learn relation
embeddings but solves the task as a multiple-choice question answering
problem. Moreover, their model was fine-tuned on the validation split of each
benchmark, which makes the results incomparable.. To provide additional
context, we have also obtained results with GPT-4.888Details of our prompt can
be found in the appendix. Following earlier work on using LLMs for solving
analogy questions Yuan et al. (2023), we ask the model to generate
explanations Wei et al. (2022) and include a few solved questions as part of
the prompt. The results of GPT-4 should be interpreted with some caveats,
however. For instance, it is well-known that the results of LLMs can be highly
sensitive to the choice of the prompt, so it is likely that better results are
possible. Moreover, it is possible that GPT-4 has seen some of the datasets
during training, which could lead to inflated results. Besides GPT-4, we also
include the results that were obtained by Yuan et al. (2023) using ChatGPT and
InstructGPT003 with chain-of-thought prompting.

|  SAT |  U2 |  U4 |  BA |  GO |  SC |  EK
---|---|---|---|---|---|---|---
GPT-4 |  70.3 |  71.9 |  68.8 |  90.8 |  99.0 |  23.5 |  51.3
ChatGPT† |  65.8 |  68.9 |  70.8 |  90.6 |  97.8 |  - |  -
InstructGPT${}_{003}^{\dagger}$ |  48.9 |  59.0 |  57.9 |  82.8 |  96.7 |  - |  -
RelBERT |  73.6 |  67.5 |  63.0 |  80.9 |  81.4 |  27.3 |  48.7
Condensed |  70.6 |  62.7 |  60.9 |  79.2 |  78.6 |  27.9 |  48.1
Direct |  63.8 |  58.3 |  54.4 |  75.7 |  79.2 |  25.7 |  40.9
Cond<0.25 |  74.5 |  68.9 |  64.1 |  82.0 |  82.4 |  27.2 |  49.4
Cond<0.5 |  74.2 |  68.4 |  64.1 |  82.7 |  82.4 |  27.6 |  48.1
Direct<0.25 |  73.3 |  70.2 |  64.1 |  82.7 |  83.8 |  25.6 |  48.7
Direct<0.5 |  74.2 |  71.1 |  64.8 |  82.9 |  84.4 |  25.7 |  44.8
Table 2: Overall results of the different models (accuracy). Results with
$\dagger$ were taken from Yuan et al. (2023).

###  7.2 Results

Table 1 shows our main results, focusing on four methods: RelBERT (RelB),
condensed relation embedding chain comparison (_Cond_), direct relation chain
comparison (_Dir_) and GPT-4. The results are broken down based on our
confidence in RelBERT’s prediction, as computed by (3). The overall results
per dataset are summarised in Table 2. In this table, we also report the LLM
results obtained by Yuan et al. (2023). We furthermore consider four hybrid
methods. The idea of these methods is to rely on RelBERT for the easy
questions and on either Cond or _Direct_ for the hard questions. For instance,
$\textit{Cond}_{<0.5}$ uses Cond for questions with difficulty levels below
0.5 (as estimated using (3)) and RelBERT for the others. A number of
conclusions can be drawn.

#### Confidence scores faithfully predict question difficulty

We can see that the performance of RelBERT is closely aligned with the
predicted difficulty level. On average, across all datasets, the accuracy
ranges from 31.1% for the hardest questions to 71.2% for the easiest
questions. This pattern can be observed for all datasets. Moreover, the
predicted difficulty level is also aligned with the performance of the other
models. For instance, GPT-4 on average also performs best on the questions
with high confidence values, especially for SAT, U4, E-KAR and SCAN. For
Google and BATS, GPT-4 performs well throughout. This suggests that the
confidence score (3) is successful in predicting intrinsic question
difficulty, rather than merely predicting where RelBERT is likely to fail.

#### Relation embedding chains are helpful for hard questions

Focusing on the performance for the hardest questions, with confidence levels
in [0.0,0.25), we can see that _Condensed_ outperforms RelBERT on all
datasets, with the exception of SCAN (where the results are close). _Direct_
outperforms in most datasets as well, but not in SCAN and SAT. For the
questions with a difficult range in [0.25,0.50), _Direct_ still outperforms
RelBERT in most cases, with E-KAR now the only exception. For the questions
with confidence level in the range [0.50,0.75) the performance of RelBERT is
generally quite similar to that of _Condensed_ and _Direct_. Finally, for the
easiest questions, RelBERT is clearly better, with the exception of SCAN. In
accordance with these observations, in Table 2 we can see that the hybrid
models generally outperform RelBERT, except for SCAN and E-KAR where their
performance is similar. For SAT, three of the hybrid models outperform the
state-of-the-art result of RelBERT. Finally, comparing _Condensed_ and
_Direct_ there is no clear winner, with the former performing better for the
easiest questions and the latter performing better for the hardest ones.

#### GPT-4 performs best overall

It is also notable how similar the GPT-4 results are to the ChatGPT results
obtained by Yuan et al. (2023). However, GPT-4 and ChatGPT do not provide
relation embeddings and can thus not replace models such as RelBERT in many
applications (e.g. for retrieval tasks). In Table 1, we can also see that
_Condensed_ and _Direct_ can sometimes outperform GPT-4 on the hardest
questions, namely for U4 and E-KAR. Interestingly, these are also the
benchmarks with the highest intended difficulty level for humans. Moreover,
for SCAN, which requires making suitable abstractions, GPT-4 generally
underperforms.

|  SAT |  U2 |  U4 |  BA |  GO |  SC |  EK
---|---|---|---|---|---|---|---
Direct |  63.8 |  58.3 |  54.4 |  75.7 |  79.2 |  25.7 |  40.9
$\textit{sim}_{2}$ |  67.4 |  60.5 |  56.9 |  80.3 |  79.8 |  29.1 |  42.9
$\textit{sim}_{3}$ |  59.3 |  54.8 |  53.0 |  66.9 |  70.8 |  24.4 |  42.2
CN + ss |  57.6 |  58.3 |  55.1 |  67.9 |  66.4 |  22.3 |  39.0
CN + mlp |  58.8 |  64.5 |  59.3 |  77.4 |  78.0 |  22.8 |  44.8
CN only |  56.7 |  57.5 |  57.4 |  71.2 |  62.0 |  21.4 |  41.6
mlp only |  60.8 |  57.0 |  54.9 |  74.9 |  75.4 |  22.9 |  42.2
CN types |  40.4 |  42.1 |  40.5 |  56.3 |  50.4 |  15.3 |  37.7
Table 3: Analysis of direct relation chain comparison.  |  SAT |  U2 |  U4 |  BA |  GO |  SC |  EK
---|---|---|---|---|---|---|---
Condensed |  70.6 |  62.7 |  60.9 |  79.2 |  78.6 |  27.9 |  48.1
CN + ss |  69.4 |  63.2 |  60.4 |  76.9 |  74.6 |  27.3 |  48.1
CN + mlp |  67.1 |  64.5 |  60.9 |  79.7 |  81.4 |  25.4 |  48.1
CN only |  68.8 |  59.2 |  60.4 |  74.0 |  71.2 |  24.4 |  48.1
mlp only |  65.9 |  67.1 |  59.3 |  78.0 |  80.2 |  25.1 |  48.1
Table 4: Analysis of condensed relation chain comparison.

###  7.3 Analysis

We now provide further analysis about the direct and condensed relation chain
comparison methods. We also include a qualitative analysis to better
understand the nature of relation embedding chains.

#### Direct Relation Chain Comparison

Table 3 compares our default configuration (Direct), which uses
$\textit{sim}_{1}$ to measure the similarity between relation embedding
chains, with a number of variants. First, the rows labelled $\textit{sim}_{2}$
and $\textit{sim}_{3}$ show the impact of replacing (4) by, respectively, (5)
and (6) for measuring the similarity. The model based on $\textit{sim}_{2}$
combines elements of the direct and condensed relation chain comparisons.
Accordingly, we can see that its performance is typically in between that of
these two methods. However, for BATS, Google and SCAN it actually outperforms
both. The comparatively poor results for $\textit{sim}_{3}$ show that the
order of the relation embeddings matters.

Next, the table compares a number of variations in how the relation embedding
chains themselves are constructed. As we discussed in Section 4, our main
method combines ConceptNet with two augmentation strategies: semantic
smoothing (_ss_) and missing link prediction (_mlp_). The rows labelled _CN +
ss_ and _CN + mlp_ show the impact of only using one of these augmentation
strategies. Note that our default configuration for Direct is _CN + mlp + ss_.
Furthermore, the row labelled _CN only_ shows the results when we only use
ConceptNet paths, without applying either of the two augmentation strategies.
Finally, the row labelled _mlp only_ shows the results of a variant which only
relies on the links predicted by the missing link prediction strategy, without
using ConceptNet at all. As can be seen, the links from ConceptNet and those
predicted using _mlp_ lead to the best performance. In fact, the _CN + mlp_
variant achieves the best results in several cases. Interestingly, _mlp_ on
its own already performs quite well, which shows that relation chains can be
constructed even without the help of an external KG.

Finally, we also evaluate a variant which does not rely on relation embeddings
(_CN types_). In this case, we only use ConceptNet paths for finding
intermediate concepts. We then compute the compatibility
$\textit{comp}(a,b,x,y)$ as follows:

$\displaystyle\sum_{i=1}^{u}\max_{j=1}^{v}\mathbbm{1}[r_{ac_{i}}=r_{xz_{j}}\wedge
r_{c_{i}b}=r_{z_{j}y}]$

Here $r_{xy}$ is the ConceptNet relation type of the edge that connects $x$
and $y$, and $\mathbbm{1}[\alpha]$ is 1 if the condition $\alpha$ is satisfied
and 0 otherwise. As can be seen, this variant performs poorly. The contrast
between this method and _CN only_ reflects the impact of (i) the noisy nature
of ConceptNet and (ii) the fact that the fixed relation types are often less
informative than relation embeddings.

| Query | RelBERT | Correct | Interm. query | Interm. answer
---|---|---|---|---|---
SAT | reprehensible : condemn | depraved : admire | estimable : praise | condemnable | praiseworthy
U2 | blandishment : coax | surplus : squander | eulogy : praise | seduction | homage
U4 | vernacular : regional | budget : austere | fluctuation : irregular | national | normal
SCAN | processing : bug | memorize : mistake | thinking : mistake | computing | working
BATS | package : parcel | hieroglyph : cloth | flower : blossom | courier | grower
Table 5: Examples of analogy questions which are incorrectly answered by RelBERT, while being correctly answered by _Direct_. For each question, we also show the intermediate concept $c$ for the query pair $(a,b)$ and the intermediate concept $z$ for the answer pair $(x,y)$ for which $\textit{sim}_{1}(\mathbf{r_{ac}};\mathbf{r_{cb}},\mathbf{r_{xz}};\mathbf{r_{zy}})$ is maximal. | Dataset | Query | Candidates
---|---|---|---
Easy | SAT | abbreviation:word |  abridge:book, laminate: layer, inhibit:idea, expedite:mail, invoke:deity
U2 | sewing:craft |  gasoline:fuel, salt:food, fate:science, sunrise:art
U4 | pragmatic:practical |  trivial:negligible, irritating:pleasing, tenacious:faltering, opaque:translucent
BATS | market:marketplace |  sofa:couch, murder:clothes, lazy:help, shirt:button
Google | hand:hands |  horse:horses, swimming:swam, dollars:goats, dollar:mouse
SCAN | earth:air |  sun:space, planet:orbit, planet:sun, planet:elliptical, planet:space, planet:gravity, …
E-KAR | bird:wings |  fish:fin, sheep:dog, locust:cicada, cattle:grass
Hard | SAT | shallow:depth |  apathetic:caring, salty:ocean, cloudy:height, lurid:shock, pious:faith
U2 | aloof:connected |  deliberate:accidental, rigid:firm, ethereal:fleeting, logical:calculating
U4 | fastidious:particular |  fanatical:enthusiastic, edible:delicious, adamant:opposed, manipulative:masterful
BATS | shirt:button |  tonne:kilogram, pie:tripod, fridge:appliance, sonata:door
Google | Yerevan:Armenia |  Zagreb:Croatia, Azerbaijan:Denmark, Dushanbe:Tunis, boy:girl
SCAN | money:effective |  time:efficient, schedule:quick, time:schedule, time:quick, time:slow, …
E-KAR | shopping:bank card |  cooking:natural gas, driving:steering wheel, running:sneakers, travel:navigator
Table 6: Top: examples of analogy questions with a confidence in $[0.75,1.0]$, i.e. questions predicted to be easy. Bottom: examples of analogy questions with a confidence in $[0,0.25)$, i.e. questions predicted to be hard. In each case, the correct answer is shown in bold.  Pair | Score
---|---
horse : pony | 0.99
intermarriage : intramarriage | 0.99
foresight : hindsight | 0.96
addressable : unaddressable | 0.95
balloon : popped | 0.52
call : text | 0.50
at one time : individually | 0.49
fate : choice | 0.47
farmer : slicker | 0.10
moment : ages | 0.09
time : fine | 0.01
chad : chat | 0.01
Table 7: Examples of the predicted informativeness scores for concept pairs
from ConceptNet.

#### Condensed Relation Chain Comparison

Table 4 compares our default approach based on condensed relation embedding
chains (_Condensed_) with some variants. Note that our default configuration
for Condensed is _CN + mlp + ss_. Specifically, as in Table 3, we analyse the
impact of the semantic smoothing (_ss_) and missing link prediction (_mlp_)
strategies. We can broadly observe the same patterns; e.g. we again find that
_CN + mlp_ achieves the best results in several cases and that it is possible
to obtain competitive results without using a KG.

#### Qualitative Analysis

Table 7 shows some examples of informativeness scores for word pairs from
ConceptNet. As these examples illustrate, concept pairs with high scores
consistently have a clear relationship. Those with the lowest scores either do
not have any obvious relationship, or they have a relationship which is not
captured by their RelBERT embedding. As an example of this latter case, the
table contains pairs that are linked by the “sounds like” (_chad : chat_) and
“rhymes with” relation (_time : fine_).

Table 5 shows examples where RelBERT made an incorrect prediction while
_Direct_ picked the right answer. In each case, we also show the most
influential intermediate concept for the query and answer pairs. Specifically,
we find the intermediate concept $c$ for the query $(a,b)$ and the
intermediate concept $z$ for the answer $(x,y)$ for which
$\textit{sim}_{1}(\mathbf{r_{ac}};\mathbf{r_{cb}},\mathbf{r_{xz}};\mathbf{r_{zy}})$
is maximal. These examples illustrate some of the different roles that
intermediate concepts can play. For instance, in the example from SAT, the
intermediate worth _condemnable_ makes it possible to characterise the pair
_reprehensible:condemn_ in terms of a near-synonym
(_reprehensible:condemnable_) and a kind of morphological variation
(_condemnable:condemn_). The example from U4 illustrates a case where the
close similarity between two terms (_vernacular:regional_) is more easily
recognised by RelBERT as a composition of two antonyms (_vernacular:national_
and _national:regional_). The example from SCAN illustrates a case where the
word pairs involved are only indirectly related.

The top half of Table 6 shows examples of questions that were predicted to be
easy. As can be seen, the word pairs involved have a clear and direct
relationship. The bottom half of the table similarly shows examples of
questions that were predicted to be hard. These examples are hard for
different reasons. Some examples are challenging because they involve a clear
but indirect relationship (e.g.  _shopping:bank card_). Others involve rather
abstract relations (e.g.  _aloof:connected_). The example from Google reflects
the fact that RelBERT was not trained on named entities.

##  8 Conclusions

Relations between concepts are typically modelled either as KG paths or as
relation embeddings. In this paper, we have proposed a hybrid approach, which
uses paths whose edges correspond the relation embeddings. This is useful to
model relationships between concepts that are only indirectly related, and
more generally for concept pairs whose relation embedding is unreliable. Our
approach crucially relies on a classifier which can predict the
informativeness of a RelBERT embedding. This classifier is used (i) to
identify analogy questions where we should not rely on RelBERT embeddings;
(ii) to select reliable RelBERT vectors for training a model for condensing
relation embedding chains; and (iii) to identify informative paths beyond
those in ConceptNet. We have relied on GPT-4 to generate training data for the
informativeness classifier, which allowed us to address the lack of suitable
existing datasets.

## Limitations

Our evaluation has focused on solving analogy questions, which are useful
because they allow for a direct evaluation of different approaches for
modelling relationships. While analogies play an important role in certain
downstream tasks, we essentially regard this as in intrinsic evaluation task.
It thus remains a matter for future work to analyse the usefulness of relation
embedding chains in downstream tasks (e.g. retrieving relevant cases for case-
based reasoning methods). From a practical point of view, using relation
embedding chains is more involved than directly using RelBERT embeddings. To
address this, the augmentation strategies can be applied offline, by creating
a graph of related concepts. While the corresponding RelBERT embeddings can
also be computed offline, storing millions of relation embeddings takes up a
large amount of space. In certain applications, it may thus be preferable to
compute the required RelBERT embeddings only when they are needed.

#### Acknowledgments

This work was supported by EPSRC grants EP/V025961/1 and EP/W003309/1.

## References

  * Bhavya et al. (2022) Bhavya Bhavya, Jinjun Xiong, and Chengxiang Zhai. 2022.  Analogy generation by prompting large language models: A case study of instructgpt.  _arXiv preprint arXiv:2210.04186_.
  * Boteanu and Chernova (2015) Adrian Boteanu and Sonia Chernova. 2015.  Solving and explaining analogy questions using semantic networks.  In _Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA_ , pages 1460–1466. AAAI Press.
  * Chen et al. (2022) Jiangjie Chen, Rui Xu, Ziquan Fu, Wei Shi, Zhongqiao Li, Xinbo Zhang, Changzhi Sun, Lei Li, Yanghua Xiao, and Hao Zhou. 2022.  E-KAR: A benchmark for rationalizing natural language analogical reasoning.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3941–3955, Dublin, Ireland. Association for Computational Linguistics.
  * Czinczoll et al. (2022a) Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022a.  Scientific and creative analogies in pretrained language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2094–2100, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Czinczoll et al. (2022b) Tamara Czinczoll, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2022b.  Scientific and creative analogies in pretrained language models.  _CoRR_ , abs/2211.15268.
  * Das et al. (2022) Rajarshi Das, Ameya Godbole, Ankita Naik, Elliot Tower, Manzil Zaheer, Hannaneh Hajishirzi, Robin Jia, and Andrew McCallum. 2022.  Knowledge base question answering by case-based reasoning over subgraphs.  In _International conference on machine learning_ , pages 4777–4793. PMLR.
  * Gladkova et al. (2016) Anna Gladkova, Aleksandr Drozd, and Satoshi Matsuoka. 2016.  Analogy-based detection of morphological and semantic relations with word embeddings: what works and what doesn’t. In _Proceedings of the NAACL Student Research Workshop_ , pages 8–15, San Diego, California. Association for Computational Linguistics.
  * Gu et al. (2019) Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. 2019\.  Scene graph generation with external knowledge and image reconstruction.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 1969–1978. Computer Vision Foundation / IEEE.
  * Jiang et al. (2022) Jinhao Jiang, Kun Zhou, Ji-Rong Wen, and Xin Zhao. 2022.  $great~{}truths~{}are~{}always~{}simple:$ a rather simple knowledge encoder for enhancing the commonsense reasoning capacity of pre-trained models.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1730–1741, Seattle, United States. Association for Computational Linguistics.
  * Jurgens et al. (2012) David Jurgens, Saif Mohammad, Peter Turney, and Keith Holyoak. 2012.  SemEval-2012 task 2: Measuring degrees of relational similarity.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 356–364, Montréal, Canada. Association for Computational Linguistics.
  * LearningExpress (2002) LearningExpress. 2002.  _501 Word Analogies Questions and Answers_ , 1 edition.  Delmar Cengage Learning.
  * Li et al. (2023) Shuyi Li, Shaojuan Wu, Xiaowang Zhang, and Zhiyong Feng. 2023.  An analogical reasoning method based on multi-task learning with relational clustering.  In _Companion Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023_ , pages 144–147. ACM.
  * Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019.  KagNet: Knowledge-aware graph networks for commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2829–2839, Hong Kong, China. Association for Computational Linguistics.
  * Lin and Pantel (2001) Dekang Lin and Patrick Pantel. 2001.  Discovery of inference rules for question-answering.  _Nat. Lang. Eng._ , 7(4):343–360.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mikolov et al. (2013) Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013.  Linguistic regularities in continuous space word representations.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 746–751, Atlanta, Georgia. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA_ , pages 4444–4451. AAAI Press.
  * Sultan and Shahaf (2022) Oren Sultan and Dafna Shahaf. 2022.  Life is a circus and we are the clowns: Automatically finding analogies between situations and processes.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3547–3562, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sun et al. (2022) Yueqing Sun, Qi Shi, Le Qi, and Yu Zhang. 2022.  JointLK: Joint reasoning with language models and knowledge graphs for commonsense question answering.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5049–5060, Seattle, United States. Association for Computational Linguistics.
  * Turney (2005) Peter D. Turney. 2005.  Measuring semantic similarity by latent relational analysis.  In _IJCAI-05, Proceedings of the Nineteenth International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, UK, July 30 - August 5, 2005_ , pages 1136–1141. Professional Book Center.
  * Turney (2012) Peter D. Turney. 2012.  Domain and function: A dual-space model of semantic relations and compositions.  _J. Artif. Intell. Res._ , 44:533–585.
  * Turney et al. (2003) Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. 2003.  Combining independent modules in lexical multiple-choice problems.  In _Recent Advances in Natural Language Processing III, Selected Papers from RANLP 2003, Borovets, Bulgaria_ , volume 260 of _Current Issues in Linguistic Theory (CILT)_ , pages 101–110. John Benjamins, Amsterdam/Philadelphia.
  * Ushio et al. (2021a) Asahi Ushio, Jose Camacho-Collados, and Steven Schockaert. 2021a.  Distilling relation embeddings from pretrained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9044–9062, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ushio et al. (2021b) Asahi Ushio, Luis Espinosa Anke, Steven Schockaert, and Jose Camacho-Collados. 2021b.  BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3609–3624, Online. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _NeurIPS_.
  * Yasunaga et al. (2021) Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021.  QA-GNN: Reasoning with language models and knowledge graphs for question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 535–546, Online. Association for Computational Linguistics.
  * Yuan et al. (2023) Siyu Yuan, Jiangjie Chen, Changzhi Sun, Jiaqing Liang, Yanghua Xiao, and Deqing Yang. 2023.  ANALOGYKB: unlocking analogical reasoning of language models with A million-scale knowledge base.  _CoRR_ , abs/2305.05994.
  * Zhang et al. (2022a) Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022a.  Subgraph retrieval enhanced model for multi-hop knowledge base question answering.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5773–5784.
  * Zhang et al. (2022b) Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, and Jure Leskovec. 2022b.  Greaselm: Graph reasoning enhanced language models.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Zhou et al. (2019) Yilun Zhou, Steven Schockaert, and Julie Shah. 2019.  Predicting conceptnet path quality using crowdsourced assessments of naturalness.  In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_ , pages 2460–2471. ACM.

##  Appendix A Generating Examples of Related Concept Pairs with GPT-4

Two concepts can be related in various ways, including, but not limited to,
the following:

  * •

Semantic relationship: This includes synonyms, antonyms, hyponyms, hypernyms,
meronyms, holonyms, etc.

  * •

Function or purpose, e.g., _key_ and _lock_.

  * •

Manner, way, or style: Concepts that describe the way in which another concept
is accomplished, e.g., _limp_ and _walk_.

  * •

Symbol or representation: Concepts where one represents or symbolizes the
other, e.g., _dove_ and _peace_.

  * •

Degrees of intensity: Concepts that describe different degrees of intensity of
a particular situation, e.g., _shower_ and _monsoon_.

  * •

Cause and effect: Concepts where one causes the other or results from the
other, e.g., _fire_ and _smoke_.

  * •

Sequence or hierarchy: Concepts that follow each other in a sequence or are
organized hierarchically, e.g., _manager_ and _employee_.

In order to identify a large list of such relation types, we asked GPT-4:

⬇

’’’

Let␣A␣and␣B␣be␣two␣concepts␣in␣natural␣language.␣When␣can␣we␣say␣that␣A␣is␣related␣to␣B,␣and␣when␣is␣A␣not␣related␣to␣B?

’’’

We identified approximately 100 relationships through this approach, and then
again used GPT-4 to generate examples for each of these relationships, using
prompts of the following form:

⬇

’’’

A␣way␣two␣concepts␣can␣be␣related␣is

Tools␣and␣materials:␣Concepts␣where␣one␣is␣a␣tool␣or␣material␣used␣to␣create␣or␣interact␣with␣the␣other,␣e.g.,␣’paintbrush’␣and␣’paint.’

Now,␣generate␣100␣high-quality␣examples␣of␣such␣concept␣pairs.

’’’

We prompted the model once for each relation type to maintain a balanced
number of examples for each relationship. The generated examples were utilized
to train our informativeness classifier.

Dataset | #Quest. | #Cand.
---|---|---
SAT | 337 | 5.0
U2 | 228 | 4.3
U4 | 432 | 4.2
BATS | 1799 | 4.0
Google | 500 | 4.0
SCAN | 1616 | 74.5
E-KAR | 154 | 4.0
Table 8: Overview of the considered datasets, showing for each dataset the number of questions in the test set (#Quest.) and the average number of options per question (#Cand.). Conf |  SAT |  U2 |  U4 |  BA |  GO |  SC |  EK
---|---|---|---|---|---|---|---
$[0.0,0.25)$ |  27 |  17 |  26 |  137 |  50 |  652 |  20
$[0.25,0.5)$ |  35 |  21 |  48 |  166 |  42 |  427 |  26
$[0.5,0.75)$ |  100 |  44 |  86 |  197 |  37 |  317 |  43
$[0.75,1.0]$ |  175 |  146 |  272 |  1299 |  371 |  220 |  65
Table 9: Number of analogy questions for each of the difficult ranges.

##  Appendix B Additional Training Details

To train the composition model, we used 200,000 concept pairs from ConceptNet
(version 5.6.0) with informativeness scores greater than 0.75. The concepts
belonged to the English language only. We did not consider ConceptNet
relations "/r/NotCapableOf", "/r/NotDesires", and "/r/NotHasProperty" for
determining intermediate concepts. Our default augmentation strategy, i.e.,
CN+MLP+ss, identified approximately 50 intermediate concepts for each pair of
concepts on average. We used Numberbatch _version 19.08_ and Glove model
_glove-wiki-gigaword-300_ in our augmentation strategy. The concept pairs for
which the strategy could not determine any path were not considered for
training. 10% of the selected concept pairs were utilized for validation. We
trained a function
$\phi:\mathbb{R}^{1024}\times\mathbb{R}^{1024}\rightarrow\mathbb{R}^{n}$,
where the validation set was used to choose the optimal dimension $n$ of the
latent space. We observed in particular that high-dimensional latent spaces
outperformed lower-dimensional ones, where we found $n=81,920$ to be optimal.
The Adam optimizer was used with a learning rate of 0.0025, and the model was
trained for 10 epochs. It is important to note that we did not use validation
splits from the analogy datasets to tune these hyperparameters, as we consider
the unsupervised setting.

##  Appendix C Details about the Analogy Question Datasets

Some basic statistics about the considered datasets are shown in Table 8.
Table 9 shows how many questions there are in each difficulty range, for each
of the datasets, where the difficulty is measured in terms of the confidence
score (3) as before.

##  Appendix D Solving Analogy Questions using GPT-4

To solve analogy questions with GPT-4, we used the following prompt, which was
inspired by LearningExpress (2002).

⬇

’’’

Many␣standardized␣tests,␣including␣high␣school␣entrance␣exams,␣the␣SATs,␣civil␣service␣exams,␣the␣GREs,␣and␣others,␣use␣analogy␣questions␣to␣test␣both␣logic␣and␣reasoning␣skills␣and␣word␣knowledge.␣These␣questions␣ask␣test␣takers␣to␣identify␣relationships␣between␣pairs␣of␣words.

In␣order␣to␣solve␣analogy␣questions,␣you␣must␣first␣have␣a␣clear␣understanding␣of␣the␣words’
definitions and then use that understanding to determine how the words are
related. The key to solving an analogy question is to precisely describe the
relationship between the pair of words and then apply the same relationship to
determine which word completes the analogy. Most analogy questions rely on
your ability to deduce the correct relationship between words and to draw
logical conclusions about the possible answer choices.

The relationships that are found in analogy questions fall into several
general types.

1) Part to Whole. In this type of question, a pair of words consists of a part
and a whole. For example, spoke : wheel. A spoke is part of a wheel.

2) Type and Category. These questions use pairs of words in which one word is
a specific type in a general category. For example, orange : citrus. An orange
is a type of citrus.

3) Degree of Intensity. These questions test your ability to discern nuance of
meaning among pairs of words. For example, shower : monsoon. A shower is light
rainfall and a monsoon is heavy rainfall.

4) Function. These questions pair words that are related through function. For
example, hammer : build. A hammer is used to build.

5) Manner. This type of analogy describes the manner, way, or style by which
an action is accomplished. For example, shamble : walk. Shamble means to walk
in an awkward manner.

6) Symbol or representation. These questions pair words in which one word is
the symbol of the other. For example, dove : peace. A dove is a symbol of
peace.

7) Action and significance. In this type of analogy one word describes an
action and the other word indicates the significance of the action. For
example, cry : sorrow. To cry signifies sorrow

Analogy questions can also be used to test word knowledge and factual content.
Word knowledge questions are generally pairs of synonyms or pairs of antonyms.
Factual content questions demand a certain level of general knowledge, and
cannot be deduced from the relationship alone.

Given the word pair, your aim is to choose the word pair from choices that is
analogously most similar. Also, give an explanation. The explanation should be
precise. I will show some examples then you will have to do it yourself.

Query = [’banana’, ’peel’]; Choices = [[’egg’, ’crack’], [’carrot’, ’uproot’],
[’apple’, ’core’], [’bread’, ’slice’], [’corn’, ’husk’]]

Answer: choice number 4; Explanation: A banana has a peel that can be removed,
and corn has a husk that can be removed.

Query = [’birds’, ’wings’]; Choices = [[’moose’, ’antlers’], [’camel’,
’hump’], [’spider’, ’legs’], [’alligator’, ’tail’], [’cat’, ’whiskers’]]

Answer: choice number 2; Explanation: Birds have wings, and spiders have legs.

Query = [’berate’, ’criticize’]; Choices = [[’goad’, ’urge’], [’accuse’,
’apologize’], [’regret’, ’remember’], [’betray’, ’follow’], [’evaluate’,
’praise’]]

Answer: choice number 0; Explanation: To berate is to criticize, and to goad
is to urge.

Now, answer the following questions:

’’’’

Conf | RelB | Cond | Dir | GPT4
---|---|---|---|---
$[0.0,0.25)$ | 26.3 | 30.0 | 29.1 | 39.9
$[0.25,0.5)$ | 39.6 | 41.4 | 41.2 | 46.5
$[0.5,0.75)$ | 53.0 | 52.2 | 50.2 | 54.5
$[0.75,1.0]$ | 81.2 | 76.9 | 71.4 | 82.8
Table 10: Results for the main experiments, micro-averaged across all datasets. Conf | RelB | Cond | Dir | GPT4
---|---|---|---|---
$[0.0,0.25)$ | 43.9 | 45.5 | 48.6 | 72.5
$[0.25,0.5)$ | 39.3 | 39.9 | 41.3 | 51.6
$[0.5,0.75)$ | 41.5 | 45.6 | 41.9 | 52.4
$[0.75,1.0]$ | 67.3 | 64.5 | 60.4 | 68.2
Table 11: Micro-averaged results, broken down according to the confidence
predicted by the informativeness classifier that was trained on Conceptnet.

##  Appendix E Additional Results

Table 1 reports the macro-averaged accuracy, as a summary of the performance
of the different methods across all datasets. In this macro average, each
_dataset_ carries the same weight. To complement this result, Table 10 instead
shows the micro-averaged result, where each _analogy question_ carries the
same weight, regardless of the dataset it appears in. The main conclusions
remain the same as the ones we could draw based on the macro-average. One
difference is that the spread in performance between the easiest and the
hardest questions is even wider for the micro-averaged results.

For comparison, in Table 11 we also show the micro-averaged result, but in
this case, we break down the results based on an informativeness classifier
that was trained on examples from ConceptNet as positive examples. In
contrast, for our main experiments, this informativeness classifier was
trained on examples we obtained from GPT-4. As can be seen in Table 11, the
informativeness classifier based on ConceptNet is far less successful in
identifying easy and hard questions. For instance, GPT-4 actually achieves the
best results on the questions that are predicted to be hardest.


&\left. - C D \hat{S'}  + D \hat{H'}  {U_y}+ D^2  \hat{Q'} - \hat{\delta'} D^2  + E D  \hat{Y'}   \right] + \omega^2 \left[  \left(\sfrac{1}{\text{We}}\right)^{2} ( \hat{P'} D - {(1 - \nu)} \hat{J'}) + \left(\sfrac{1}{\text{We}}\right) (2 X \right. \nonumber \\
&\left. \hat{P'} D - 2 X {(1 - \nu)} \hat{J'} - 6 A' D + 2 \hat{P'} {(\nu -1)}  {U_y} - 2 C' D \hat{R} - 2 C D \hat{R'} - 2 D \hat{J'}  {U_y} - 2 D^2  \hat{W'} - \right. \nonumber \\
&\left. E D  \hat{Z'} + 6 {(1 - \nu)} H'   + 2 {(1 - \nu)} N' \hat{S} + 2  {(1 - \nu)} N \hat{S'} + 2 {(1 - \nu)} M' \hat{R}  + 2 {(1 - \nu)} M \hat{R'} + 2 \right. \nonumber \\
&\left. {(1 - \nu)}^{2}  \hat{W'}+ {(1 - \nu)} D  \hat{Z'}+ {(1 - \nu)} E  \hat{\tau'}+ D^2  \hat{\tau'} ) + (1 - \nu)( 3 X  H'  + X  N' \hat{S} + X  N \hat{S'} +  X  M' \right. \nonumber \\
&\left.\hat{R} +  X  M \hat{R'} + X  D  \hat{Z'} + X  E  \hat{\tau'} + 3 A'    {U_y}+ C'  \hat{R}  {U_y}+ C  \hat{R'}  {U_y}- 3 \delta'  E  + E   {U_y} \hat{Z'} - 3  N' R  - 3  N R' \right. \nonumber \\
&\left. + 3  M' S  + 3  M S'   - 3 \omega^{2}   Q' - 3  D   Y' )- 3 X A' D   - X C' D \hat{R} - X C D \hat{R'} -  X E D  \hat{Z'}+ X D^2  \hat{\tau'} - \right. \nonumber \\
&\left. 3 C' D S   - 3 C D S'  + 3 D H'   {U_y}+ D N' \hat{S}  {U_y}+ D N \hat{S'}  {U_y}+ D M' \hat{R}  {U_y} + D M \hat{R'}  {U_y} + 3 D^2   Q'+ \right. \nonumber \\
&\left. D^2  {U_y} \hat{Z'} - 3 \delta' D^2  + 3 E D   Y' \right] + \omega \left[ \left(\sfrac{1}{\text{We}}\right)^{2} (X \hat{A'} D - X {(1 - \nu)} \hat{H'} + \hat{A'} {(\nu -1)}  {U_y}+ 2   P' D + \right. \nonumber \\
&\left. C' D \hat{S} + C D \hat{S'} - D \hat{H'}  {U_y} - D^2  \hat{Q'}- 2   {(1 - \nu)} J'+ {(1 - \nu)} N' \hat{R}  + {(1 - \nu)} N \hat{R'}  - {(1 - \nu)} M' \hat{S} \right. \nonumber \\
&\left.  - {(1 - \nu)} M \hat{S'}  + {(1 - \nu)}^{2}  \hat{Q'} ) + \left(\sfrac{1}{\text{We}}\right) ( 4 X  P' D  + 2 X C' D \hat{S} + 2 X C D \hat{S'}  + X \hat{\delta'} {(1 - \nu)} E  + X \hat{ \delta' } \right. \nonumber \\
&\left. D^2 - X E D  \hat{Y'}- 4X {(1 - \nu)} J'   + 2 X {(1 - \nu)}  N' \hat{R} + 2 X {(1 - \nu)} N \hat{R'} - 2 X {(1 - \nu)} M' \hat{S} - 2 X  \right. \nonumber \\
&\left. {(1 - \nu)} M \hat{S'} + X {(1 - \nu)} D  \hat{Y'}+ 4 P' {(\nu -1)}   {U_y}+ 2 C' {(\nu -1)} \hat{S}  {U_y}+ 2 C {(\nu -1)} \hat{S'}  {U_y} - 4 C' D R \right. \nonumber \\
&\left. - 4 C D R'   + D \hat{\delta'} E  {U_y} - 4 D J'   {U_y} + 2 D N' \hat{R}  {U_y} + 2 D N \hat{R'}  {U_y} - 2 D M' \hat{S}  {U_y}- 2 D M \hat{S'}  {U_y} - \right. \nonumber \\
&\left. 4 D^2   W' + D^2  {U_y} \hat{Y'} + \hat{\delta'} {D} {(\nu -1)} {U_y} - E {(\nu -1)}  {U_y} \hat{Y'}- 2 E D    Z ' + 4  {(1 - \nu)} N' S  + 4 {(1 - \nu)} N S' \right. \nonumber \\
&\left.  + 4 {(1 - \nu)} M' R  + 4 {(1 - \nu)} M R'   + 4 {(1 - \nu)}^{2}   W'+ 2 {(1 - \nu)} D   Z'+ 2 {(1 - \nu)} E   \tau' + 2 D^2   \tau' ) + {U_y}^{2}  \right. \nonumber \\
&\left. (D^2 \hat{\delta'}  - \hat{\delta'} E {(\nu -1)}  ) + U_y (- X \hat{\delta'} E D   + X \hat{\delta'} {(1 - \nu)} D  - 2 C' {(\nu -1)} R   - 2 C {(\nu -1)} R'   + 2 D \right. \nonumber \\
&\left. N' S   + 2 D N S'   + 2 D M' R   + 2 D M R'   + 2 D^2    Z' - 2 E {(\nu -1)}    Z' )- 2 X C' D R  - 2 X C D R'  - \right. \nonumber \\
&\left. 2 X E D   Z' + 2 X {(1 - \nu)} N' S   + 2 X {(1 - \nu)} N S'  + 2 X {(1 - \nu)} M' R  + 2 X {(1 - \nu)} M R'  + 2 X \right. \nonumber \\
&\left. {(1 - \nu)} D   Z'+ 2 X {(1 - \nu)} E   \tau' +  2 X D^2   \tau'   \right] + \left[ \left(\sfrac{1}{\text{We}}\right)^{2}( X A'  D + X  C' D \hat{R}+ X  C D \hat{R'} - X  {(1 - \nu)} \right. \nonumber \\
&\left.  H'  - X  {(1 - \nu)} N' \hat{S}- X  {(1 - \nu)} N \hat{S'}- X  {(1 - \nu)} M' \hat{R}- X  {(1 - \nu)} M \hat{R'} + A'  {(\nu -1)}  {U_y} +  C' \right. \nonumber \\
&\left. {(\nu -1)} \hat{R} {U_y}+  C {(\nu -1)} \hat{R'} {U_y} +   C' D S +  C D S'  -  D H'  {U_y}-  D N' \hat{S} {U_y}-  D N \hat{S'} {U_y} -  D M' \hat{R} {U_y} \right. \nonumber \\
&\left.  - D M \hat{R'} {U_y}-  D^2  Q' +   {(1 - \nu)} N' R +  {(1 - \nu)} N R'  -  {(1 - \nu)} M' S-  {(1 - \nu)} M S'  +  {(1 - \nu)}^{2}  Q')\right. \nonumber \\
&\left.  + \left(\sfrac{1}{\text{We}}\right) (2 X  C' D S + 2 X C D S'  + X  \delta' {(1 - \nu)} E + 2  C' {(\nu -1)} S  {U_y} + 2  C {(\nu -1)} S'  {U_y} +  D \delta' E  {U_y} \right. \nonumber \\
&\left. + 2  D N' R  {U_y}  + 2  D N R'  {U_y} - 2  D M' S  {U_y} - 2  D M S'  {U_y} +  D^2  {U_y} Y' +  \delta' {D} {(\nu -1)}  {U_y} -  E {(\nu -1)} \right. \nonumber \\
&\left.  {U_y} Y'+ X  \delta' D^2 - X  E D  Y' + 2 X  {(1 - \nu)} N' R  + 2 X  {(1 - \nu)} N R' - 2 X  {(1 - \nu)} M' S - 2 X   {(1 - \nu)} \right. \nonumber \\
&\left. M S' + X  {(1 - \nu)} D  Y') + {(U_y)}^{2} ( D^2 \delta'  - \delta' E {(\nu -1)} ) + U_y (- X \delta' E D  + X \delta' {(1 - \nu)} D  )  \right],
\label{appB:DRPi_alpha_omega}
\end{align}

[Ablowitz & Fokkas , 2003]Ablowitz M. J. and Fokkas A. S., `Complex Variables: Introduction and Applications', Cambridge University Press (2003).

[Ahmadikhamsi et al , 2020] Ahmadikhamsi S., Golfier F., Oltean C., Lefévre E. and Bahrani S. A., `Impact of surfactant addition on non-Newtonian fluid behavior during viscous fingering in Hele-Shaw cell', Phys. Fluids 32 (2020) 012103.

[Al-Housseiny et al , 2013]Al-Housseiny T. T. and Stone H. A., `Controlling viscous fingering in tapered Hele-Shaw cells', Phys. Fluids 25 (2013) 092102.

[Alhushaybari & Uddin , 2019] Alhushaybar A. and Uddin J., `Convective and absolute instability of viscoelastic liquid jets in the presence of gravity', Phys. Fluids 31 (2019) 044106.

[Alhushaybari & Uddin , 2020] Alhushaybar A. and Uddin J., `Absolute instability of free-falling viscoelastic liquid jets with surfactants', Phys. Fluids 32 (2020) 013102.

[Almgren et al , 1993]Almgren R., Dai W. S. and Hakim V., `Scaling behaviour in anisotropic Hele-Shaw flow', Phys. Rev. Lett. 71 (1993) 3461-3464.

[Alvarez-Lacalle et al , 2004]Alvarez-Lacalle E., Ortin J. and Casademunt J., `Low viscosity contrast fingering in a rotating Hele-Shaw cell', Phys. Fluids 16(4), (2004) 908-924.

[Amarouchene et al , 2001]Amarouchene Y., Bonn D.,  Meunier J. and Kellay H., `Inhibition of the finite-time singularity during droplet fission of a polymeric fluid', Phys. Rev. Lett. 86 (2001) 3558-3561.

[Anderson et al , 1998]Anderson D. M., McFadden G. B. and Wheeler A. A., `Diffuse-interface methods in fluid mechanics', Ann. Rev. Mater. Res. 30 (1998) 139-165.

[Anjos et al , 2017] Anjos P. H. A., Dias E. O. and Miranda J. A., `Inertia-induced dendriticlike patterns in lifting Hele-Shaw flows', Phys. Rev. Fluids 2 (2017) 014003.

[Azaiez & Homsy , 1994] Azaiez J. and Homsy G. M., `Linear stability of free shear flow of viscoelastic liquids', J. Fluid Mech., 268 (1994) 37-69.

[Badalassi et al , 2003]Badalassi V. E., Ceniceros H. D. and Banerjee S., `Computations of multiphase systems with phase field models', J. Comp. Phys. 190 (2003) 371-397.

[Bansal et al , 2021]Bansal D., Ghosh D. and Sircar S., `Spatiotemporal linear stability of viscoelastic free shear flows: non-affine response regime', Phys. Fluids 33 (2021) 054106.

[Ben Amar & Bonn , 2005] Ben Amar M. and Bonn D., `Fingering instabilities in adhesive failure', Physica D 209 (2005) 1-16.

[Bensimon et al , 1986] Bensimon D., Kadanoff L. P., Liang S., Shraiman B. I. and Tang C., `Viscous flow in two dimensions', Rev. Mod. Phys. 58 (1986) 977.

[Bihi et al , 2016] Bihi I., Baudoin M., Butler J. E., Faille C. and Zoueshtiagh F., `Inverse Saffman-Taylor Experiments with Particles Lead to Capillarity Driven Fingering Instabilities', Phys. Rev. Lett. 117 (2016) 034501.

[Bird et al , 1987] Bird R. B., Armstrong R. C. and Hassager O., `Dynamics of polymeric liquids, Volume 1: Fluid Mechanics', Wiley Intersciences (1987).

[Bistagnino et al , 2007]Bistagnino A., Boffetta G., Celani A., Mazzino A., Puliafito A. and Vergassola M., `Nonlinear dynamics of the viscoelastic Kolmogorov flow', J. Fluid Mech. 590 (2007) 61-80.

[Boger & Nguyen , 1978]Boger D. V. and Nguyen H., `A model viscoelastic fluid', Polym. Engng Sci. 18 (1978) 1037-1043.

[Booth , 2010]Booth R. J. S., `On the growth of the mixing zone in miscible viscous fingering', J. Fluid Mech. 655 (2010) 527-539.

[Boyer , 2002]Boyer F., `A theoretical and numerical model for the study of incompressible mixture flows', Comput. Fluids 31 (2002) 41-68.

[Briggs , 1964]Briggs R. J., `Electron-stream interaction with plasmas', MIT Press, Cambridge (1964).

[Cadot & Kumar , 2000]O. Cadot and S. Kumar, `Experimental characterization of viscoelastic effects on two- and three-dimensional shear instabilities', J. Fluid Mech. 416 (2000) 151-172.

[Cahn & Hilliard, 1958]Cahn J. W. and Hilliard J. E., `Free energy of a nonuniform system. I. Interfacial free energy', J. Chem. Phys. 28 (1958) 258-267.

[Carreau & Kee , 1979] Carreau, P. J. and De Kee D., `Review of Some Useful Rheological Equations', Can. J. Chern. Eng. 57, (1979) 3-15.

[Chandra et al , 2020]Chandra B., Shankar V. and Das D., `Early transition, relaminarization and drag reduction in the flow of polymer solutions through microtubes', J. Fluid Mech. 885 (2020) A47.

[Chang et al , 1999]Chang H. C., Demekhin E. A. and Kalaidin E., `Iterated stretching of viscoelastic jets', Phys. Fluids 11 (1999) 1717.

[Chuoke et al , 1959] Chuoke R. L., van Meurs P., and van der Pol C., `The instability of slow immiscible viscous liquid-liquid displacements in permeable media', Pet. Trans. AIME 216 (1959) 188-194.

[Combescot et al , 1986] Combescot R., Dombre T., Hakim V. and Pomeau Y., `Shape selection of Saffman-Taylor fingers', Phys. Rev. Lett. 56 (1986) 2036-2039.

[Constantin et al , 1993]Constantin P., Dupont T. F., Goldstein R., Kadanoff L. P. and Shelley M. J., `Droplet breakup in a model of the Hele-Shaw cell', Phys. Rev. E 47(6) (1993) 4169-4181.

[Couder , 2000] Couder Y., `Viscous fingering as an archetype of growth pattern', In Perpectives in fluid dynamics (eds G. K. Batchelor, H. K. Moffatt, M. G. Worster), Cambridge Univ. Press (2000) 53-104.

[Dai & Shelley , 1993] Dai W. S. and Shelley M. J., `A numerical study of the effect of surface tension and noise on an expanding Hele-Shaw bubble', Phys. Fluids A 5(9), (1993) 2131-2146.

[Degregoria & Schwartz , 1986] Degregoria A. J. and Schwartz L. W., `A boundary integral method for two phase displacement in Hele-Shaw cells', J. Fluid Mech. 164 (1986) 383-400.

[Delbende & Chomaz , 1998] Delbende I. and Chomaz J. M., `Nonlinear convective/absolute instabilities in parallel two-dimensional wakes', Phys. Fluids 10 (1998) 2724.

[Dias & Miranda , 2011] Dias E. O. and Miranda J. A., `Influence of inertia on viscous fingering patterns: Rectangular and radial flows', Phys. Rev. E 83 066312 (2011).

[Dias & Miranda , 2013] Dias E. O. and Miranda J. A., `Wavelength selection in Hele-Shaw flows: A maximum-amplitude criterion', Phys. Rev. E 88(1) (2013) 013016.

[Ding et al , 2007]Ding H., Spelt P. D. M. and Shu C., `Diffuse interface model for incompressible two-phase flows with large density ratios', J. Comp. Phys. 226 (2007) 2078-2095.

[Drazin , 1979]Drazin P. G., Zaturska M. B. and Banks W. H. H., `On the normal modes of parallel flow of inviscid stratified fluid. Part 2. Unbounded flow with propagation at infinity', J. Fluid Mech. 95(4) (1979) 681-705.

[Ecke & Backhaus , 2016] Ecke R. E. and Backhaus S., `Plume dynamics in Hele-Shaw porous media convection', Phil. Trans. A 374 (2016) 20150420.

[Fast et al , 2001]Fast P., Kondic L., Palffy-Muhoray P. and Shelley M. J., `Pattern formation in non-Newtonian Hele-Shaw flow', Phys. Fluids 13 (2001) 1191-1212.

[Fast & Shelley , 2004]Fast P. and Shelley M. J., `A moving overset grid method for interface dynamics applied to non-Newtonian Hele-Shaw flow', J. Comp. Phys. 195 (2004) 117-142.

[Fernandez et al , 2002]Fernandez J., Kurowski P., Petitjeans P. and Meiburg E., `Density-driven unstable flows of miscible fluids in a Hele-Shaw cell', J. Fluid Mech. 451 (2002) 239-260.

[Gòmez et al , 2016] Franco-Gómez A., Thompson A. B., Hazel A. L. and Juel A., `Sensitivity of Saffman-Taylor fingers to channel-depth perturbations', J. Fluid Mech. 794 (2016) 343-368.

[Gadêlha & Miranda , 2009] Gadêlha H. & Miranda J. A., `Effects of normal viscous stresses on radial viscous fingering', Phys. Rev. E 79 (2009) 066312.

[Gallaire & Brun, 2017]Gallaire F. and Brun P. T., `Fluid dynamic instabilities: theory and application to pattern forming in complex media', Phil. Trans. R. Soc. 375 (2017) 1-44.

[Goldstein et al , 1993] Goldstein R. E., Pesci A. I. and Shelley M. J., `Topology Transitions and Singularities in Viscous Flows', Phys. Rev. Lett. 70(20) (1993) 3043-3047.

[Goyal et al , 2007]Goyal N., Pichler H. and Meiburg E., `Variable-density miscible displacements in a vertical Hele-Shaw cell: linear stability', J. Fluid Mech. 584 (2007) 357-372.

[Grillet et al , 1999] Grillet A. M., Lee A. G. and Shaqfeh E. S. G., `Observations of ribbing instabilities in elastic fluid flows with gravity stabilization', J. Fluid Mech. 399 (1999) 49-83.

[Guimaraes et al , 2020] Guimaraes M. C., Pimentel N., Pinho F. T. and da Silva C. B., `Direct numerical simulations of turbulent viscoelastic jets', J. Fluid Mech. 899 (2020) A11.

[Gurtin , 1994]Gurtin M. E., `Generalized Ginzburg-Landau and Cahn-Hilliard equations based on a micro force balance', Physica D 92 (1994) 178-192.

[Haudin et al , 2014] Haudin F., Riolfo L. A., Knaepen B., Homsy G. M. and De Wit A., `Experimental study of a buoyancy-driven instability of a miscible horizontal displacement in a Hele-Shaw cell', Phys. fluids 26 (2014) 044102.

[Healy , 2009] Healy J. J., `Destabilizing effects of confinement on homogeneous mixing layers', J. Fluid Mech. 623 (2009) 241-271.

[Heussler et al , 2014]Heussler F. H. C., Oliveira R. M., John M. O. and Meiburg E., `Three-dimensional Navier-Stokes simulations of buoyant, vertical miscible Hele-Shaw displacements', J. Fluid Mech. 752 (2014) 157-183.

[Homsy , 1987] Homsy G. M., `Viscous fingering in porous-media', Annu. Rev. Fluid Mech. 19, (1987) 271Ð311.

[Hong & Langer , 1986] Hong D. C. and Langer J. S., `Analytic theory of the selection mechanism in the Saffman-Taylor problem', Phys. Rev. Lett., 56 (1986) 2032-2035.

[Hooshanginejad et al , 2019] Hooshanginejad A., Druecke B. C. and Lee S., `Stability analysis of a particle band on the fluid-fluid interface', J. Fluid Mech. 869 (2019) R2.

[Hou et al , 1997] Hou T. Y., Li Z., Osher S. and Zhao H., `A hybrid method for moving interface problems with application to the Hele-Shaw flow ', J. Comp. Phys 134 (1997) 236-252.

[Hou et al , 2001] Hou T. Y., Lowengrub J. S. and Shelley M. J., `Boundary integral methods for multicomponent fluids and multiphase materials', J. Comp. Phys. 169 (2001) 302-362.

[Huerre & Monkewitz , 1985] Huerre P. and Monkewitz P. A., `Absolute and convective instabilities in free shear layers', J. Fluid Mech. 159, (1985) 151-168.

[Huerre & Monkewitz , 1990]Huerre P. and Monkewitz P. A., `Local and global instabilities in spatially developing flows', Ann. Rev. Fluid Mech. 22 (1990) 473-537.

[Huerre et al , 2015] Huerre A., Theodoly O., Leshansky A. M., Valignat M. P., Cantat I. and Jullien M. C., `Droplets in Microchannels: Dynamical Properties of the Lubrication Film', Phys. Rev. Lett. 115 (2015) 064501.

[Jacqmin , 1999]Jacqmin D., `Calculations of two-phase Navier-Stokes flows using phase-field modeling', J. Comp. Phys. 155 (1999) 96-127.

[John et al , 2013]John M. O., Oliveira R. M., Heussler F. H. C. and Meiburg E., `Variable density and viscosity, miscible displacements in horizontal Hele-Shaw cells. Part 2. Nonlinear simulations', J. Fluid Mech. 721 (2013) 295-323.

[Kim , 2005]Kim J., `A continuous surface tension force formulation for diffuse-interface models', J. Comp. Phys. 204 (2005) 784-804.

[Kim et al , 2009] Kim H., Funada T., Joseph D. D. and Homsy G. M., `Viscous potential flow analysis of radial fingering in a Hele-Shaw cell', Phys. Fluids 21 (2009) 074106.

[Koch , 1986]Koch W., `Direct resonance in Orr-Sommerfeld problems', Acta Mech. 59 (1986) 11-29.

[Kondic et al , 1996]Kondic L., Palffy-Muhoray P. and Shelley M. J., `Models of non-Newtonian Hele-Shaw flow', Phys. Rev. E 54 (1996) R4536.

[Kondic et al , 1998]Kondic L., Palffy-Muhoray P. and Shelley M. J., `Non-Newtonian Hele-Shaw flow and the Saffman-Taylor instability', Phys. Rev. Lett. 80 (1998) 1433-1436.

[Kupfer et al , 1987]Kupfer K., Bers A. and Ram A. K., `The cusp map in the complex-frequency plane for absolute instability', Phys. Fluids 30(10) (1987) 3075-3082.

[Lagree et al , 2016]Lagrée B., Zaleski S. and Bondino I., `Simulation of Viscous Fingering in Rectangular Porous Media with Lateral Injection and Two- and Three-Phase Flows', Transp. Porous Med. 113 (2016) 491-510.

[Lee et al , 2002] Lee H., Lowengrub J. S. and Goodman J., `Modeling pinchoff and reconnection in a Hele-Shaw cell. I. The models and their calibration', Phys. Fluids 14(2) (2002) 492-513.

[Lemaire et al , 1991] Lemaire E., Levitz P., Daccord G. and Damme H. Van, `From viscous fingering to viscoelastic fracturing in colloidal fluids', Phys. Rev. Lett. 67, (1991) 2009.

[Li et al , 2007] Li S., Lowengrub J. S. and Leo P. H., `A rescaling scheme with application to the long-time simulation of viscous fingering in a Hele-Shaw cell', J. Comp. Phys. 225 (2007) 554-567.

[Li et al , 2009]Li S., Lowengrub J. S., Fontana J. and Palffy-Muhoray P., `Control of viscous fingering patterns in a radial Hele-Shaw cell', Phys. Rev. Lett. 102 (2009) 174501.

[Lindner et al , 2000]Lindner A., Bonn D. and Meunier J., `Viscous fingering in a shear-thinning fluid', Phys. Fluids 12(2) (2000) 256-261.

[Lindner et al , 2002]Lindner A., Bonn D., Poire E. C., Ben Amar M. and Meunier J., `Viscous fingering in non-Newtonian fluids', J. Fluid Mech. 469 (2002) 237-256.

[Lindner et al , 2006]Chevalier C., Ben Amar M., Bonn D. and Lindner A., `Inertial effects on Saffman-Taylor viscous fingering', J. Fluid Mech. 552 (2006) 83-97.

[Lindner & Wagner , 2009]Lindner A. and Wagner C.,`Viscoelastic surface instabilities', C. R. Physique 10 (2009) 712-727.

[Lingwood , 1997]Lingwood R. J., `Absolute instability of the Ekman layer and related rotating flows', J. Fluid Mech. 331 (1997) 405-428.

[Logvinov et al , 2010]Logvinov O. A., Ivashnyov O. E. and Smirnov N. N., `Evaluation of viscous fingering width in Hele-Shaw flows', Acta Astronaut. 67 (2010) 53-59.

[Lopez et al , 2019] Veiga-López F., Martnez-Ruiz D., Fernándes-Tarrazo E. and Sáchez-Sanz M.,`Experimental analysis of oscillatory premixed flames in a Hele-Shaw cell propagating towards a closed end', Comb. Flame 201 (2019) 1-11.

[Lowengrub & Truskinovsky , 1998]Lowengrub J. S. and Truskinovsky L. Quasi-incompressible Cahn-Hilliard fluids and topological transitions', Proc. R. Soc. Lond. A 454 (1998) 2617-2654.

[Maher , 1985] Maher J. V., `Development of viscous fingering patterns', Phys. Rev. Lett. 54 (1985) 1498-1501.

[Maxworthy , 1989] Maxworthy T., `Experimental study of interface instability in a Hele-Shaw cell', Phys. Rev. A 39(11), (1989) 5863-5866.

[McLean & Saffman , 1981] McLean J. and Saffman P., `The effect of surface tension on the shape of fingers in a HeleÐShaw cell', J. Fluid Mech. 102 (1981) 455-469.

[Meiburg & Homsy, 1988] Meiburg E. and Homsy G. M., `Nonlinear unstable viscous fingering in Hele-Shaw flows. II. Numerical simulation', Phys. Fluids 31 (1988) 429-439.

[Meiburg , 1989]Meiburg E., `Bubbles in a Hele-Shaw cell: numerical simulation and three-dimensional effects', Phys. Fluids A 1 (1989) 938-946.

[Miller , 2005] Miller J. C., `Shear flow instabilities in viscoelastic fluids', Ph.D. dissertation, Univ. Cambridge (2005).

[Mora & Manna , 2010]Mora S. and Manna M., `Saffman-Taylor instability of viscoelastic fluids: From viscous fingering to elastic fractures', Phys. Rev. E 81 (2010) 026305.

[Morrow et al , 2019]Morrow L. C., Moroney T. J. and McCue S. W., `Numerical investigation of controlling interfacial instabilities in non-standard Hele-Shaw configurations', J. Fluid Mech. 877 (2019) 1063-1097.

[Nakamura , 1997]Nakamura Y., `Spatiotemporal dynamics of forced periodic flows in a confined domain', Phys. Fluids 9 (1997) 3275.

[Oertel , 1990]Oertel Jr. H., `Wakes behind blunt bodies', Ann. Rev. Fluid Mech. 22 (1990) 539-564.

[Oliveira & Meiburg , 2011]Oliveira R. M. and Meiburg E., `Miscible displacements in Hele-Shaw cells: three-dimensional Navier-Stokes simulations', J. Fluid Mech. 687 (2011) 431-460.

[Park & Homsy , 1985] Park C. W. and Homsy G. M., `The instability of long fingers in Hele-Shaw flows', Phys. Fluids 28 (1985) 1583-1585.

[Paterson , 1985] Paterson L., `Fingering with miscible fluids in a Hele Shaw cell', Phys. Fluids 28(1), (1985) 26-30.

[Patne & Shankar , 2017] Patne R. and Shankar V., `Absolute and convective instabilities in combined Couette-Poiseuille flow past a neo-Hookean solid', Phys. Fluids 29 (2017) 124104.

[Petrolo et al , 2020] Petrolo D., Chiapponi L., Longo S., Celli M., Bartella A. and Di Federico V., `Onset of DarcyÐBénard convection under throughflow of a shear-thinning fluid', J. Fluid Mech. 889 (2020) R2.

[Pier , 2008] Pier B., `Local and global instabilities in the wake of a sphere', J. Fluid Mech. 603 (2008) 39-61.

[Pipe , 2005] Pipe C., `Experiments investigating the effects of fluid elasticity on laminar vortex shedding from a cylinder', MS Thesis, EPFL (2005).

[Poiré & Ben Amar , 1998] Poiré E. C. and Ben Amar M., `Finger behavior of a shear thinning fluid in a Hele-Shaw cell', Phys. Rev. Lett. 81 (1998) 2048-2051.

[Puzović et al , 2018] Puzović D. P., Peng G. G., Lister J. R., Heil M. and Juel A., `Viscous fingering in a radial elastic-walled Hele-Shaw cell', J. Fluid Mech. 849 (2018) 163-191.

[Rabaud et al , 1988] Rabaud M., Couder Y. and Gerard N., `Dynamics and stability of anomalous Saffman-Taylor fingers', Phys. Rev. A 37 (1988) 935-947.

[Rallison & Hinch , 1995]Rallison J. M. and Hinch E. J., `Instability of a high-speed submerged elastic jet', J. Fluid Mech. 288 (1995) 311-324.

[Ray & Zaki , 2014] Ray P. K. and Zaki T. A., `Absolute instability in viscoelastic mixing layers', Phys. Fluids 26 (2014) 014103.

[Ray & Zaki , 2015] Ray P. K. and Zaki T. A., `Absolute/convective instability of planar viscoelastic jets', Phys. Fluids 27 (2015) 014110.

[Reinelt , 1987]Reinelt D. A., `The effect of thin film variations and traverse curvature on the shape of fingers in a Hele-Shaw cell', Phys. Fluids 30 (1987) 2617.

[Reinelt , 1987]Reinelt D. A., `Interface conditions for two-phase displacement in Hele-Shaw cells', J. Fluid Mech. 183 (1987) 219-234.

[Richter et al , 2011]Richter D., Shaqfeh E. S. G. and Laccarino G., `Floquet stability analysis of viscoelastic flow over a cylinder', J. Non-Newt. Fluid Mech. 166 (2011) 554-565.

[Ro & Homsy , 1995]Ro J. S. and Homsy G. M., `Viscoelastic free surface flows: thin film hydrodynamics of Hele-Shaw and dip coating flows', J. Non-Newt. Fluid Mech. 57 (1995) 203-225.

[Ruith & Meiburg , 2000]Ruith M. and Meiburg E., `Miscible rectilinear displacements with gravity override. Part 1. Homogeneous porous medium', J. Fluid Mech. 420 (2000) 225-257.

[Saffman & Taylor , 1958] Saffman P. and Taylor G., `The penetration of a fluid into a porous medium or Hele-Shaw cell containing a more viscous liquid', Proc. R. Soc. A 245 (1958) 312-329.

[Sahin & Owens , 2004]Sahin M. and Owens R. G., `On the effects of viscoelasticity on two-dimensional vortex dynamics in the cylinder wake', J. Non-Newt. Fluid Mech. 123 (2004) 121-139.

[Saintyves et al , 2013] Saintyves B., Dauchot O. and Bouchaud E., `Bulk Elastic Fingering Instability in Hele-Shaw Cells', Phys. Rev. Lett. 111 (2013) 047801.

[Sarkar & Jasnow , 1989]Sarkar S. K. and Jasnow D., `Viscous fingering in an anisotropic Hele-Shaw cell', Phys. Rev. A 39 (1989) 5299-5307.

[Schröder et al , 2002] Schröder M., Kassner K., Rehberg I., Claret J. and Sagués F. , `Experimental investigation of the initial regime in fingering electrodeposition: dispersion relation and velocity measurements', Phys. Rev. E 65(4) (2002) 041607.

[Shelley et al , 1993]Shelley M. J., Goldstein R. E. and Pesci A. I., `Topological Transitions in Hele-Shaw Flow', in Singularities in Fluids, Plasmas and Optics (NATO ASI Series C) 404, Springer (1993) 167-188.

[Shoji et al , 2020] Shoji T., Harris E. W., Bernard A., Schein S. G. and Karagozian A. R., `On the origins of transverse jet shear layer instability transition', J. Fluid Mech. 890 (2020) A7-1.

[Shokri et al , 2017]Shokri H., Kayhani M. H. and Norouzi M., `Nonlinear simulation and linear stability analysis of viscous fingering instability of viscoelastic liquids', Phys. Fluids 29 (2017) 033101.

[Shraiman , 1986] Shraiman B., `Velocity selection in the Saffman-Taylor problem', Phys. Rev. Lett. 56 (1986) 2028-2031.

[Sircar & Bansal , 2019]Sircar S. and Bansal D., `Spatiotemporal linear stability of viscoelastic free shear flows: dilute response regime', Phys. Fluids 31 (2019) 084104.

[Slim et al , 2013] Slim A. C., Bandi M. M., Miller J. C. and Mahadevan L., `Dissolution-driven convection in a Hele-Shaw cell', Phys. fluids 25 (2013) 024101.

[Sun & Beckermann , 2004]Sun Y. and Beckermann C., `Diffuse interface modeling of two-phase flows based on averaging:mass and momentum equations', Physica D 198 (2004) 281-308.

[Suzuki et al , 2020] Suzuki R. X., Nagatsu Y., Mishra M. and Ban T., `Phase separation effects on a partially miscible viscous fingering dynamics', J. Fluid Mech. 898 (2020) A11.

[Tabeling & Libchaber , 1985] Tabeling P. and Libchaber A., `Film draining and the Saffman-Taylor problem', Phys. Rev. A 33(1) (1985) 794-796.

[Tan & Homsy , 1988] Tan C. T. & Homsy G. M. `Simulation of nonlinear viscous fingering in miscible displacement', Phys. Fluids 31(6) (1988) 1330.

[Tan & Homsy , 1992] Tan C. T. & Homsy G. M. `Viscous fingering with permeability heterogeneity', Phys. Fluids 4(6) (1992) 1099-1101.

[Tryggvason & Aref , 1983]Tryggvason G. and Aref H., `Numerical experiments on Hele-Shaw flows with sharp interface', J. Fluid Mech. 136 (1983) 1-30.

[Upchurch & Meiburg , 2008]Upchurch E. and Meiburg E., `Miscible porous media displacements driven by non-vertical injection wells', J. Fluid Mech. 607 (2008) 289-312.

[Vihinen et al , 1997] Vihinen I., Honohan A. M. and Lin S. P., `Image of absolute instability in a liquid jet', Phys. Fluids 9, (1997) 3117.

[Whitaker , 1994]Whitaker N., `Some numerical methods for the Hele-Shaw equations', J. Comp. Phys. 111 (1994) 81-88.

[Wilson , 1975] Wilson S. D. R., `A note on the measurement of dynamic contact angles', J. Colloid Interface Sci. 51(3), (1975) 532-534.

[Xu et al , 2019] Xu L., Szymczak P., Toussaint R., Fleckøy E. G. and Mâløy, `Experimental Observation of Dissolution Finger Growth in Radial Geometry', Front. Phys. 10(96) (2019) 1-10.

[Yue et al , 2004]Yue P., Feng J. J., Liu C. and Shen J., `A diffuse-interface method for simulating two-phase flows of complex fluids', J. Comp. Phys. 515 (2004) 293-397.

[Zimmerman & Homsy , 1991] Zimmerman W. B. & Homsy G. M. `Nonlinear viscous fingering in miscible displacement with anisotropic dispersion', Phys. Fluids A 3(8) (1991) 1859-1872.

[Zimmerman & Homsy , 1992] Zimmerman W. B. & Homsy G. M. `Three dimensional viscous fingering: A numerical study', Phys. Fluids A 4(9) (1992) 1901-1914.

[Zimmerman & Homsy , 1992] Zimmerman W. B. & Homsy G. M. `Viscous fingering in miscible displacements: Unification of effects of viscosity contrast, anisotropic dispersion and velocity dependence of dispersion on nonlinear finger propagation', Phys. Fluids A 4(11) (1992) 2348-2359.