#  An In-Context Schema Understanding Method for Knowledge Base Question
Answering

Yantao Liu, Zixuan Li∗, Xiaolong Jin∗, Long Bai, Saiping Guan,
Jiafeng Guo Xueqi Cheng
School of Computer Science and Technology, University of Chinese Academy of
Sciences;
CAS Key Laboratory of Network Data Science and Technology,
Institute of Computing Technology, Chinese Academy of Sciences.
{liuyantao22s, lizixuan, jinxiaolong, bailong18b<EMAIL_ADDRESS>
{guojiafeng<EMAIL_ADDRESS>
###### Abstract

The Knowledge Base Question Answering (KBQA) task aims to answer natural
language questions based on a given knowledge base. As a kind of common
methods for this task, semantic parsing-based ones first convert natural
language questions to logical forms (e.g., SPARQL queries) and then execute
them on knowledge bases to get answers. Recently, Large Language Models (LLMs)
have shown strong abilities in language understanding and may be adopted as
semantic parsers in such kind of methods. However, in doing so, a great
challenge for LLMs is to understand the schema of knowledge bases. Therefore,
in this paper, we propose an In-Context Schema Understanding (ICSU) method for
facilitating LLMs to be used as a semantic parser in KBQA. Specifically, ICSU
adopts the In-context Learning mechanism to instruct LLMs to generate SPARQL
queries with examples. In order to retrieve appropriate examples from
annotated question-query pairs, which contain comprehensive schema information
related to questions, ICSU explores four different retrieval strategies.
Experimental results on the largest KBQA benchmark, KQA Pro, show that ICSU
with all these strategies outperform that with a random retrieval strategy
significantly (from 12% to 78.76% in accuracy).

##  1 Introduction

The Knowledge Base Question Answering (KBQA) task has been a challenging
problem in the Natural Language Processing (NLP) field, which aims to answer
natural language questions based on given knowledge bases (KBs). A kind of the
common methods for this task is semantic parsing-based methods, where natural
language questions are first converted into logical forms, such as SPARQL
queries, and then executed on knowledge bases to retrieve answers. As shown in
Figure 1, the question, “Among the constitutional monarchies with an inflation
rate less than 6500 percentage, which one has the smallest unemployment
rate?”, is parsed into corresponding SPARQL query, and then executed on the KB
to obtain its answer “Eswatini”.

Figure 1: An illustrative disgram of LLM-based KBQA.

Recently, LLMs (e.g., ChatGPT) show the impressive natural language
understanding abilities and performs well on a variety of NLP tasks Hendrycks
et al. (2021b, a). Moreover, LLMs also demonstrate strong abilities to
generate formal language, such as programming language Chen et al. (2021);
Nijkamp et al. (2023b, a). Therefore, we would like to know that whether LLMs
can be used as a semantic parser for KBQA by combining the above two
abilities. Previous work Tan et al. (2023) claims that LLMs are not capable of
generating accurate SPARQL queries because they are not aware of the schemes
about the KB (e.g., entity and relation types). In this paper, we intend to
re-check this issue.

Actually, enabling LLMs to generate accurate SPARQL queries has two steps.
First, LLMs should align the semantic elements in the questions to the
elements in the KB. Second, LLMs compose the aligned elements to generate
proper SPARQL queries. The first step challenges LLMs majorly because the LLMs
can not align some semantic elements to the schema-related elements in the KB.
For example, as shown in Figure 1, the schema-related elements, such as
relation “<pred:unit>” should be aligned by the corresponding semantic
elements “6500 percentage” in the question. One straightforward approach is
attaching all schema-related elements in the prompt. However, this approach is
not suitable since the size of such elements is usually quite large.
Considering that each question only relates to a few schema-related elements
and the annotated queries contain the corresponding schema-related elements
about the question, we can let LLMs be aware of such elements via using a few
annotated examples (e.g., question-SPARQL pairs) over the given KB as prompts.

Motivated by this, we propose an In-Context Schema Understanding (ICSU)
method, which leverages LLMs to conduct semantic parsing for KBQA. First, ICSU
explores three different example retrieval strategies based on raw questions,
anonymized questions, SPARQL queries, and further propose a hybrid strategy
that mixed above three. Follow that, ICSU adopts the In-context Learning
mechanism to instruct LLMs to generate SPARQL queries with examples retrieved
via above four strategies. Comparing with a random retrieval strategy, ICSU
shows a significant improvement in accuracy (from 12% to 70%+ on text-
davinci-003), which demonstrates the effectiveness of the proposed model.

##  2 Related Work

###  2.1 Knowledge Base Question Answering

KBQA task methodologies are primarily divided into two categories: semantic
parsing-based and information retrieval-based Herzig et al. (2021); Nie et al.
(2022); Miller et al. (2016); Saxena et al. (2020); Schlichtkrull et al.
(2018); Zhou et al. (2018); Qiu et al. (2020). Semantic parsing-based methods,
such as STAGG Yih et al. (2015), convert natural language questions into
formal language queries for execution on knowledge bases, offering precise
answers and interpretability. However, these methods, recently formulated as a
Seq2Seq generation task within an encoder-decoder framework, face semantic and
structural discrepancies between natural language questions and formal
language queries Wu et al. (2021); Shin et al. (2021); Xu et al. (2020); Wu et
al. (2021). In contrast, information retrieval-based methods form a question-
specific graph from the knowledge base, ranking entities based on their
question relevance. Compared with the latter one, semantic parsing-based
methods can provide more accurate answers and better interpretability.

###  2.2 Large Language Models and In-Context Learning

Large Language Models, such as GPT-$3$ Brown et al. (2020), LLaMA Touvron et
al. (2023), Taori et al. (2023b) have become noteworthy due to their
impressive performance on a variety of downstream tasks without the need for
fine-tuning. In-Context learning is a key technique for LLMs that allows LLMs
to make predictions relying on contexts augmented with a few examples.

Recently, some methods adopt LLMs in KBQA tasks. Tan et al. (2023) evaluates
the performance of LLMs on KBQA tasks via directly using LLMs as knowledge
bases. Baek et al. (2023) searches the relevant facts and attaches them to the
input question from the knowledge base to enhance the ability of LLMs on KBQA.

##  3 The Proposed ICSU method

Models | LLaMA-7B | Alpaca-7B | ChatGPT | text-davinci-003
---|---|---|---|---
ICSU (w.o. Examples) | 0.00 | 0.00 | 0.00 | 0.00
ICSU (Random) | 1.21 | 2.87 | 5.01 | 12.72
ICSU (Raw) | 14.90 | 40.80 | 68.97 | 71.58
ICSU (Anonymized) | 20.50 | 52.27 | 73.32 | 75.32
ICSU (SPARQL) | 23.50 | 54.82 | 73.66 | 76.37
ICSU (Hybrid) | 21.51 | 52.88 | 76.16 | 78.76
Table 1:  The experimental results on KQA Pro.

###  3.1 LLM Prompting for KBQA

To enable ICSU to generate accurate SPARQL queries, we design prompt $x$ for
each question. Specifically, $x=(i,e,q)$, contains three elements: 1) A task
instruction $i$ that provides a brief overview of the semantic parsing task in
KBQA; 2) An example set $e$ containing several examples that provide schema
information of the given question; 3) An input question $q$ that requires LLMs
to provide the corresponding SPARQL query.

The key problem for ICSU is how to retrieve proper examples that contain
comprehensive schema information related to questions. In the following, we
will introduce the details of the proposed there example retrieval strategies.

###  3.2 Example Retrieval Strategies

ICSU investigates three different strategies for retrieving examples from the
annotated question-query pairs. Furthermore, ICSU also contains a hybrid
strategy with the combination of the three strategies.

####  3.2.1 Raw-Question Based Strategy

Driven by the intuition that the more similar an example is, the more schema
information it can provide, ICSU computes similarities between the input
question $q$ and the annotated questions (i.e., questions in the training
set), and retrieves the top-k annotated question-query pairs with the highest
similarity scores. Specifically, ICSU adopts MPNet Song et al. (2020), a
sentence embedding method, to get the embeddings of the questions and
calculates the similarities based on negative Euclidean score.

####  3.2.2 Anonymized-Question Based Strategy

Specific entities usually cannot be shared across different questions.
Besides, schema information is more about the entity types and relation types
rather than the specific entities. Therefore, we propose to anonymize
questions by replace the entities with their corresponding entity types. More
specifically, we use FLERT Schweter and Akbik (2020), an 18-class NER model,
to recognize types of entities. For different entities of the same type in the
question, a number suffix is added for distinction.

For example, given a question “Which movie is shorter, The Greatest Story Ever
Told or Rhinestone?”, it will be transformed to "Which movie is shorter,
[WORK_OF_ART_0] or [WORK_OF_ART_1]?". The similarity between the anonymized
questions is then calculated in the same manner as Section 3.2.1.

####  3.2.3 SPARQL Based Strategy

The above two strategies are based on the natural language questions but not
SPARQL, where the annotated queries are not fully utilized. Since the
corresponding SPARQL of the input question is not available, we propose to
generate a draft SPARQL query for the input question 111The draft SPARQL query
can be generated using the method described above. We use anonymized-question
based strategy to ensure better quality of draft SPARQL queries. and then
retrieve the examples according the similarity between SPARQL queries. The
similarity is then calculated in the same manner as Section 3.2.1.

####  3.2.4 Hybrid Strategy

Each of these three strategies has different retrieval preferences. To
increase the diversity of the retrieved examples, we combine these three
strategies to form a hybrid strategy. In particular, the order in which we
combine the three strategies is as follows: the anonymized-question based
strategy, the SPARQL based strategy, and the raw-question based strategy.

##  4 Experiment

###  4.1 Datasets and Metric

KQA Pro Cao et al. (2022) is selected as the benchmark to evaluate the
performance of ICSU. As the largest KBQA benchmark, KQA Pro contains 117,790
natural questions along with their corresponding SPARQL queries. We adopt
execution accuracy as our metric following the work Nie et al. (2022), which
is determined by whether the generated logical form queries return correct
answers. For queries with multiple valid answers, we require the execution
results to exactly match all ground-truth answers.

###  4.2 Experimental Setup

Our experiments are conducted using four different LLMs: LLaMA-7BTouvron et
al. (2023), Alpaca-7BTaori et al. (2023a), text-davinci-003, and ChatGPT
Ouyang et al. (2022). We adopt ICSU without any examples and with randomly
retrieved examples as our baselines. The number of In-Context examples is
experimentally set to $6$.

###  4.3 Experimental Results

As illustrated in Table 1, ICSU (w.o. Examples) and ICSU (Random) denote ICSU
without examples and with randomly retrieved examples, respectively. ICSU
(Raw), ICSU (Anonymized), ICSU (SPARQL) and ICSU (Hybrid) denote the proposed
ICSU with example retrieval strategies mentioned in Section 3.2. The results
of ICSU (w.o. Examples) demonstrate that LLMs totally failed to be a semantic
parser because they are not aware of any schema information of the given
questions. All the four strategies outperform ICSU (Random) drastically.
Notably, the proposed strategies with text-davinci-003 achieve improvement of
more than 58.86%, which verifies the effectiveness of the proposed ICSU.
Moreover, ICSU (Anonymized) demonstrates a significant performance improvement
compared to the ICSU (Raw), which verifies that the entity-specific
information in the questions misleads the model to select entity-specific
examples instead of examples providing more useful schema information.
Furthermore, ICSU (SPARQL) also offers satisfactory improvement compared to
the ICSU (Anonymized). ICSU (SPARQL) utilizes the useful information in
annotated queries more effectively and thus achieves better performance.

Compared to other LLMs, text-davinci-003 achieves the best result with other
strategies. Considering that text-davinci-003 is a hundred billion LLM
compared to other LLMs, this result may indicate that increasing the model
scale might enhance the semantic parsing ability of LLMs. ICSU (Hybrid) gets
the best performance with ChatGPT and text-davinci-003, while underperforms
ICSU (SPARQL) with LLaMA-7B and Alpaca-7B. ICSU (Hybrid) increase the
diversity of examples and powerful LLMs, like ChatGPT or text-davinci-003,
achieve the best results due to their abilities to utilize more rich schema
information from diverse examples. However, according to the results of ICSU
(raw), LLaMA-7B and Alpaca-7B cannot capture the useful schema information
from these entity-specific examples from the raw-question based strategy. The
performance of ICSU (Hybrid) with LLaMA-7B and Alpaca-7B is also influenced by
such less helpful examples.

However, the significant performance gap between OpenAI’s LLMs (text-
davinci-003 and ChatGPT) and those in the Open Source Community (LLaMA-7B,
Alpaca-7B) cannot be ignored. It indicates that the open source community
still has a long way to go to match OpenAI’s pace.

###  4.4 Detail Analysis

Figure 2: Detail analysis on the correlation between relation coverage and
accuracy.

To study how the schema information contributes to the final results, we
conduct detailed analysis of ICSU with ChatGPT on validation set. We use the
relation coverage (the percentage of relations in the ground truth SPARQL are
mentioned in the retrieved examples) to reflect the question-related schema
information contained in examples. Specifically, we conduct statistic on
(relation coverage, accuracy) pairs of the models with four proposed retrieval
strategies under $k(k=1,...,6)$ examples and plot them on Figure 2. It can be
observed that the accuracy increases when the relation coverage gets larger,
which verifies the motivation that the ability of LLMs as the semantic parser
gets more powerful along with the increasing of schema information contained
in the retrieved examples.

##  5 Conclusion

This research has proven the efficacy of In-Context Schema Understanding
(ICSU) in augmenting Large Language Models’ (LLMs) proficiency for Knowledge
Base Question Answering (KBQA), thus providing an innovative strategy for
semantic parsing that enables LLMs to generate SPARQL queries from examples
and better understand knowledge base schemas. A significant leap in accuracy,
from 12% to over 70%, was achieved by implementing three retrieval strategies:
raw-question based, anonymized-question based, SPARQL based and the hybrid
strategy, outdoing the random retrieval one and showing promising prospects
for incorporating LLMs into KBQA.

## Limitations

Our study investigated the application of Large Language Models (LLMs) as
semantic parsers in Knowledge Based Question Answering (KBQA) tasks using the
In-Context Schema Understanding (ICSU) method. Experimental findings indicated
performance enhancements, though certain inherent constraints should be
acknowledged. A significant constraint is the dependency on examples derived
from annotated question-query pairs for in-context learning, which improved
the LLMs’ ability to create SPARQL queries but assumes the availability of
high-quality, annotated pairs. This presumption could hinder the model’s
generalization capabilities, particularly in situations where such pairs are
either lacking or not diverse enough to encapsulate a spectrum of potential
queries.

## References

  * Baek et al. (2023) Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023.  Knowledge-augmented language model prompting for zero-shot knowledge graph question answering.  _arXiv preprint arXiv:2306.04136_.
  * Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_.
  * Cao et al. (2022) Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022.  KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_ , pages 6101–6119. Association for Computational Linguistics.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.  Evaluating large language models trained on code.
  * Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a.  Aligning ai with shared human values.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Herzig et al. (2021) Jonathan Herzig, Peter Shaw, Ming-Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. 2021.  Unlocking compositional generalization in pre-trained models using intermediate representations.  _arXiv preprint arXiv:2104.07478_.
  * Miller et al. (2016) Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016.  Key-value memory networks for directly reading documents.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 1400–1409. The Association for Computational Linguistics.
  * Nie et al. (2022) Lunyiu Nie, Shulin Cao, Jiaxin Shi, Jiuding Sun, Qi Tian, Lei Hou, Juanzi Li, and Jidong Zhai. 2022.  GraphQ IR: Unifying the semantic parsing of graph query languages with one intermediate representation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5848–5865, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Nijkamp et al. (2023a) Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023a.  Codegen2: Lessons for training llms on programming and natural languages.  _ICLR_.
  * Nijkamp et al. (2023b) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023b.  Codegen: An open large language model for code with multi-turn program synthesis.  _ICLR_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Qiu et al. (2020) Yunqi Qiu, Yuanzhuo Wang, Xiaolong Jin, and Kun Zhang. 2020.  Stepwise reasoning for multi-relation question answering over knowledge graph with weak supervision.  In _WSDM ’20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020_ , pages 474–482. ACM.
  * Saxena et al. (2020) Apoorv Saxena, Aditay Tripathi, and Partha P. Talukdar. 2020.  Improving multi-hop question answering over knowledge graphs using knowledge base embeddings.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 4498–4507. Association for Computational Linguistics.
  * Schlichtkrull et al. (2018) Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks.  In _The Semantic Web - 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings_ , volume 10843 of _Lecture Notes in Computer Science_ , pages 593–607. Springer.
  * Schweter and Akbik (2020) Stefan Schweter and Alan Akbik. 2020.  Flert: Document-level features for named entity recognition.
  * Shin et al. (2021) Richard Shin, Christopher H. Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021.  Constrained language models yield few-shot semantic parsers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 7699–7715. Association for Computational Linguistics.
  * Song et al. (2020) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020.  Mpnet: Masked and permuted pre-training for language understanding.  _Advances in Neural Information Processing Systems_ , 33:16857–16867.
  * Tan et al. (2023) Yiming Tan, Dehai Min, Yu Li, Wenbo Li, Nan Hu, Yongrui Chen, and Guilin Qi. 2023\.  Evaluation of chatgpt as a question answering system for answering complex questions.  _arXiv preprint arXiv:2303.07992_.
  * Taori et al. (2023a) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023a.  Alpaca: A strong, replicable instruction-following model.  _Stanford Center for Research on Foundation Models._
  * Taori et al. (2023b) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023b.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wu et al. (2021) Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, and Xunliang Cai. 2021.  From paraphrasing to semantic parsing: Unsupervised semantic parsing via synchronous semantic decoding.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 5110–5121. Association for Computational Linguistics.
  * Xu et al. (2020) Silei Xu, Sina J. Semnani, Giovanni Campagna, and Monica S. Lam. 2020.  Autoqa: From databases to QA semantic parsers with only synthetic training data.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 422–434. Association for Computational Linguistics.
  * Yih et al. (2015) Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015.  Semantic parsing via staged query graph generation: Question answering with knowledge base.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1321–1331, Beijing, China. Association for Computational Linguistics.
  * Zhou et al. (2018) Mantong Zhou, Minlie Huang, and Xiaoyan Zhu. 2018.  An interpretable reasoning network for multi-relation question answering.  In _Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018_ , pages 2010–2022. Association for Computational Linguistics.