0 Research

# Exploring Effectiveness of Explanations for Appropriate Trust: Lessons from
Cognitive Psychology

Ruben S. Verhagen e-mail<EMAIL_ADDRESS>*Author sharing first
authorship. Siddharth Mehrotra e-mail<EMAIL_ADDRESS>*Author sharing
first authorship. Mark A. Neerincx<EMAIL_ADDRESS>Secondoary
affiliation: TNO, Soesterberg, The Netherlands Catholijn M. Jonker
<EMAIL_ADDRESS>Secondoary affiliation: Leiden University, The
Netherlands Myrthe L. Tielman<EMAIL_ADDRESS>Delft University
of Technology, The Netherlands

###### Abstract

The rapid development of Artificial Intelligence (AI) requires developers and
designers of AI systems to focus on the collaboration between humans and
machines. AI explanations of system behavior and reasoning are vital for
effective collaboration by fostering appropriate trust, ensuring
understanding, and addressing issues of fairness and bias. However, various
contextual and subjective factors can influence an AI system explanation’s
effectiveness. This work draws inspiration from findings in cognitive
psychology to understand how effective explanations can be designed. We
identify four components to which explanation designers can pay special
attention: perception, semantics, intent, and user & context. We illustrate
the use of these four explanation components with an example of estimating
food calories by combining text with visuals, probabilities with exemplars,
and intent communication with both user and context in mind. We propose that
the significant challenge for effective AI explanations is an additional step
between explanation generation using algorithms not producing interpretable
explanations and explanation communication. We believe this extra step will
benefit from carefully considering the four explanation components outlined in
our work, which can positively affect the explanation’s effectiveness.

Human-centered computingVisualizationVisualization techniques; Human-centered
computingVisualizationVisualization design and evaluation methods

##  1 Introduction

Humans and Artificial Intelligence (AI) systems are increasingly collaborating
on tasks ranging from the medical to the financial domain. For such human-
machine collaboration to be effective, mutual understanding and trust are of
paramount importance [15, 18, 31]. AI explanations are a crucial and powerful
way to increase human understanding and trust in the system. Explanations can
help in explaining the decisions and inner workings of the ”black-box” data-
driven machine learning algorithms and actions & reasoning of goal-driven
agents [2, 11, 27, 10, 19, 25].

Unfortunately, AI systems often lack transparency and explainability,
providing a broad call for explainable AI (XAI). For example, EU GDPR requires
organizations that deploy AI systems to provide relevant information to
affected people about the inner workings of the algorithms [37]. Furthermore,
in addition to helping people understand AI systems, AI explanations can also
contribute to identifying and addressing issues of fairness and bias, which
are difficult to do without. Explanations can also help form appropriate trust
in the AI system. For example, Dodge et al. [8] pointed out that when people
trust the explanation, they are more likely to trust the underlying ML system.

A common challenge in popular explanation methods such as LIME (Local
Interpretable Model-agnostic Explanations) by Ribeiro et al. [29], SHAP
(Shapley Additive Explanations) by Lundberg & Lee [24], and global-local
explanation by Lundberg et al. [23] is the effectiveness of the explanation to
help users calibrate their trust and increase understanding of the
explanation. Various aspects can impact the effectiveness of AI explanations,
such as the user’s domain, system expertise or cognitive and perceptual
biases. However, this list is far from conclusive. Therefore it is crucial to
investigate the following two research questions:

  1. 1.

What should be the content of the explanation to make it effective?

  2. 2.

How can the visual delivery or design of the explanation make it effective?

##  2 Explanation Effectiveness

In this work, we take inspiration from cognitive psychology, design, and data
visualization techniques to explore ways how effective explanations can be
designed. A large amount of work in cognitive psychology focuses on
explanations in human-human interactions, what makes them effective and how
context can affect the same. For example, Lombrozo [22] provides two
properties of the structure of explanations that helps in reasoning, (1)
explanations accommodate novel information in the context of prior beliefs,
and (2) do so in a way that fosters generalization. The author showcase that
explanations provides a unique window onto the mechanisms of learning and
inference in human reasoning.

David B. Leake in his book “Evaluating Explanations” from cognitive psychology
theories has described how context involving both explainer beliefs and goals
can help in deciding an explanation’s goodness [21]. Similarly, Khemlani et
al. shows how mental models represent causal assertions, and how these models
underlie deductive, inductive, and abductive reasoning yielding effective
explanations [17]. Since explanations determine how humans understand the
world in fundamental terms, Tworek and Cimpian helps in exploring human biases
towards the inheritance of judgments in people’s explanations over socio-moral
understanding [35].

The previously mentioned works and recent research in human-AI interaction
[32, 26, 43, 36, 12] helps us in exploring our two research questions based on
four components (Perception, Semantics, Intent, and User & Context). These
components are uniquely visible at the intersection of studies in human-AI
interaction and cognitive psychology. We will now describe these components in
detail.

###  2.1 Perception

Perception in XAI can refer to the set of mental processes we use to make
sense of a given explanation by an AI system. Perception is also frequently
related to the modality in which explanation is presented and how our brain
interprets it. For instance, example-based explanations are useful in cases
where it’s hard to explain AI reasoning. Here, developers can create an
explanation in the form of a real-life example or a story to depict the
system’s processing, in line with how we as humans are primed to relate to
real-life examples or episodes. Empirical research has demonstrated that when
explanations take the form of a story, they can help humans in decision making
[16]. According to Rumelhart [30], stories often contain initiating events,
goals, actions, consequences, and accompanying states in a particular causal
configuration. Knowing the structure of stories can allow humans to form
appropriate trust concerning the perception of the explanation.

For saliency maps, categorical model confidence visualizations, n-best
visualizations, and related ways of visual explanations, we propose that it is
important to consider how the human brain is organized to see structures,
logic, and patterns. Our brain organization helps us as humans to make sense
of the world. A seminal work by Wertheimer [39] introduced seven Gestalt
principles of visual perception in the form of heuristics. We now briefly
describe those principles in relation to visual explanations in XAI:

  1. 1.

figure-ground: visualization of the explanation can display objects as being
either in the foreground or the background. For example, an identified object
with a higher confidence score can be displayed in the foreground, leaving
every other object in the background.

  2. 2.

similarity: clustering of similar objects can be shown to the user such that
the user can easily group similar objects together. For example, we can use a
variety of design elements, like color and organization, to establish similar
groups.

  3. 3.

proximity: for pixel-wise importance in saliency map of human face detection,
calculated using integrated gradients; explanations can focus on (a) the areas
around the eyes, (b) lines anywhere on the face, and (c) regions around the
mouth and nose. This relative nearness of the objects can strongly influence
the user’s understanding of the explanation.

  4. 4.

common region: the principle of common region states that when objects are
located within the same closed region, humans perceive them as being grouped
together. Therefore, for any two distinct objects, a visual explanation can
put them in different regions and vice-versa.

  5. 5.

continuity and closure: continuity refers to the arrangement of elements while
closure refers to identification of recognizable patterns. We can apply these
principles in explanations presented in a commonly used dashboard. For
example, these principles explain why only two axes, rather than a full
enclosure, are required on a graph to define the space in which the data
appears. Also, it can become obvious which groups belong to the subgroup when
the hierarchy is expanded.

  6. 6.

focal point: the premise of the focal point principle is that anything
visually outstanding first captures and retains the viewer’s attention. In
terms of explanations, assigning each point a score reflecting its
contribution to the ML model-recognition loss can follow this principle. The
saliency map can then explicitly explain which points are the key to model
recognition, as explained by Zheng et al. [42].

Zena O’Connor’s contemporary work on colour, and contrast in combination with
gestalt theories provide the effectiveness of visual communication design in
graphical design, which can be helpful for the XAI community [28]. For
example, the author examined the role of colour and contrast within the
context of Gestalt theories of perception. The author also showcased how
colour and contrast not only help to distinguish contours, detail, and depth,
but they also help to attract and divert attention, thereby clearly
delineating key areas of text. This result can help design explanations when
multiple models are used to describe a decision by an AI system.

###  2.2 Semantics

Semantics in XAI refers to how AI system designers can convey the meaning in
language to their users. This meaning of the language can help users get a
grip on what the expressions of a natural language contribute to the overall
process of interpretation. Cann et al. [6] provide three basic types of sense
relations to clearly communicate the meaning of the language:

  1. 1.

Synonymy: sameness of sense; for example, X’s pullover is yellow v/s X’s
sweater is yellow.

  2. 2.

Hyponymy: sense inclusion; for example, X wants a hamburger v/s X wants to eat
a hamburger.

  3. 3.

Antonymy: oppositeness in sense; for example, This water is cold v/s This
water is not hot.

It becomes essential to be aware of these sense relations when designing
explanations for AI systems because as soon as AI system designers start
probing what concept of meaning they should articulate, they may slip away
into a quag of open-endedness.

Researchers have focused on resolving linguistic ambiguities in visual data
communication. For example, Setlur et al. [33] provide heuristics to resolve
partial utterances based on syntactic and semantic constraints. Similarly, Law
et al. [20] have proposed ways such as encouraging skepticism and open-
mindedness to reduce the illusion of causality when using question-answering
systems. Similarly, Gaba et al. [9] have provided guidelines for designing
natural language interfaces and recommendation tools to better support natural
language comparisons in visual communication. These guidelines include use of
basic charts as reasonable responses for a variety of comparisons, providing
necessary information useful for the comparisons, and exposing the provenance
of how implicit entities are such as “cheap” and “best-selling”.

Humans often adapt cognitive approaches to explanations where any ambiguity
and vagueness can affect human-AI communication. This fuzziness in
communication can result in distrust in the AI system and can be disastrous in
high-risk scenarios where lives can be at stake. Taken into consideration the
fuzziness of explanations, Nauta et al. [26] provide a summary of Co-12
properties on explanation quality. The properties related to semantics (and
perception) fall in the category of content focusing on correctness,
completeness, contrastivity, continuity, consistency, and covariate
complexity. These properties provide an aggregated view of what to evaluate in
an explanation. We believe that an explanation’s cohesiveness ties all these
properties together. In cohesiveness, one can focus on creating semantically
resonant explanations that enrich human understanding. When an explanation is
cohesive it can also help in preserving users’ mental models and memory. For
example, providing infographic information with confidence scores based on
reasoning with facts from verified and trusted sources can help make an
explanation cohesive. Furthermore, exploring concepts around scaling or hiding
information can create the right balance for explanation effectiveness.

Apart from focusing on effective explanations qualities, it is also essential
to consider how data literacy can impact the perceived effectiveness of an
explanation. Wolf’s and Gottwald’s popular work on Tales of Literacy sheds
light on how digital literacy is differently represented in the human brain
[40]. The authors present ”vocabulary” of language and reading development to
appreciate better the critical importance of specific literacy development
aspects with digital systems worldwide. On a similar note, Setlur and Cogley
[32] provide an overview of data literacy and how it forces us to think about
how everyone navigates the world. Authors call out to researchers working in
this domain to focus on cultural differences in data literacy and how people
around the globe can interpret an explanation in multiple ways.

Any recommendation in form of an explanation by an AI system is based on a
probabilistic model. One of the challenges for effective explanations is
identifying a baseline for communicating these probabilities. Based on the
work by Wolf [41], we propose that ability of neural networks in the brain
helps and hinders humans in their attempt to read and process information
related to probabilities. Since our way of processing information differs at
the individual level, so does our understanding of an explanation. Therefore,
it becomes vital to consider diverse populations and the moral components of
the design and use of visualizations for explanations in an easy to understand
manner [7].

We believe that AI systems will not be explainable without first addressing
the scarcity of semantics in explanations. Browne and Swift [4] highlights
that “producing satisfactory explanations for deep learning systems will
require that we find ways to interpret the semantics of hidden layer
representations in deep neural networks”. There have been a few attempts
looking at semantics in explanations, such as by Jin et al. [14] exploring
hierarchical visualization of compositional semantics to help users create
trust in deep neural networks. Similarly, Jacob et al. [13] investigating how
users can guide the generation of counterfactual explanations by specifying a
set of semantic regions of the query image. Although both reported works show
promising results, extracting semantic explanations from hidden units is far
from a solved problem.

###  2.3 Intent

Figure 1: In this snapshot of a user study, an AI agent provides an estimation
of calories of the dish (shown in bottom right) out of four available options
(shown in bottom left). Here the estimation of calories is accompanied by
confidence scores of each identified ingredient(s) in a form of a table. Also,
the AI agent provides both textual and visual explanations for its reasoning
of selecting an option for calories estimation.

The intent in XAI refers to the purpose of the explanation, ideally balancing
semantics with goals. Therefore, intent can heavily influence the perception
and semantics components when designing explanations. For example,
explanations aimed at convincing users to engage in a healthier lifestyle
should take different forms than explanations aimed at improving firefighters’
situation in high stake decisions. For recommending food options to a user to
keep a healthy lifestyle, explanations can be designed to (a) convince users
with confidence estimates of the nutrition of food, (b) predict the outcome of
eating the healthy food, or (c) showcase low confidence when a system can’t
recommend a healthy diet based on available food option to foster appropriate
trust. In contrast, explanations aimed at improving the situation and moral
awareness of firefighters can explain the potential consequences of the most
important situational features to consider when making a decision.

From the literature of cognitive psychology, JS Bruner [5] sheds light on
Macfarlane’s work, where authors studied how mothers are irresistibly imputing
intent to the cries, gestures, expressions, and postures of newborns. As
communication presupposes intent in communicating an objective, one can learn
so much just by understanding a mother’s intent to understand her child.
Bruner also highlights that in interpreting the infant’s communicative intent
- correctly or incorrectly - the mother has a rich variety of cues to use, and
so too for the child. Relating this to the XAI context, we propose that the AI
system designers need to adapt their system designs to match the user’s intent
based on available cues, similar to how a mother does for her child, as
outlined by Bruner.

Neerincx et al. and Anjomshoae et al. [27, 2] provide three explanation phases
to convey intent of an explanation: generation, communication, and reception.
Explanation generation mostly concerns with the explainability methods used to
construct relevant elements and information to share, such as SHAP or LIME
explainers. Explanation communication refers to the form and content and
reception relates to how the receiving user utilizes and understands the
explanation. In the work by Miller [25], the author did emphasize the crucial
role of intention in explanations, arguing that determining the goal of an
explanation is key to providing a good explanation. We resonate with this
statement and propose that the current field of XAI can pay more attention to
explanation intent. Specifically, intent can guide explanation communication
(i.e., content/semantics and form/perception), which can show its benefits in
explanation reception.

###  2.4 User and Context

We believe that defining explanation intent includes considering both user and
context. Recent works emphasize the importance of personalized and context-
aware explanations [2, 3, 25], as it is believed that they improve explanation
effectiveness. For example, the same explanation used to convince user A to
live healthier will likely not work if user B has a completely different value
profile or personality. Likewise, when a firefighter is under extreme time
pressure, visual explanations are probably much more effective than verbal
ones. Taken together, we believe that simply defining explanation intent as to
inform, convince, or support will not suffice. Instead, explanation designers
can define the goal by combining explanation intent, receiver/user, and
context (e.g., convince conservative user X to start exercising more).

##  3 Use case example

Wang and Yin provided three desiderata for designing effective AI explanations
[38]. These desiderata include (a) designing explanations to improve people’s
understanding of the AI model, (b) helping people recognize the uncertainty
underlying an AI prediction, and (c) empowering people to trust the AI
appropriately. In the following example of an AI explanation, we follow these
desiderata together with the components of explanations that we discussed
earlier.

Our use case example showcases a snapshot of the “estimating calories on a
plate” user study, see Figure 1. We designed the task around nutrition as an
approachable domain for everyone. In this user study, an AI agent helps the
user to estimate the food calories with the help of explanations. The AI agent
estimates the calories of a dish shown in the right corner of the figure. The
agent also explains its confidence scores (based on an AI-driven food
recognition model). In addition, the categorical visualization for each
ingredient is displayed as a form of visual explanation. The user must decide
among the four options to select the closest calorie count based on the AI
agent’s suggestion and get a +10 score for a correct selection, whereas a
wrong selection costs -10 points.

In the above use case, the confidence scores of ingredients are generated
using a machine learning classifier in real-time. The visual explanations are
based on categorical visualization (Figure 1, right) inspired by the example
provided in the Google PAIR guidebook [1]. These visual explanations were
designed keeping in mind the four components discussed in this paper as
follows:

  1. 1.

In our use case, perception refers to different categorizations such as best,
good, or unsure match by providing clear meaning of the categorizations to the
user.

  2. 2.

For intent, we decided to use a single image for each categorization to convey
the actual semantics we perceived as system designers. The text-based
explanations in the figure are handcrafted following the notion of situation
vignettes [34]. Here every explanation starts with the intent the AI agent
wants to convey, followed by the perception of the authority (in this case
United Nations Food & Nutrition website) on which the suggestion is based.

  3. 3.

The next part of the explanation is to clearly deliver the semantics of the
reasoning behind the suggestion to the user in non-ambiguous language. For
semantics, we adopted Rumelhart’s [30] guidelines and followed gestalt
heuristics for designing visual explanation.

  4. 4.

The user and context are also taken into account by relating the explanation
with the closest dish in the database and providing logic-based reasoning that
users can adhere to. Furthermore, the combination of visual and textual
explanation is designed to elicit an appropriate amount of trust that the AI
agent can foster, distinct from promoting the trust in the agent.

Our method of showcasing explanations is a way to incorporate the components
that we have discussed in this work. This approach has known limitations, such
as scaling the explanations with complex data or crafting these explanations
in real-time. We believe that these limitations can be overcome with further
advancements in the field of XAI with methods such as ensemble ML, feature-
based, rule-based, and training-data based explanations.

##  4 Conclusion

Previous work has shown that the efficiency of human-AI interaction can be
improved when AI systems explain their behavior and reasoning. However, how to
create explanations that are both intuitive and easily understandable by users
is a challenging task. In this work, we have discussed four components of
designing effective explanations based on insights from cognitive psychology.
Furthermore, we provided an example to incorporate those components in
creating a textual and visual explanation for a use-case example. With this
work, we propose the need to provide emphasis on these components to make
explanations effective. We also showed how communicating results in an
interpretable manner needs to be as intuitive as dialogue. However, we must
note that explanations can increase the chance that humans will accept the
AI’s recommendation, regardless of its correctness. Therefore, explanations
should strive to create appropriate trust rather than helping promote trust in
the AI system. Also, we highlight that it is important to avoid explanations
biasing users towards certain actions or decisions, especially in high-stakes
decision-making tasks.

###### Acknowledgements.

The authors wish to thank the three anonymous reviewers for their constructive
comments and review of the manuscript.

## References

  * [1] Google pair. people + ai guidebook, May 2019.
  * [2] S. Anjomshoae, A. Najjar, D. Calvaresi, and K. Främling.  Explainable agents and robots: Results from a systematic literature review.  In 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13–17, 2019, pp. 1078–1088. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
  * [3] A. B. Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. García, S. Gil-López, D. Molina, R. Benjamins, et al.  Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai.  Information fusion, 58:82–115, 2020.
  * [4] K. Browne and B. Swift.  Semantics and explanation: why counterfactual explanations produce adversarial examples in deep neural networks.  arXiv preprint arXiv:2012.10076, 2020.
  * [5] J. S. Bruner.  From communication to language—a psychological perspective.  Cognition, 3(3):255–287, 1974.
  * [6] R. Cann, R. Kempson, and E. Gregoromichelaki.  Semantics: An Introduction to Meaning in Language.  Cambridge University Press, 2009.
  * [7] M. Correll.  Ethical dimensions of visualization research.  In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–13, 2019.
  * [8] J. Dodge, Q. V. Liao, Y. Zhang, R. K. Bellamy, and C. Dugan.  Explaining models: an empirical study of how explanations impact fairness judgment.  In Proceedings of the 24th international conference on intelligent user interfaces, pp. 275–285, 2019.
  * [9] A. Gaba, V. Setlur, A. Srinivasan, J. Hoffswell, and C. Xiong.  Comparison conundrum and the chamber of visualizations: An exploration of how language influences visual design.  arXiv preprint arXiv:2208.03785, 2022.
  * [10] R. Guidotti, A. Monreale, S. Ruggieri, F. Turini, F. Giannotti, and D. Pedreschi.  A survey of methods for explaining black box models.  ACM Comput. Surv., 51(5), Aug. 2018. doi: 10 . 1145/3236009
  * [11] D. Gunning.  Explainable artificial intelligence (xai).  Defense Advanced Research Projects Agency (DARPA), nd Web, 2(2), 2017.
  * [12] M. Hearst, M. Tory, and V. Setlur.  Toward interface defaults for vague modifiers in natural language interfaces for visual analysis.  In 2019 IEEE Visualization Conference (VIS), pp. 21–25. IEEE, 2019\.
  * [13] P. Jacob, É. Zablocki, H. Ben-Younes, M. Chen, P. Pérez, and M. Cord.  Steex: Steering counterfactual explanations with semantics.  arXiv preprint arXiv:2111.09094, 2021.
  * [14] X. Jin, Z. Wei, J. Du, X. Xue, and X. Ren.  Towards hierarchical importance attribution: Explaining compositional semantics for neural sequence models.  In International Conference on Learning Representations, 2019.
  * [15] M. Johnson and A. Vera.  No ai is an island: The case for teaming intelligence.  AI Magazine, 40(1):16–28, Mar. 2019. doi: 10 . 1609/aimag . v40i1 . 2842
  * [16] D. H. Jonassen.  Designing for decision making.  Educational technology research and development, 60(2):341–359, 2012.
  * [17] S. S. Khemlani, A. K. Barbey, and P. N. Johnson-Laird.  Causal reasoning with mental models.  Frontiers in human neuroscience, 8:849, 2014.
  * [18] G. Klein, D. D. Woods, J. M. Bradshaw, R. R. Hoffman, and P. J. Feltovich.  Ten challenges for making automation a ”team player” in joint human-agent activity.  IEEE Intelligent Systems, 19(6):91–95, 2004. doi: 10 . 1109/MIS . 2004 . 74
  * [19] P. Langley, B. Meadows, M. Sridharan, and D. Choi.  Explainable agency for intelligent autonomous systems.  In AAAI, vol. 17, pp. 4762–4763, 2017.
  * [20] P.-M. Law, L. Y.-H. Lo, A. Endert, J. Stasko, and H. Qu.  Causal perception in question-answering systems.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2021.
  * [21] D. B. Leake.  Evaluating Explanations: A Content Theory.  Psychology Press, 1992.
  * [22] T. Lombrozo.  The structure and function of explanations.  Trends in cognitive sciences, 10(10):464–470, 2006.
  * [23] S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S.-I. Lee.  From local explanations to global understanding with explainable ai for trees.  Nature machine intelligence, 2(1):56–67, 2020.
  * [24] S. M. Lundberg and S.-I. Lee.  A unified approach to interpreting model predictions.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, p. 4768–4777. Curran Associates Inc., Red Hook, NY, USA, 2017.
  * [25] T. Miller.  Explanation in artificial intelligence: Insights from the social sciences.  Artificial Intelligence, 267:1–38, 2019. doi: 10 . 1016/j . artint . 2018 . 07 . 007
  * [26] M. Nauta, J. Trienes, S. Pathak, E. Nguyen, M. Peters, Y. Schmitt, J. Schl ötterer, M. van Keulen, and C. Seifert.  From anecdotal evidence to quantitative evaluation methods: A systematic review on evaluating explainable ai.  arXiv preprint arXiv: 2201.08164, 2022.
  * [27] M. A. Neerincx, J. van der Waa, F. Kaptein, and J. van Diggelen.  Using perceptual and cognitive explanations for enhanced human-agent team performance.  In D. Harris, ed., Engineering Psychology and Cognitive Ergonomics, pp. 204–214. Springer International Publishing, Cham, 2018. doi: 10 . 1007/978-3-319-91122-9_18
  * [28] Z. O’Connor.  Colour, contrast and gestalt theories of perception: The impact in contemporary visual communications design.  Color Research & Application, 40(1):85–92, 2015.
  * [29] M. T. Ribeiro, S. Singh, and C. Guestrin.  ”why should I trust you?”: Explaining the predictions of any classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 1135–1144, 2016.
  * [30] D. Rumelhart.  Understanding and Summarizing Brief Stories.  Basic processes in reading: Perception and comprehension, 1979.
  * [31] E. Salas, D. E. Sims, and C. S. Burke.  Is there a “big five” in teamwork?  Small Group Research, 36(5):555–599, 2005. doi: 10 . 1177/1046496405277134
  * [32] V. Setlur and B. Cogley.  Functional Aesthetics for Data Visualization.  Wiley, 2022.
  * [33] V. Setlur, M. Tory, and A. Djalali.  Inferencing underspecified natural language utterances in visual analysis.  In Proceedings of the 24th International Conference on Intelligent User Interfaces, pp. 40–51, 2019.
  * [34] M. Strack and C. Gennerich.  Personal and situational values predict ethical reasoning.  Europe’s Journal of Psychology, 7(3):419–442, 2011.
  * [35] C. M. Tworek and A. Cimpian.  Why do people tend to infer “ought” from “is”? the role of biases in explanation.  Psychological Science, 27(8):1109–1122, 2016.  PMID: 27485133. doi: 10 . 1177/0956797616650875
  * [36] G. Vilone and L. Longo.  Notions of explainability and evaluation approaches for explainable artificial intelligence.  Information Fusion, 76:89–106, 2021.
  * [37] P. Voigt and A. Von dem Bussche.  The eu general data protection regulation (gdpr).  A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):10–5555, 2017.
  * [38] X. Wang and M. Yin.  Effects of explanations in ai-assisted decision making: Principles and comparisons.  ACM Transactions on Interactive Intelligent Systems (TiiS), 2022\.
  * [39] M. Wertheimer.  Gestalt theory.  1938\.
  * [40] M. Wolf and S. Gottwald.  Tales of Literacy for the 21st Century.  Oxford University Press, 2016.
  * [41] M. Wolf and C. J. Stoodley.  Proust and the squid: The story and science of the reading brain.  Harper Perennial New York, 2008.
  * [42] T. Zheng, C. Chen, J. Yuan, B. Li, and K. Ren.  Pointcloud saliency maps.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1598–1606, 2019.
  * [43] J. Zhou, A. H. Gandomi, F. Chen, and A. Holzinger.  Evaluating the quality of machine learning explanations: A survey on methods and metrics.  Electronics, 10(5):593, 2021.


# Disconnected quark loop contributions to nucleon observables in lattice QCD

A. Abdel-Rehim(a), C. Alexandrou (a,b), M. Constantinou (b), V. Drach (c), K.
Hadjiyiannakou (b), K. Jansen (b,c), G. Koutsou (a), A. Vaquero (a) (a)
Computation-based Science and Technology Research Center, Cyprus Institute, 20
Kavafi Street, Nicosia 2121, Cyprus
(b) Department of Physics, University of Cyprus, P.O. Box 20537, 1678 Nicosia,
Cyprus
(c) NIC, DESY, Platanenallee 6, D-15738 Zeuthen, Germany

###### Abstract

We perform a high statistics calculation of disconnected fermion loops on
Graphics Processing Units for a range of nucleon matrix elements extracted
using lattice QCD. The isoscalar electromagnetic and axial vector form
factors, the sigma terms and the momentum fraction and helicity are among the
quantities we evaluate. We compare the disconnected contributions to the
connected ones and give the physical implications on nucleon observables that
probe its structure.

###### pacs:

11.15.Ha, 12.38.Gc, 12.38.Aw, 12.38.-t, 14.70.Dj

##  I Introduction

Lattice QCD simulations are currently performed near or at the physical value
of the light quark mass. This allows a study of hadron structure that can
provide valuable information for phenomenology and experiment. However, a
number of important observables are computed neglecting disconnected quark
loop contributions. The evaluation of disconnected quark loops is therefore of
paramount importance if we want to eliminate a systematic error inherent in
the determination of hadron matrix elements in lattice QCD. The computation of
disconnected quark loops within the lattice QCD formulation requires the
calculation of the so-called all-to-all or time-slice-to-all propagators, for
which the computational resources required to estimate them with, e.g.
stochastic methods, are much larger than those required for the corresponding
connected contributions. In addition, they are prone to large gauge noise. It
is for these reasons that in most hadron structure studies up to now the
disconnected contributions were neglected, introducing an uncontrolled
systematic uncertainty Alexandrou (2012).

Recent progress in algorithms, however, combined with the increase in
computational power, have made such calculations feasible. On the algorithmic
side, a number of improvements like the one-end trick Boucaud et al. (2008);
Michael and Urbach (2007); Dinter et al. (2012), dilution Bernardson et al.
(1994); Viehoff et al. (1998); O’Cais et al. (2005); Foley et al. (2005);
Alexandrou et al. (2012a), the Truncated Solver Method (TSM) Alexandrou et al.
(2012a); Collins et al. (2007); Bali et al. (2010) and the Hopping Parameter
Expansion (HPE) Boucaud et al. (2008); McNeile and Michael (2001) have led to
a significant reduction in both stochastic and gauge noise associated with
disconnected quark loops. Moreover, using special properties of the twisted
mass fermion Lagrangian Frezzotti et al. (2001); Frezzotti and Rossi (2004a,
b, c) one can further enhance the signal-to-noise ratio by taking the
appropriate combination of flavors. On the hardware side, graphics cards
(GPGPUs or GPUs) can provide a large speedup in the evaluation of quark
propagators and contractions. In particular, for the TSM, which relies on a
large number of inversions of the Dirac matrix in single or half precision,
GPUs provide an optimal platform.

In this paper, the aim is to use our findings on the performance of recently
developed methods Alexandrou et al. (2013a) to compute to high accuracy the
disconnected contributions that enter in the determination of nucleon form
factors, sigma terms and first moments of parton distributions. The evaluation
will be performed using one ensemble generated with two light degenerate
quarks and a strange and charm quark with masses fixed to their physical
values ($N_{f}=2+1+1$) using the twisted mass fermion discretization. The
lattice size is $32^{3}\times 64$, the lattice spacing extracted from the
nucleon mass Alexandrou et al. (2013b) $a=0.082(1)(4)$ fm and the pion mass
about 370 MeV. This ensemble will be hereafter referred to as the B55.32
ensemble. The aim is to compare the disconnected contributions computed using
${\cal O}(10^{5})$ measurements to the connected ones and assess the
importance of the disconnected contributions to nucleon observables computed
in lattice QCD for this given ensemble. The paper is organized as follows: in
Section II we summarize the algorithms and variance reduction techniques
employed, and in Section III we present the main numerical results of this
paper, namely the disconnected contributions to nucleon generalized form
factors. In Section IV we compare the disconnected contributions with the
corresponding connected ones. In Section V we give our conclusions and
outlook.

##  II Methods for disconnected calculations

###  II.1 Truncated Solver Method

The exact computation of all-to-all (or time-slice-to-all) propagators on a
lattice volume of physical interest is outside our current computer power,
since this would require volume (or spatial volume) times inversions of the
Dirac matrix, whose size ranges from $\sim 10^{7}$ for a $24^{3}\times 48$
lattice to $\sim 10^{9}$ for the largest volumes of $96^{3}\times 192$
considered nowadays. We will use the Truncated Solver Method (TSM) combined
with the one-end trick to evaluate the disconnected contributions. This method
was shown to be optimal for most observables involved in nucleon structure
computations Alexandrou et al. (2013a). For completeness we summarize here the
methods and refer the reader to Ref. Alexandrou et al. (2013a) for a more
detailed description and the comparison against other methods.

The usual approach to evaluate disconnected contributions is to compute an
unbiased stochastic estimate of the all-to-all propagator Bitar et al. (1989)
by generating a set of $N_{r}$ sources $\left|\eta_{r}\right\rangle$ randomly
drawn from e.g. $\mathbb{Z}_{2}\otimes i\mathbb{Z}_{2}$. Solving for
$\left|s_{r}\right\rangle$ in

$M\left|s_{r}\right\rangle=\left|\eta_{r}\right\rangle$ (1)

and calculating

$M_{E}^{-1}:=\frac{1}{N_{r}}\sum_{r=1}^{N_{r}}\left|s_{r}\right\rangle\left\langle\eta_{r}\right|\approx
M^{-1}$ (2)

provides an unbiased estimate of the all-to-all propagator as
$N_{r}\rightarrow\infty$. Since, in general, the number of noise vectors
$N_{r}$ required is much smaller than the lattice volume $V$, the computation
becomes feasible. How large $N_{r}$ should be depends on the observable.

Figure 1: The error on the isoscalar momentum fraction $\delta\langle
x\rangle_{u+d}$ as a function of $N_{\rm HP}+N_{\rm LP}$ for 68000
measurements. The three leftmost points (red squares) correspond to $N_{\rm
LP}=0$ and the three rightmost to $N_{\rm HP}=24$. The dotted line is the
result of fitting to the Ansatz $1/\sqrt{a+\frac{b}{N_{\rm HP}+N_{\rm LP}}}$.

The TSM is a way to increase $N_{r}$ at a reduced computational cost. The idea
behind the method is the following: instead of inverting to high precision the
stochastic sources in Eq. (1), we can aim at a low precision (LP) estimate

$\left|s_{r}\right\rangle_{LP}=\left(M^{-1}\right)_{LP}\left|\eta_{r}\right\rangle,$
(3)

where the number of inversions of the Conjugate Gradient (CG) used is
truncated. The criterion for the low precision inversions can be selected by
specifying a relaxed stopping condition in the CG e.g. by allowing a
relatively large value of the residual, which in turn determines the number of
iterations required to invert a source to low precision. Following Refs.
Alexandrou et al. (2012a, 2013a), we choose a stopping condition at fixed
value of the residual $|\hat{r}|_{\rm LP}\sim 10^{-2}$. $N_{\rm HP}$ is then
selected by requiring that the bias introduced when using $N_{\rm LP}$ low
precision vectors is corrected. We estimate the correction $C_{E}$ to the bias
stochastically by inverting a number of sources to high and low precision, and
calculating the difference,

$C_{E}:=\frac{1}{N_{\rm HP}}\sum_{r=1}^{N_{\rm
HP}}\left[\left|s_{r}\right\rangle_{\rm HP}-\left|s_{r}\right\rangle_{\rm
LP}\right]\left\langle\eta_{r}\right|,$ (4)

where the $\left|s_{r}\right\rangle_{\rm HP}$ are calculated by solving Eq.
(1) up to high precision, so our final estimate becomes

$\displaystyle M_{E_{TSM}}^{-1}:=$ $\displaystyle\frac{1}{N_{\rm
HP}}\sum_{r=1}^{N_{\rm HP}}\left[\left|s_{r}\right\rangle_{\rm
HP}-\left|s_{r}\right\rangle_{\rm LP}\right]\left\langle\eta_{r}\right|$ (5)
$\displaystyle+$ $\displaystyle\frac{1}{N_{\rm LP}}\sum_{j=N_{\rm HP}}^{N_{\rm
HP}+N_{\rm LP}}\left|s_{r}\right\rangle_{\rm LP}\left\langle\eta_{r}\right|,$

which requires $N_{\rm HP}$ high precision (HP) inversions and $N_{\rm
HP}+N_{\rm LP}$ low precision inversions. The ratio of the number of HP
inversions to the LP ones is determined with the criterion of choosing as
large a ratio as possible while still ensuring that the final result is
unbiased. In this work, we will compute fermion loops with the complete set of
$\Gamma$-matrices up to one-derivative operators. The tuning is, thus,
performed using an operator that requires a large number of stochastic noise
vectors, such as the nucleon isoscalar momentum fraction $\langle
x\rangle_{u+d}$ and we optimize $N_{\rm HP}$ and $N_{\rm LP}$ so as to get the
smallest error at the lowest computational cost. In Fig. 1 we show the error
on $\langle x\rangle_{u+d}$ as one varies $N_{\rm HP}$ and $N_{\rm LP}$. As
can be seen, the error decreases like $1/\sqrt{a+\frac{b}{N_{\rm HP}+N_{\rm
LP}}}$ with $a$ and $b$ positive parameters. Fixing $N_{\rm HP}=24$ and
increasing $N_{\rm LP}$ reduces the error rapidly until $N_{\rm LP}$ reaches
about $N_{\rm LP}\sim~{}300$. In Ref. Alexandrou et al. (2013a) we showed that
a ratio of $N_{\rm LP}$ to $N_{\rm HP}$ of about 20 can be considered
sufficient to produce an unbiased estimate for the class of observables
considered here. Therefore, in this work we take $N_{\rm HP}=24$ and choose
$N_{\rm LP}=500$ for the light quark sector. For the strange and charm quarks
we take $N_{\rm LP}=300$. These values were shown to also be optimal for the
isoscalar axial charge Alexandrou et al. (2013a).

###  II.2 The one-end trick

The twisted mass fermion (TMF) formulation allows the use of a very powerful
method to reduce the variance of the stochastic estimate of the disconnected
diagrams. From the discussion given in section II.1, the standard way to
proceed with the computation of disconnected diagrams would be to generate
$N_{r}$ stochastic sources $\eta_{r}$, invert them as indicated in Eq. (1),
and compute the disconnected diagram corresponding to an operator $X$ as

$\displaystyle\frac{1}{N_{r}}\sum_{r=1}^{N_{r}}\left\langle\eta^{\dagger}_{r}Xs_{r}\right\rangle$
$\displaystyle=$ $\displaystyle\textrm{Tr}\left(M^{-1}X\right)$ (6)
$\displaystyle+$ $\displaystyle O\left(\frac{1}{\sqrt{N_{r}}}\right),$

where the operator $X$ is expressed in the twisted basis. However, if the
operator $X$ involves a $\tau_{3}$ acting in flavor space, one can utilize the
following identity of the twisted mass Dirac operator with $+\mu$ denoted by
$M_{u}$ and $-\mu$ denoted by $M_{d}$:

$M_{u}-M_{d}=2i\mu a\gamma_{5}.$ (7)

Inverting this equation we obtain

$M^{-1}_{u}-M^{-1}_{d}=-2i\mu aM_{d}^{-1}\gamma_{5}M_{u}^{-1}.$ (8)

Therefore, instead of using Eq. (6) for the operator $X\tau_{3}$, we can
alternatively write

$\displaystyle\frac{2i\mu a}{N_{r}}\sum_{r=1}^{N_{r}}\left\langle
s^{\dagger}_{r}\gamma_{5}Xs_{r}\right\rangle=$
$\displaystyle\textrm{Tr}\left(M_{u}^{-1}X\right)-\textrm{Tr}\left(M_{d}^{-1}X\right)$
$\displaystyle+O\left(\frac{1}{\sqrt{N_{r}}}\right)=$ $\displaystyle-2i\mu
a\textrm{Tr}\left(M_{d}^{-1}\gamma_{5}M_{u}^{-1}X\right)$
$\displaystyle+O\left(\frac{1}{\sqrt{N_{r}}}\right).$ (9)

Two main advantages result due to this substitution: i) the fluctuations are
effectively reduced by the $\mu$ factor, which is small in current
simulations, and ii) an implicit sum of $V$ terms appears in the right hand
side (rhs) of Eq. (8). The trace of the left hand side (lhs) of the same
equation develops a signal-to-noise ratio of $1/\sqrt{V}$, but thanks to this
implicit sum, the signal-to-noise ratio of the rhs becomes $V/\sqrt{V^{2}}$.
In fact, using the one-end trick yields for the same operator a large
reduction in the errors for the same computational cost as compared to not
using it Boucaud et al. (2008); Michael and Urbach (2007); Dinter et al.
(2012). A similar approach proved to be very successful in the determination
of the $\eta^{\prime}$ mass Jansen et al. (2008); Ottnad et al. (2012);
Michael et al. (2013). The identity given in Eq. (8) can only be applied when
a $\tau_{3}$ flavor matrix appears in the operator expressed in the twisted
basis. For other operators one can use the identity

$M_{u}+M_{d}=2D_{W},$ (10)

where $D_{W}$ is the Dirac-Wilson operator without a twisted mass term. After
some algebra, one finds

$\displaystyle\frac{2}{N_{r}}\sum_{r=1}^{N_{r}}\left\langle
s^{\dagger}_{r}\gamma_{5}X\gamma_{5}D_{W}s_{r}\right\rangle$ $\displaystyle=$
$\displaystyle\textrm{Tr}\left(M_{u}^{-1}X\right)+\textrm{Tr}\left(M_{d}^{-1}X\right)$
(11) $\displaystyle+$ $\displaystyle O\left(\frac{1}{\sqrt{N_{r}}}\right).$

This lacks the $\mu$-suppression factor, which, as we will see in the
following sections and as discussed in more detail in Ref. Alexandrou et al.
(2013a), introduces a considerable penalty in the signal-to-noise ratio.

Because of the volume sum that appears in Eq. (8) and Eq. (11), the sources
must have entries on all sites, which in turn means that we can compute the
fermion loop at all time slices where the operator is inserted in a single
inversion. This allows us to evaluate the three-point function for all
combinations of source-sink time separation and insertion time slices, which
will prove essential in identifying the contribution of excited state effects
for the different operators.

##  III Results

In this section we present results from a high statistics evaluation of all
the disconnected contributions involved in the evaluation of nucleon form
factors and first moments of generalized parton distributions as well as sigma
terms. As already mentioned, the analysis is performed using an ensemble of
$N_{f}=2+1+1$ twisted mass configurations simulated with pion mass of
$am_{\pi}=0.15518(21)(33)$ and strange and charm quark masses fixed to
approximately their physical values (B55.32 ensemble) Baron et al. (2010). The
lattice size is $32^{3}\times 64$ giving $m_{\pi}L\sim 5$. We use the one-end
trick method combined with the TSM with $N_{\rm HP}=24$ and $N_{\rm LP}=500$
noise vectors for the light quark loops. For the strange and charm quark
sector we use $N_{\rm HP}=24$ and $N_{\rm LP}=300$. Using 2,300 gauge-field
configurations, with 16 source positions for the two-point function and by
averaging results for the proton/neutron and forward/backward propagating
nucleons we effectively have $\sim 150,000$ measurements.

An advantage of the one-end trick is that, having the loop at all time slices,
we can combine with two-point functions produced at any source time slice.
Furthermore, since the noise sources are defined on all sites, we obtain the
fermion loops at all insertion time slices. We can thus compute all possible
combinations of source-sink time separations and insertion times in the three-
point function. This feature enables us to use the summation method, in
addition to the plateau method, with no extra computational effort.

The summation method has been known for a long time Maiani et al. (1987);
Gusken (1999) and has been revisited in the study of $g_{A}$ Capitani et al.
(2010). In both the plateau and summation approaches, one constructs ratios of
three- to two-point functions in order to cancel unknown overlaps and
exponentials in the leading contribution such that the matrix element of the
ground state is isolated. For general momentum transfer we consider the ratio

$R(t_{\rm ins},t_{s}){=}\frac{G^{3pt}(\Gamma^{\nu},{\vec{p}},{\vec{q}},t_{\rm
ins},t_{s})}{G^{2pt}(\vec{p}^{\prime},t_{s})}\sqrt{\frac{G^{2pt}(\vec{p},t_{s}{-}t_{\rm
ins})G^{2pt}(\vec{p}^{\prime},t_{\rm
ins})G^{2pt}(\vec{p}^{\prime},t_{s})}{G^{2pt}(\vec{p}^{\prime},t_{s}{-}t_{\rm
ins})G^{2pt}(\vec{p},t_{\rm ins})G^{2pt}(\vec{p},t_{s})}}$ (12)

where the two- and three-point functions are given respectively by

$\displaystyle G^{2pt}(\vec{q},t_{s})=$
$\displaystyle\sum_{\vec{x}_{s}}\,e^{-ix_{s}\cdot\vec{q}}\,{\Gamma^{0}_{\beta\alpha}}\,\langle{J_{\alpha}(t_{s},\vec{x}_{s})}{\overline{J}_{\beta}(0,\vec{0})}\rangle$
(13) $\displaystyle G^{3pt}(\Gamma^{\nu},\vec{p},\vec{q},t_{\rm ins},t_{s})=$
$\displaystyle\sum_{\vec{x}_{\rm ins},\vec{x}_{s}}\,e^{i\vec{x}_{\rm
ins}\cdot\vec{q}}\,e^{-i\vec{x}_{s}\cdot\vec{p}}\,\Gamma^{\nu}_{\beta\alpha}\,\langle{J_{\alpha}(t_{s},\vec{x}_{s})}\mathcal{O}^{\mu_{1}\cdots\mu_{n}}(t_{\rm
ins},\vec{x}_{\rm ins}){\overline{J}_{\beta}(0,\vec{0})}\rangle\,.$ (14)

$q=p^{\prime}-p$ is the momentum transfer, $t_{s}$ is the time separation
between the sink and the source with the source taken at zero, and $t_{\rm
ins}$ the time separation between the current insertion and the source. We
consider the complete set of operators $\mathcal{O}^{\mu_{1},\cdots,\mu_{n}}$
up to one derivative, namely the scalar $\bar{\psi}\,\psi$, vector
$\bar{\psi}\,\gamma^{\mu}\psi$, axial-vector
$\bar{\psi}\,\gamma^{5}\,\gamma^{\mu}\psi$ and the tensor
$\bar{\psi}\sigma^{\mu\nu}\psi$ currents, and the one-derivative vector
$\bar{\psi}\,\gamma^{\\{\mu_{1}}D^{\mu_{2}\\}}\psi$ and axial-vector
$\bar{\psi}\,\gamma_{5}\,\gamma^{\\{\mu_{1}}D^{\mu_{2}\\}}\psi$ operators. We
consider kinematics for which the final momentum $\vec{p}^{\prime}=0$ when we
consider the connected contributions. For the evaluation of disconnected
contributions we use kinematics where $\vec{p}=\vec{p}^{\prime}\neq 0$ as well
as $\vec{p}^{\prime}=0$. The projection matrices ${\Gamma^{0}}$ and
${\Gamma^{k}}$ are given by:

${\Gamma^{0}}=\frac{1}{4}(\mathds{1}+\gamma_{0})\,,\quad{\Gamma^{k}}={\Gamma^{0}}i\gamma_{5}\sum_{k=1}^{3}\gamma_{k}\,.$
(15)

For zero momentum transfer the ratio simplifies to

$R(t_{ins},t_{s})=\frac{G^{3pt}(\Gamma^{\nu},\vec{p},t_{ins},t_{s})}{G^{2pt}(t_{,}\vec{p})}$
(16)

The leading time dependence of the ratio $R(t_{\rm ins},t_{s})$ is given by

$R(t_{ins},t_{s})=R_{GS}+O(e^{-\Delta E_{K}t_{ins}})+O(e^{-\Delta
E_{K}(t_{s}-t_{ins})}),$ (17)

where $R_{GS}$ is the matrix element of interest, and the other contributions
come from the undesired excited states of energy difference $\Delta E_{K}$. In
the plateau method, one plots $R(t_{ins},t_{s})$ as a function of $t_{\rm
ins}$. For large time separations $t_{\rm ins}$ and $t_{s}-t_{\rm ins}$ when
excited state effects are negligible this ratio becomes a constant (plateau
region) and therefore fitting it to a constant yields $R_{GS}$. In the
alternative summation method, one performs a sum over $t_{\rm ins}$ to obtain:

$R_{\rm sum}(t_{s})=\sum_{t_{\rm ins}=0}^{t_{\rm ins}=t_{s}}R(t_{\rm
ins},t_{s})=t_{s}R_{GS}+a+O(e^{-\Delta E_{K}t_{s}})$ (18)

where $a$ is a constant and the exponential contributions coming from the
excited states decay as $e^{-\Delta E_{K}t_{s}}$ as opposed to the plateau
method where excited states are suppressed like $e^{-\Delta
E_{K}(t_{s}-t_{ins})}$, with $0\leq t_{\rm ins}\leq t_{s}$. Therefore, we
expect a better suppression of the excited states for the same $t_{s}$. Note
that one can exclude from the summation the initial and final time slices
$t_{s}$ and $0$ without affecting the dependence on $t_{s}$ in Eq. (18). The
results given in this work are obtained excluding these contact terms from the
summation. The drawback of the summation method is that one requires knowledge
of the three point function for all insertion times and multiple sink times
and one needs to fit to a straight line with two fitting parameters instead of
one.

$Z_{A}$ | $Z_{T}$ | $Z_{P}$ | $Z_{DV}^{\mu\mu}$ | $Z_{DV}^{\mu\neq\nu}$ | $Z_{DA}^{\mu\mu}$ | $Z_{DA}^{\mu\neq\nu}$
---|---|---|---|---|---|---
0.757(3) | 0.769(1) | 0.506(4) | 1.019(4) | 1.053(11) | 1.086(3) | 1.105(2)
Table 1: Renormalization constants in the chiral limit at $\beta=1.95$ in the
$\overline{\rm MS}$-scheme at $\mu=2$ GeV. $Z_{A}$, $Z_{T}$ and $Z_{P}$ are
the renormalization constants for the axial-vector, tensor and scalar
currents, and $Z_{DV}$ and $Z_{DA}$ for the one-derivative vector and axial-
vector operators ${\cal O}^{\mu\nu}$. The errors given are statistical.

Figure 2: The disconnected contribution to the ratio from which $\sigma_{\pi
N}$ is extracted. On the upper panel we show the ratio as a function of the
insertion time slice with respect to the mid-time separation ($t_{\rm
ins}-t_{s}/2$) for source-sink time separations, $t_{\rm s}=$14$a$ (red filled
circles), $t_{\rm s}=16a$ (blue filled squares), $t_{\rm s}=18a$ (green open
squares) and $t_{\rm s}=20a$ (yellow filled triangles). In the central panel
we show the summed ratio, for which the fitted slope yields the desired matrix
element. On the lower panel we show the results obtained for the fitted slope
of the summation method for various choices of the initial and final fit time
slices. The star shows the choice for which the gray bands are plotted in the
upper and central panels.

Figure 3: The ratio from which the strange quark content of the nucleon,
$\sigma_{s}$, is extracted. The notation is the same as that of Fig. 2.

Figure 4: The ratio from which the charm quark content of the nucleon,
$\sigma_{c}$, is extracted. The notation is the same as that of Fig. 2.

Figure 5: The disconnected contribution to the renormalized ratio which yields
the isoscalar axial charge of the nucleon, $g_{A}^{u+d}$. The upper panel
shows the ratio as a function of the insertion time slice with respect to the
mid-time separation ($t_{\rm ins}-t_{s}/2$) for source-sink separations
$t_{\rm s}=8a$ (red filled circles), $t_{\rm s}=10a$ (blue filled squares),
$t_{\rm s}=12a$ (green open squares) and $t_{\rm s}=14a$ (yellow filled
triangles). The central panel shows the summed ratio and the lower panel the
results obtained for the fitted slope of the summation method for various
choices of the initial and final fit time slices as explained in the text. The
star shows the choice of $t_{i}$, which yields the gray bands shown in the
upper and central plots.

Figure 6: The strange-quark contribution to the renormalized ratio yielding
the nucleon axial charge $g_{A}^{s}$. The notation is the same as that of Fig.
5.

Figure 7: The charm-quark contribution to the renormalized ratio yielding the
nucleon axial charge $g_{A}^{c}$. The notation is the same as that of Fig. 5.

Figure 8: The disconnected contribution to the renormalized ratio yielding the
nucleon isoscalar tensor charge $g_{T}^{u+d}$. The notation is the same as
that of Fig. 2.

Figure 9: The disconnected contribution to the renormalized ratio yielding the
nucleon isoscalar momentum fraction $\langle x\rangle_{u+d}$. The notation is
the same as that of Fig. 5.

Figure 10: The disconnected contribution to the renormalized ratio yielding
nucleon isoscalar helicity moment $\langle x\rangle_{\Delta u+\Delta d}$. The
notation is the same as that of Fig. 5.

Figure 11: Disconnected contributions to the renormalized ratio yielding the
isoscalar axial-vector and pseudo-scalar form-factors $G_{A}$ and $G_{p}$
(upper), the electric form-factor $G_{E}$ (center) and the magnetic form-
factor $G_{M}$ (lower) at the lowest non-zero momentum transfer allowed for
this lattice size.  Figure 12: The renormalized ratio which yields the
strange-quark contribution to the axial charge of the nucleon, $g_{A}^{s}$. In
the left panel, the plateau method is used on the first half of the ensemble
(A-set), while the summation method is used on the second half of the ensemble
(B-set). In the right panel, he plateau method is used on the A-set, while the
summation method is used on the B-set.

Before comparing the lattice matrix elements $R_{\rm GS}$ with experiment we
need to renormalize them. We denote the renormalized ratio by
$\tilde{R}(t_{\rm ins},t_{s})$. Regarding the renormalization of the sigma
terms, the twisted mass formulation has the additional advantage of avoiding
any mixing, even though we are using Wilson-type fermions Dinter et al.
(2012). For the case of the axial charge, renormalization involves mixing from
the three quark sectors. For the tree-level Symanzik improved gauge action
this mixing was shown to be a small effect of a few percent Skouroupathis and
Panagopoulos (2009). We expect this to hold also for the Iwasaki action used
in this work and for the other isoscalar quantities. In this work, we neglect
the small difference in the renormalization constant between connected and
disconnected contributions and we use the same renormalization constants as
for the connected piece. They are given in Table 1. The value of $Z_{P}$ needs
a pole subtraction and is taken from Ref. Blossier et al. (2011); ETMC , while
all the others have been calculated using the approach given in Refs.
Alexandrou et al. (2011); Alexandrou et al. (2012b). All the renormalization
constants, except $Z_{A}$ which is scheme and scale independent, are converted
from RI-MOM to $\overline{\rm MS}$ at a scale of $\mu=2$ GeV. The conversion
factors for $Z_{T}$ are taken from Ref. Gracey (2003), and for the one-
derivative operators from Ref. Alexandrou et al. (2011), computed to three-
loops. We remark that in the twisted basis the scalar charge is renormalized
with $Z_{P}$.

In Fig. 2 we show the results for the disconnected contribution to the ratio
from which the $\sigma_{\pi N}$-term is extracted. The ratio is plotted versus
the time separation of the current insertion $t_{\rm ins}$ from the source,
shifted by $t_{s}/2$. When this ratio becomes time independent (plateau
region) fitting to a constant yields $\sigma_{\pi N}$. As can be seen,
however, increasing the source-sink time separation increases the value
extracted from fitting to the plateau (plateau value). We observe that one
requires a source-sink time separation of at least 18 to 20 time slices in
order for the plateau value to stabilize. This is a distance of $\gtrsim 1.5$
fm, which is significantly larger than the nominal source-sink separation of
1.0 fm-1.2 fm typically used in nucleon matrix element calculations. In the
central panel we show the ratio summed over the insertion time slice as given
in Eq. (18) referred to as summation method (SM) as a function of the source-
sink time separation time. As explained earlier, by fitting the ratio to a
straight line one obtains the desired matrix element as the slope. This is
done for several choices of the initial and final fit time slices ($t_{i}$ and
$t_{f}$ respectively) with the results displayed in the lower panel of the
figure. As one increases the initial fit time slice the excited state
contributions are expected to become smaller and thus the fitted value
stabilizes. Note, however, that the slope changes and one needs to vary the
fit range until the slope converges. Therefore, if one has only a small number
of source-sink time separations one may miss the variation of the slope. As in
the case of the plateau method where we take the smallest $t_{s}$ for which
excited states are sufficiently suppressed, it is desirable to take the
smallest $t_{i}$ for which the excited states no longer contribute
significantly, since the error to signal ratio increases with $t_{i}$. Taking
the value of the slope to be the one given by the star yields the value of
$\sigma_{\pi N}$ shown by the gray band in the upper panel of the figure. As
can be seen, the resulting value is in agreement with the (colored) band
obtained from the plateau method for $t_{s}/a=20$.

A similar analysis is undertaken for the strange- and charm-quark sigma terms,
shown in Figs. 3 and 4 respectively. For $\sigma_{s}$, similar remarks can be
made as in the case of $\sigma_{\pi N}$, most notably concerning the large
source-sink separation required for the plateau method to converge. As
expected, the results between the summation and the plateau method are
consistent also in this case, when excited states are suppressed. Non-zero
results for $\sigma_{s}$ were also obtained in Ref. Gong et al. (2013) using
optimal noise sources and low-mode substitution techniques. For the case of
the charm content, our results are consistent with zero both when using the
plateau method as well as when using the summation method allowing us only to
obtain an upper bound to its value. In Ref. Gong et al. (2013) a non-zero
result was obtained as one approaches the chiral limit. Since our aim in this
work is to compute quark loops using high statistics for one ensemble we will
address the quark mass dependence in a follow-up work.

Similar analyses are carried out for the disconnected contributions entering
the ratios determining the nucleon axial charge. For observables like $g_{A}$
where one does not have the $\tau^{3}$ flavor combination in the twisted basis
it is advantageous to use the discrete symmetries of the twisted mass
formulation Frezzotti and Rossi (2004b, c), namely parity combined with
isospin flip $u\leftrightarrow d$, $\gamma_{5}$-isospin hermiticity, and
charge-$\gamma_{5}$-isospin hermiticity, in order to reduce gauge noise.
Considering the properties of the quark loops and of the nucleon two-point
functions that enter in the computation of the disconnected three-point
function under these symmetries one can derive appropriate products taking
their real or imaginary parts thus suppressing gauge noise. This was shown to
be advantageous in the calculation of the first moments of the unpolarized
momentum distribution in Ref. Deka et al. (2009). These symmetries are used
for the results shown from now on. In Figs. 5, 6 and 7 we show, respectively,
results for the ratio from which the nucleon matrix elements of the axial-
vector current yielding the isoscalar $g_{A}$,the strange $g_{A}^{s}$ and the
charm $g_{A}^{c}$ are extracted. We first note that for the case of
$g_{A}^{u+d}$ we observe less contamination from excited states than in the
case of the sigma terms. This is evident from the smaller source-sink time
separations required in order for the plateau or summation method to converge.
Furthermore, we clearly observe a non-zero value for the case of the
disconnected contributions to the isoscalar $g_{A}$ as well as for
$g_{A}^{s}$. For $g_{A}^{c}$ our results are consistent with zero and we can
only give an upper bound to its value. The nucleon tensor charge $g_{T}^{u+d}$
is also computed and the ratio from which is extracted is shown in Fig 8. We
observe a very small value for the disconnected contribution, with an error of
about 90%. For the summation method the statistical uncertainty does not allow
a meaningful fit.

The nucleon matrix elements involving derivative operators probe moments of
parton distributions, which are extracted from deep inelastic scattering
measurements. In this work we compute the disconnected contributions to the
isoscalar nucleon momentum fraction $\langle x\rangle_{u+d}$, which involves
the vector derivative operator and the isoscalar nucleon polarized moment
$\langle x\rangle_{\Delta u+\Delta d}$ involving the axial-vector derivative
operator. We apply the symmetries of the twisted mass action discussed above
as well as consider a moving frame and thus have the nucleon carrying non-zero
equal initial and final momentum for three-point functions with zero momentum
transfer. We find that, when the nucleon carries the lowest momentum allowed
for this lattice, the statistical error is reduced. The disconnected
contributions to the ratios, from which the matrix elements of the vector and
axial-vector derivative operators, are extracted are shown in Figs. 9 and 10
respectively. For $\langle x\rangle_{u+d}$ we find a value consistent with
zero both with the plateau and summation method. Having one unit of momentum
improves the signal enabling us to deduce an upper bound on the value of this
matrix element. For $\langle x\rangle_{\Delta u+\Delta d}$ the statistical
errors remain large but nevertheless we obtain a non-zero value. Considering a
moving nucleon leads in this particular case to a substantial reduction in the
error. We note that increasing the sink-source time separation is crucial in
order for this observable to develop a non-zero result. This is clearly seen
in the slope which becomes non-zero for $t_{s}/a>8$. Since a large $t_{s}$
also leads to larger errors it is no surprise that such a large number of
statistics is needed to obtain a meaningful signal. This may also indicate
that even larger number of statistics are needed to stabilize further the
signal.

Apart from matrix elements for zero momentum transfer presented so far,
disconnected contributions arise in the isoscalar electromagnetic and axial
form factors at finite momentum. Computationally, these are straightforward to
extract, since one takes the Fourier transform of the insertion coordinate of
the loop to obtain the matrix element at all momenta. The finite momentum
matrix elements, however, are expected to be nosier than the zero-momentum
ones, since the energy factors appearing in the exponents of the signal are
larger. The disconnected contributions to the axial form-factors, electric
form-factor and magnetic form-factor are shown in Fig. 11 for a single unit of
momentum transfer. Due to the structure of the matrix elements and the way
these are computed on the lattice, for the case of the axial form factors
$G_{A}$ and $G_{p}$, the plot shows the ratio of a linear combination from
which these form factors are extracted after the plateau fit. $G_{E}$ and
$G_{M}$, on the other hand, can be extracted from different ratios allowing us
to plot them separately. We note that we perform a similar analysis for these
quantities as for the zero-momentum case where both plateau and summation
methods are investigated for the optimal fit ranges. For the axial form-
factors we obtain a clearly non-zero value. For the electromagnetic case, the
disconnected contributions for both the isoscalar electric and magnetic form
factors are statistically consistent with zero.

Figure 13: Connected contributions to the ratio yielding $\sigma_{\pi N}$
(upper) and nucleon isoscalar axial charge (lower), for various source-sink
time separations are shown. Results obtained from a fit to a constant to the
ratio (colored band) and from a linear fit to the summed ratio (gray band) are
also displayed.

.

Figure 14: Connected contributions to the renormalized ratio yielding the
isoscalar nucleon momentum fraction (upper), the isoscalar nucleon helicity
moment (center) and the axial and pseudo-scalar form factors $G_{A}(Q^{2})$
and $G_{p}(Q^{2})$ at a single unit of momentum (lower) are shown. For the
momentum fraction and helicity, we show the results obtained from a fit to a
constant to the renormalized ratio (colored band) and from a linear fit to the
summed renormalized ratio (gray band).

.

Finally we comment on the issue of correlations. The summation and plateau
methods for various quantities are compared using the same set of gauge
configurations and found to be consistent. Since these results can be
correlated, the difference between the results of the two methods maybe
underestimated. Thus, it is worthwhile to investigate the two methods using
different sets of configurations. To perform this check we split our ensemble
into two equal sets, which we will refer to as A-set and B-set, and redo our
analysis on these two sets separately. We show the result for the case of the
strange-quark contribution to the axial charge in Fig. 12. As can seen, the
values computed in each set both using the plateau and summation methods are
in agreement. Furthermore, the plateau computed using the A-set is consistent
with the summation method computed using the B-set and vice versa. This
agreement indicates that the consistency between the results extracted using
the summation and plateau methods on the full ensemble is not due to possible
correlations.

##  IV Comparison with connected contribution

The main motivation for calculating disconnected fermion loops is to eliminate
the systematic uncertainty, which arises when these are omitted from
calculations of hadronic matrix elements. For instance, the nucleon axial
charge is typically computed in the isovector combination, where the fermion
loops of the up- and down- quarks cancel. However, if one is interested in the
intrinsic spin fraction carried by the individual quarks, one needs, in
addition to the isovector, the isoscalar combination, which involves
disconnected diagrams. Typically, in lattice QCD calculations up to now, the
disconnected contributions have been omitted. It is, therefore, important to
identify how large the contributions of disconnected diagrams are, in order to
bound the systematic error introduced when these are neglected.

Observable | connected | disconnected | total
---|---|---|---
Results at zero momentum transfer ($Q^{2}=0$)
$\sigma_{\pi N}$ | [MeV] | 164.6(7.2) | 16.6(2.4) | 181.3(7.6)
$\sigma_{s}$ | [MeV] |  | 21.7(3.6) | 21.7(3.6)
$\sigma_{c}$ | [MeV] |  | 16(30) | 16(30)
$g_{S}^{u+d}$ |  | 6.30(27) | 0.639(95) | 6.94(29)
$g_{S}^{s}$ |  |  | 0.246(41) | 0.246(41)
$g_{A}^{u+d}$ |  | 0.576(13) | -0.0699(89) | 0.506(15)
$g_{A}^{s}$ |  |  | -0.0227(34) | -0.0227(34)
$g_{T}^{u+d}$ |  | 0.673(13) | -0.0016(14) | 0.671(13)
$\langle x\rangle_{u+d}$ |  | 0.586(22) | 0.027(76) | 0.614(80)
$\langle x\rangle_{\Delta u+\Delta d}$ |  | 0.1948(51) | -0.058(22) | 0.136(23)
$J^{u}$ |  | 0.2781(94) | -0.076(77) | 0.202(78)
$J^{d}$ |  | -0.0029(94) | -0.076(77) | -0.078(78)
$\Delta\Sigma^{u}/2$ |  | 0.4273(50) | -0.0174(75) | 0.4098(55)
$\Delta\Sigma^{d}/2$ |  | -0.1389(50) | -0.0174(75) | -0.1564(55)
Results for $\vec{q}^{2}=(2\pi/L)^{2}$ or $Q^{2}\simeq$0.19 GeV2
$G^{u+d}_{E}$ |  | 2.2698(78) | 0.024(21) | 2.293(22)
$G^{u+d}_{M}$ |  | 2.088(49) | -0.066(75) | 2.022(89)
$G^{u+d}_{A}$ |  | 0.5155(94) | -0.0564(72) | 0.459(11)
$G^{u+d}_{p}$ |  | 9.81(65) | -1.90(35) | 7.90(74)
$B^{u+d}_{20}$ |  | -0.035(16) | -0.33(29) | -0.36(29)
$G^{p}_{E}$ |  | 0.7453(32) | 0.0040(58) | 0.7493(47)
$G^{n}_{E}$ |  | 0.0113(32) | 0.0040(58) | 0.0153(47)
$G^{p}_{M}$ |  | 1.847(28) | -0.011(42) | 1.836(31)
$G^{n}_{M}$ |  | -1.151(28) | -0.011(42) | -1.162(31)
Table 2:  The connected and disconnected contributions to the various nucleon
observables for the B55.32 ensemble are given in column two and three, whereas
column four has the total contribution. The form factors $G_{E}$, $G_{M}$,
$G_{A}$ and $G_{p}$, and generalized form factor $B_{20}$ are given for
$\vec{q}=2\pi/L$. The disconnected contributions were obtained using about
150,000 measurements.

In order to assess the importance of disconnected contributions, we evaluate
the connected contributions to the isoscalar matrix elements of the operators
discussed in the previous section. In Figs. 13 and 14 we show the renormalized
ratios from which the connected part of the isoscalar matrix elements are
extracted. These results are obtained using 1200 gauge field configurations
and inverted for multiple source-sink time separations to allow applying the
summation method. We stress that, for the evaluation of the connected
contributions unlike the case of the disconnected, to obtain multiple source-
sink time separations one needs to do a new set of inversions for each sink-
source time separation.

The multiple source-sink time separations are computed more efficiently by
using the EigCG Stathopoulos and Orginos (2010); Stathopoulos et al. (2009)
method to deflate the lowest eigenvalues with every new right-hand-side. For
the connected contributions shown here, we compute the sequential propagators
for eight source-sink time separations, namely from $t_{s}=4a$ to $t_{s}=18a$
for every even time separation. In addition, the sequential propagators are
computed for both unpolarized and polarized nucleon sinks, meaning in total 16
sequential propagators per configuration, or 16$\times$12=192 right-hand-sides
are needed, one for each color-spin component. Our EigCG is set up such that
ten eigenvalues per right-hand-side are deflated, stopping after a total of 24
right-hand-sides, after which the deflated space is kept constant at 240
eigenvalues for the remaining 168 right-hand-sides. With this setup, and at
this pion mass, we observe a speedup of more than 3 times, i.e. the 192 right-
hand-sides are computed for the same computational cost needed to compute 64
right-hand-sides when not using EigCG.

The ratios yielding the connected contribution to $\sigma_{\pi N}$, and the
isoscalar $g_{A}$ are shown in Fig. 13. These can be compared with the
corresponding ratios yielding the disconnected contributions to $\sigma_{\pi
N}$ and isoscalar $g_{A}$ shown in Figs. 2 and 5, respectively. As can be
seen, the behavior of the connected contributions is similar to the
disconnected ones, namely the sigma term shows large excited state
contamination requiring large sink-source separations whereas in the case of
$g_{A}^{u+d}$ the excited states are negligible even for $t_{s}/a=10$. For a
better comparison between connected and disconnected contributions we collect
the results extracted from the plateau method for all nucleon observables in
Table 2. The disconnected contribution to the $\sigma_{\pi N}$ and isoscalar
$g_{A}$ are found to be larger than 10% of the connected contribution at this
quark mass. Clearly for both $\sigma_{\pi N}$ and $g_{A}^{u+d}$ these are
sizable effects and have to be taken into account. The scalar charge derives
from the same matrix element as the sigma term and therefore it also requires
inclusion of disconnected contributions. For the case of the momentum
fraction, the disconnected contribution is found to be consistent with zero as
can be seen in Fig. 9, and therefore we can only give an upper bound to its
size to be included in the systematic error of $\langle x\rangle_{u+d}$. For
the polarized moment $\langle x\rangle_{\Delta u+\Delta d}$, on the other
hand, one obtains a sizable non-zero result. Note that the disconnected
contribution is negative decreasing the value of $\langle x\rangle_{\Delta
u+\Delta d}$ quite substantially. The disconnected contribution to the tensor
charge is essentially zero not affecting its total value.

A comment can also be made for the case of the disconnected contributions to
the nucleon form factors computed at non-zero momentum shown in Fig. 11 at a
single unit of momentum transfer squared. For the electromagnetic form-factors
$G_{E}$ and $G_{M}$, we find that the disconnected contributions are
consistent with zero and with magnitude less than 1%. With the connected
contributions at this momentum transfer being of $O(1)$, this means that the
disconnected contributions will, at most, be at the 1% level. For the case of
the axial form factor $G_{A}^{u+d}$, the disconnected contribution is about
10% that of the connected and thus, it must be included. In the case of the
pseudo-scalar form factor $G_{p}$, we find that the disconnected contribution
is of similar magnitude as the connected one and thus it is crucial in order
to get reliable results for this observable to include the disconnected part.

Having the complete set of isoscalar matrix elements with both connected and
disconnected contributions, one can combine with the corresponding isovector
matrix elements, which do not depend on disconnected contributions, to obtain
the separate quark contributions to nucleon matrix elements. This is done in
Table 2 for all the various quantities considered in this work. Namely, the
up- and down-quark contributions to the nucleon spin $\Delta\Sigma^{u}/2$ and
$\Delta\Sigma^{d}/2$ are obtained by combining the isovector and isoscalar
axial charges. Including the disconnected contributions affects the values of
the intrinsic spin in particular in the case of the d-quark. In contrast, the
values of the nucleon total spin $J^{u}$ and $J^{d}$, obtained by combining
the isoscalar and isovector vector generalized form-factors $A_{20}$ and
$B_{20}$, are not affected and the disconnected contributions only contribute
an upper bound to the error. Finally, the proton/neutron electric and magnetic
form factors $G^{p/n}_{E}$ and $G^{p/n}_{M}$ at a single unit of momentum
transfer squared, which for this lattice size and quark mass corresponds to
$Q^{2}\simeq 0.19$ GeV2, are obtained from the isovector and isoscalar proton
electric and proton magnetic form-factors assuming flavor-SU(2) isospin
symmetry between up- and down-quarks. Only the value of $G_{E}^{n}$ is
affected although, within error bars, it is still consistent with the
connected value.

##  V Conclusions

The computation of disconnected contributions for flavor singlet quantities
has become feasible, due to the development of new techniques to reduce the
gauge and stochastic noise, and due to the increase in computational
resources. In this work, we use the truncated solver method and the one-end
trick on GPUs for the determination of disconnected contributions to the
nucleon matrix elements. The usage of GPUs is particularly important, due to
its efficiency in the evaluation of disconnected diagrams using the TSM, since
GPUs can yield a large speedup when employing single- and half-precision for
the computation of the LP inversions and contractions. The calculation is
performed for one ensemble of $N_{f}=2+1+1$ twisted mass fermions using very
high statistics. This is necessary in order to reduce the gauge noise and
obtain statistically significant results.

The results for all observables are analyzed using both the plateau and the
summation methods. A careful analysis of excited states is performed and we
find that the methods yield results that are compatible, as expected when
excited states contributions are negligible and identification of the fitting
ranges in both methods are well selected. Therefore, agreement of the values
extracted with the plateau and summation methods provides a good consistency
check. Since the one-end trick provides results for all sink-source
separations at no additional computational cost, such a check can be always
carried out.

Comparison of the connected to the disconnected contributions reveals clearly
that the latter are important for a number of observables related to nucleon
structure. For the sigma terms and scalar charge the disconnected
contributions amount to 10% the total value and thus they must be taken into
account. Similarly for the isoscalar axial charge we find more than 10%
contributions that must be taken into account in the discussion of the spin
carried by quarks in the proton. The disconnected contribution reduces the
value of $\Sigma^{d}$ by more than 10%, an effect that is important if we aim
at a few % accuracy. On the other hand, we find that the disconnected
contributions to the electromagnetic form factors at low $q^{2}$-values are
less than 1% at this pion mass. For the axial form factor $G_{A}$ the
disconnected contributions are sizable and persist at the level of 10% of the
value of the connected contribution even at non-zero momentum-transfer. For
$G_{p}$ the disconnected contribution is even larger reaching 20%.

In the future we plan to compute the disconnected contributions to these
quantities using simulations at physical pion mass. Such a computation will
require very large computational resources in order to obtain results with
meaningful statistical errors.

## Acknowledgments

A. V. and M. C. are supported by funding received from the Cyprus Research
Promotion Foundation under contracts EPYAN/0506/08 and and
TECHNOLOGY/$\Theta$E$\Pi$I$\Sigma$/0311(BE)/16 respectively. K. J. is partly
supported by RPF under contract
$\Pi$PO$\Sigma$E$\Lambda$KY$\Sigma$H/EM$\Pi$EIPO$\Sigma$/0311/16. This
research was in part supported by the Research Executive Agency of the
European Union under Grant Agreement number PITN-GA-2009-238353 (ITN
STRONGnet) and the infrastructure project INFRA-2011-1.1.20 number 283286
(HadronPhysics3), and the Cyprus Research Promotion Foundation under contracts
KY-$\Gamma$A/0310/02 and NEA Y$\Pi$O$\Delta$OMH/$\Sigma$TPATH/0308/31
(infrastructure project Cy-Tera, co-funded by the European Regional
Development Fund and the Republic of Cyprus through the Research Promotion
Foundation). Computational resources were provided by the Cy-Tera machine and
Prometheus (partly funded by the EU FP7 project PRACE-2IP under grant
agreement number: RI-283493) of CaSToRC, Forge at NCSA Illinois (USA),
Minotauro at BSC (Spain), and by the Jugene Blue Gene/P machine of the Jülich
Supercomputing Center awarded under PRACE.

## References

  * Alexandrou (2012) C. Alexandrou, Prog.Part.Nucl.Phys. 67, 101 (2012), eprint 1111.5960.
  * Boucaud et al. (2008) P. Boucaud et al. (ETM), Comput. Phys. Commun. 179, 695 (2008), eprint 0803.0224.
  * Michael and Urbach (2007) C. Michael and C. Urbach (ETM Collaboration), PoS LAT2007, 122 (2007), eprint 0709.4564.
  * Dinter et al. (2012) S. Dinter et al. (ETM Collaboration), JHEP 1208, 037 (2012), eprint 1202.1480.
  * Bernardson et al. (1994) S. Bernardson, P. McCarty, and C. Thron, Comput.Phys.Commun. 78, 256 (1994).
  * Viehoff et al. (1998) J. Viehoff et al. (TXL Collaboration), Nucl.Phys.Proc.Suppl. 63, 269 (1998), eprint hep-lat/9710050.
  * O’Cais et al. (2005) A. O’Cais, K. J. Juge, M. J. Peardon, S. M. Ryan, and J.-I. Skullerud (TrinLat Collaboration), Nucl.Phys.Proc.Suppl. 140, 844 (2005), eprint hep-lat/0409069.
  * Foley et al. (2005) J. Foley, K. Jimmy Juge, A. O’Cais, M. Peardon, S. M. Ryan, et al., Comput.Phys.Commun. 172, 145 (2005), eprint hep-lat/0505023.
  * Alexandrou et al. (2012a) C. Alexandrou, K. Hadjiyiannakou, G. Koutsou, A. O’Cais, and A. Strelchenko, Comput.Phys.Commun. 183, 1215 (2012a), eprint 1108.2473.
  * Collins et al. (2007) S. Collins, G. Bali, and A. Schafer, PoS LAT2007, 141 (2007), eprint 0709.3217.
  * Bali et al. (2010) G. S. Bali, S. Collins, and A. Schafer, Comput.Phys.Commun. 181, 1570 (2010), eprint 0910.3970.
  * McNeile and Michael (2001) C. McNeile and C. Michael (UKQCD Collaboration), Phys.Rev. D63, 114503 (2001), eprint hep-lat/0010019.
  * Frezzotti et al. (2001) R. Frezzotti, P. A. Grassi, S. Sint, and P. Weisz (Alpha), JHEP 0108, 058 (2001), eprint hep-lat/0101001.
  * Frezzotti and Rossi (2004a) R. Frezzotti and G. C. Rossi, Nucl. Phys. Proc. Suppl. 128, 193 (2004a), eprint hep-lat/0311008.
  * Frezzotti and Rossi (2004b) R. Frezzotti and G. C. Rossi, JHEP 08, 007 (2004b), eprint hep-lat/0306014.
  * Frezzotti and Rossi (2004c) R. Frezzotti and G. C. Rossi, JHEP 10, 070 (2004c), eprint hep-lat/0407002.
  * Alexandrou et al. (2013a) C. Alexandrou, M. Constantinou, V. Drach, K. Hadjiyiannakou, K. Jansen, et al. (2013a), eprint 1309.2256.
  * Alexandrou et al. (2013b) C. Alexandrou, M. Constantinou, S. Dinter, V. Drach, K. Jansen, et al. (2013b), eprint 1303.5979.
  * Bitar et al. (1989) K. Bitar, A. Kennedy, R. Horsley, S. Meyer, and P. Rossi, Nucl.Phys. B313, 377 (1989).
  * Jansen et al. (2008) K. Jansen, C. Michael, and C. Urbach (ETM Collaboration), Eur.Phys.J. C58, 261 (2008), eprint 0804.3871.
  * Ottnad et al. (2012) K. Ottnad et al. (ETM collaboration), JHEP 1211, 048 (2012), eprint 1206.6719.
  * Michael et al. (2013) C. Michael, K. Ottnad, and C. Urbach (2013), eprint 1310.1207.
  * Baron et al. (2010) R. Baron et al., JHEP 06, 111 (2010), eprint 1004.5284.
  * Maiani et al. (1987) L. Maiani, G. Martinelli, M. Paciello, and B. Taglienti, Nucl.Phys. B293, 420 (1987).
  * Gusken (1999) S. Gusken (1999), eprint hep-lat/9906034.
  * Capitani et al. (2010) S. Capitani, B. Knippschild, M. Della Morte, and H. Wittig, PoS LATTICE2010, 147 (2010), eprint 1011.1358.
  * Skouroupathis and Panagopoulos (2009) A. Skouroupathis and H. Panagopoulos, Phys.Rev. D79, 094508 (2009), eprint 0811.4264.
  * Blossier et al. (2011) B. Blossier et al. (ETM Collaboration), PoS LATTICE2011, 233 (2011), eprint 1112.1540.
  * (29) ETMC (European Twisted Mass Collaboration), in preparation.
  * Alexandrou et al. (2011) C. Alexandrou, M. Constantinou, T. Korzec, H. Panagopoulos, and F. Stylianou, Phys.Rev. D83, 014503 (2011), eprint 1006.1920.
  * Alexandrou et al. (2012b) C. Alexandrou, M. Constantinou, T. Korzec, H. Panagopoulos, and F. Stylianou, Phys.Rev. D86, 014505 (2012b), eprint 1201.5025.
  * Gracey (2003) J. Gracey, Nucl.Phys. B662, 247 (2003), eprint hep-ph/0304113.
  * Gong et al. (2013) M. Gong, A. Alexandru, Y. Chen, T. Doi, S. Dong, et al., Phys.Rev. D88, 014503 (2013), eprint 1304.1194.
  * Deka et al. (2009) M. Deka, T. Streuer, T. Doi, S. Dong, T. Draper, et al., Phys.Rev. D79, 094502 (2009), eprint 0811.1779.
  * Stathopoulos and Orginos (2010) A. Stathopoulos and K. Orginos, SIAM J.Sci.Comput. 32, 439 (2010), eprint 0707.0131.
  * Stathopoulos et al. (2009) A. Stathopoulos, A. Abdel-Rehim, and K. Orginos, J.Phys.Conf.Ser. 180, 012073 (2009).