According to normative economic models, mainly formulated in mathematical terms, rational decision makers should be indifference when choosing between a safe option and a risky option leading on average to the same payoff (e.g., von Neumann and Morgenstern, 1947). In practical terms, this means that a rational decision maker has no reason to prefer either option when offered choice between e.g., a certain, small reward vs. an uncertain, larger one whose size is five-fold and whose probability of delivery is at "*p"* = 20% (i.e., at the indifference point). However, both human and nonhuman animals are not similar to such "rational" entity, as their instinct will guide their choice towards some kind of a preference: they are generally risk-averse for gains (e.g., Kahneman and Tversky, 1979; Kacelnik and Bateson, 1996), with the notable exception of nonhuman primates, for which the picture is more complicated (Stevens, 2010). To explain this pattern of behavior, it has been proposed that risk-related preferences could reflect the environments in which species evolved and, in particular, their feeding ecology (Heilbronner et al., 2008), leading to "ecologically rational" decisions (Gigerenzer and Todd, 1999). In order to test the above ecological hypothesis, risk preferences were compared in phylogenetically closely related primate species employing two main paradigms.

In the most simple paradigm, the subject is given a series of choices between two options: the "safe" option yields a reward that is constant in amount, whereas the "risky" option yields a reward that varies probabilistically around the mean, with the two options leading on average to the same payoff. Individuals' attitude towards risk is inferred on the basis of their preference for the safe option (indicating risk aversion), for the risky option (indicating risk seeking) or for neither option (indicating risk neutrality) (Kacelnik and Bateson, 1996, 1997). Bonobos (*Pan paniscus*) and chimpanzees (*Pan troglodytes*), two closely related species that evolved behavioral differences possibly as a result of their different ecologies (Wrangham and Pilbeam, 2001), received an experimental schedule whereby they were offered choices between two different upside-down bowls, covering the safe option (always four food items) and the risky option (either one or seven food items with equal probability; Heilbronner et al., 2008). The two species differed markedly in their risk preferences: chimpanzees were risk-seeking, whereas bonobos were risk-averse. Their feeding ecology offers a plausible explanation for this difference: bonobos feed mainly on terrestrial herbaceous vegetation, an abundant and reliable food source, whereas chimpanzees feed primarily on fruit, a more variable food source (Wrangham and Peterson, 1996). Thus, since chimpanzees often rely on more unpredictable food sources than bonobos, this evolutive force may have shaped their behavioral regulations so that to render them tolerant to, if not attracted from, a reward uncertainty. As such, an ecological feature may have led them to be more risk-seeking than their sister species (Heilbronner et al., 2008; Stevens, 2010).

A methodologically similar study conducted on individuals belonging to different lemur species (*Lemur catta*, *Eulemur mongoz*, *Varecia rubra*) showed that, as bonobos, lemurs were clearly risk-averse (MacLean et al., 2012). Subjects were required to choose between two images on a touch-screen, associated to a safe option and to a risky option, respectively. The safe option always led to one food item, whereas the payoff of the risky option varied across two experiments. In a first experiment, the risky option corresponded either to two food items or to zero food items with equal probability (leading to an average payoff of one food item, as the risky option). In a second experiment, the payoff of the risky option was gradually increased across trials up to 7.5 times the safe option. In the first experiment, lemurs strongly preferred the safe option; in the second experiment, half of the subjects switched to risk seeking only when the potential payoff of the uncertain option was at least five times higher than that of the safe option. These results are somewhat puzzling if compared to the findings obtained by Heilbronner et al. (2008) in chimpanzees. However, it can be hypothesized that animals living in a relatively productive environment compared to lemurs, like chimpanzees, can exploit also risky resources, and thus evolve a risk-seeking attitude, without incurring in the danger of starvation. In contrast, for animals living in very harsh environments, like lemurs (that have also evolved several anatomical and behavioral traits as adaptations to their unpredictable habitats; Wright, 1999), risk proneness is not advantageous in the long term and is better to rely on low-quality, yet stable resources (Caraco, 1981; McNamara, 1996).

In a more complex paradigm, Haun et al. (2011) investigated whether, when choosing between a safe and a risky option, the four nonhuman great ape species (*Pan paniscus*, *Pan troglodytes*, *Gorilla gorilla*, and *Pongo abelii*) make decisions based on the expected value, defined as the probability of receiving the reward multiplied by the amount of the reward. In each trial, subjects choose between a safe option, consisting in a small food item hidden under a yellow cover positioned to the right of the subject, and a risky option, consisting in a large food item put in one of four brown bowls placed in a row in front of the subject and hidden under a blue cover. The probability of receiving the reward was manipulated by increasing the number of blue cups covering the four brown bowls (varying from *P* = 100%, when one blue cup covered the brown bowl containing the risky option, to *P* = 25%, when four blue cups covered all the brown bowls), whereas the relative value of the risky option was increased by decreasing the size of the small food item. Overall, apes preferred the risky option, although their preferences were influenced by the expected value. In fact, subjects chose the safe option more often when (i) the safe reward increased in size compared to the risky reward, and (ii) the probability to receive the risky reward decreased. As for species differences, chimpanzees were more risk-seeking than bonobos (as in Heilbronner et al., 2008) also when tested in this more complex paradigm, and orang-utans, whose feeding ecology is somewhat similar to that of chimpanzees (Knott, 1999), were also risk-seeking.

Interestingly, similar differences in risk preferences have been observed in human small-scale societies, possibly as an effect of cultural differences and environmental conditions (Kuznar, 2001; Henrich and McElreath, 2002) that deserves further investigation.

#### **CONTEXTUAL AND SOCIAL FACTORS AFFECTING RISK PREFERENCES IN NONHUMAN PRIMATES**

Several neurophysiological studies in nonhuman primates have employed risk preference tasks to understand whether single neurons track the subjective value rather than the objective value of a chosen option (McCoy and Platt, 2005; O'Neill and Schultz, 2010; So and Stuphorn, 2010; but see Yamada et al., 2013). In a first study, McCoy and Platt (2005) tested rhesus macaques (*Macaca mulatta*) in a visual gambling task and measured the activity of single neurons in the posterior cingulate cortex. Macaques were presented with choices between visual targets offering on average the same reward but differing in reward uncertainty. They had to choose whether directing their gaze to a safe target (offering a 150 ms access to fruit juice) or to a risky target (randomly offering either a shorter or longer than 150 ms access to juice, resulting on average in 150 ms access). Overall, monkeys strongly preferred the risky target and its selection increased with the degree of risk, regardless of the internal state of the subjects. Also neuronal activity increased with increasing variance in payoff of the risky option, mirroring the macaques' risk proneness observed at the behavioral level. Interestingly, macaques continued to prefer the risky option even when the probability of receiving the larger outcome was reduced from 50 to 30% and thus its payoff was smaller than that of the safe option.

In the above study, rhesus macaques were consistently riskseeking and the same pattern was observed also in subsequent studies carried out by the same Authors and in other neurophysiological laboratories (Hayden et al., 2008b, 2010; Long et al., 2009; Watson et al., 2009; O'Neill and Schultz, 2010; So and Stuphorn, 2010; Heilbronner et al., 2011; but see Yamada et al., 2013). Interestingly, macaques' choices are not explained by nonlinear utility functions (as proposed by Lee, 2005) since they preferred an uncertain option, in which the delivery of the larger payoff was unpredictable, to an alternating option, in which the delivery of the larger payoff was predictably alternating across trials (Hayden et al., 2008a). Thus, borrowing the distinction between uncertainty and risk favored in the field of behavioral economics (Knight, 1921; Camerer and Weber, 1992; Tversky and Kahneman, 1992), macaques are not only risk prone, but also uncertainty-seeking.

However, not in all conditions do rhesus macaques exhibit a preference for risky options. In fact, when another macaque sample was tested in a risk preference task under different conditions, their behavior ranged from risk aversion to risk neutrality, but none of them was risk-seeking (Behar, 1961). Thus, although rhesus macaques' ecology may suggest a general predisposition for risk proneness (Goldstein and Richard, 1989; Richard et al., 1989), Heilbronner and Hayden (2013) proposed that macaques' risk preferences are driven by some features of the task design typically used in neurophysiological studies, such as (i) the small stakes involved in these experiments (typically 0.1–0.3 ml of juice); (ii) the large amount of trials (the same decision problem is typically presented hundreds or thousands of times to the same subject); and (iii) the short intertrial intervals (ITIs).

At least for the latter point, an experiment showed that this might be the case. Whereas in McCoy and Platt (2005), where macaques were risk-seeking, the average ITI was 3 s, in other nonhuman animal studies, where individuals were risk-averse (reviewed in Kacelnik and Bateson, 1996), the ITI was much longer (usually 30 s). Thus, Hayden and Platt (2007) presented rhesus macaques with a novel version of the visual gambling task in which the variance of the risky option was kept constant and the ITI varied from 1 s to 90 s. They found interestingly that, as the ITI increased, macaques' preference for the risky option decreased and monkeys turned to risk neutrality at 90 s ITI. To explain this pattern, Hayden and Platt (2007) hypothesized that macaques interpreted the risky option as a certain reward available at a future time and, since the higher payoff may occur on the next trial, the subjective expected utility of the risky option depends on the length of the ITI. Interestingly, when humans were tested with a paradigm as similar as possible to that usually employed with macaques, they were more risk-seeking than in typical oneshot gambling experiments employing questionnaires (Hayden and Platt, 2009).

However, the above factors cannot explain the risk-seeking behavior observed in chimpanzees and orangutans (Heilbronner et al., 2008; Haun et al., 2011), where the stakes involved where comparatively high, the number of trials lower, and the ITIs longer than in the macaque studies. Although the results on chimpanzees appears to be very robust and have been replicated with larger samples (Rosati and Hare, 2012, 2013), it cannot generally be excluded that the different risk preferences obtained in the nonhuman primate studies reviewed so far were due to individual differences. In fact, in rhesus macaques, risk sensitivity appears to be partly determined by the serotonergic system: serotonin depletion increases risk proneness (Long et al., 2009), a finding consistent with recent rodent data (Koot et al., 2012). Similarly, the length polymorphisms of the serotonin transporter gene promotor (known as 5-HTTLPR, the serotonin-transporterlinked polymorphic region) is crucial as well (Watson et al., 2009), in relation to interspecific and intraspecific behavioral variability. Wendland and colleagues (2006) found, in macaque species, that the 5-HTTLPR was responsible for interspecific behavioral variability. In contrast, Chakraborty et al. (2010) proposed that this particular polymorphism had a role in intraspecific variability, which in turn may account for the greater ecological success of 5-HTTLPR polymorphic species. An example of its consequences in the wild is represented by the presumed selective emigration of rhesus macaques over the Himalyan Mountains into China in the early history of the species (Champoux et al., 1997; Heinz et al., 1998). According to Belsky et al. (2009), this particular polymorphism may confer an advantage when dealing with novel, possibly hostile environments. Relative to Indian-derived monkeys, Chinese-hybrid macaques with higher prevalence of the long repeat allele of the 5-HTTLPR show predispositions to aggressive and risk-taking behaviors, as well as lower levels of serotonin as indicated via its metabolite (Champoux et al., 1997; Heinz et al., 1998). Nonetheless, although feeding ecology and inter-individual differences are likely to influence risk preferences, the findings obtained in rhesus macaques underline the importance of carefully controlling all task and environmental parameters when comparing risk preferences among different species.

Finally, as observed in humans (Bault et al., 2008; Ermer et al., 2008; Hill and Buss, 2010), another important factor affecting nonhuman primates' risk preferences seems to be the social context in which the individuals make decisions. To our knowledge, there is only one study evaluating this aspect in nonhuman primates (Rosati and Hare, 2012). Chimpanzees and bonobos were presented with choices between a safe option, yielding an intermediately preferred food item, and a risky option, yielding either a low-preferred or a high-preferred food item, in a competitive context and in a play context. In both contexts an experimenter interacted with the subject before the presentation of the decision-making task: in the competitive context, the experimenter first offered the subject a food item and then, when the subject attempted to take it, immediately pulled it out of the subject's reach; in the play context, the experimenter tickled or chased the subject. Apes' behavior in each condition was compared with a neutral context, in which the experimenter was present but not interacting with the subject. All subjects chose the risky option more in the competitive than in the neutral context, whereas the play context did not increase risk proneness. Probably, an eco-ethological explanation is very likely given that feeding competition and consequent loss of resources is a potential problem for all group-living species. In this frame it can be proposed that, in the competitive context, the salience and attractiveness of the larger option would be increased notwithstanding its uncertainty.

#### **THE EVOLUTIONARY ORIGINS OF BIASES IN DECISIONS UNDER RISK**

When making choices between risky options, humans show the so-called "reflection effect", i.e., the tendency to evaluate gambles in relation to an arbitrary reference point. The same individual can decide differently, being risk-seeking when some options are framed as losses and risk-averse when the same, identical options are framed as gains (Kahneman and Tversky, 1979; Tversky and Kahneman, 1981).

Nonhuman animals apparently share with humans the reflection effect and other behavioral biases (e.g., Waite, 2001; Marsh and Kacelnik, 2002; Shafir et al., 2002). This can be either because of an early emergence of economic biases during evolution, or because of convergent evolution. Only the study of nonhuman primates, our closest relatives, can allow to disentangle the topic and select one between these two hypotheses. To this aim, in recent years a series of studies investigated decision-making under risk in capuchin monkeys (*Sapajus* spp., formerly *Cebus apella*2 ) that, despite 35 million year of independent evolution, show many striking analogies with humans in terms of encephalization index, ontogeny, lifespan, and various cognitive traits (Fragaszy et al., 2004).

In a first study (Chen et al., 2006), capuchins were tested in a token exchange task, in which they were provided with a starting budget of 12 tokens that could be exchanged with one of two experimenters, as they preferred. Preliminary experiments demonstrated that capuchins can behave rationally in this framework: when the two experimenters provided the same amount of two equally preferred different food types, capuchins exchanged a similar amount of tokens with each of them; however, when one experimenter doubled the amount of food provided in exchange for one token or showed two food items and delivered either one or two pieces with the same probability, capuchins reliably shifted their preference towards her, showing that they were able to maximize their payoff. In the main experiment, capuchins were presented with choices between experimenters providing a risky "trade" of either one or two food items with equal probability, but the amount of food initially displayed to the subject was different: one experimenter showed one food item and added a "gain" of one additional food item in half of the trials, whereas the other experimenter showed two food items and subtracted a "loss" of one food item in half of the trials. Although the two experimenters provided on average

2Recent molecular analysis has revealed that capuchin monkeys, formerly identified as the single genus *Cebus*, are two genera, with the robust (tufted) forms (including libidinosus, xanthosternos, apella and several other species) now recognized as the genus *Sapajus*, and the gracile forms retained as the genus *Cebus* (Lynch Alfaro et al., 2012). The nomenclature for *Sapajus* is registered with ZooBank (urn:lsid:zoobank.org:act:3AAFD645-6B09-4C88-B243- 652316B55918). Animals identified as *Cebus apella* in laboratory colonies outside of South America may be any combination of the several species (e.g., *C. apella*, *C. libidinosus*, *C. nigritus*) recognized as separate species since 2001 (Groves, 2001; Fragaszy et al., 2004), but previously considered *C. apella*.

the same payoff, capuchins preferred to exchange their tokens with the first experimenter, although—according to a rational perspective—they should have been indifferent between the two options. These results demonstrate that, as in humans, they chose on the basis of an arbitrary reference point (namely, the initial food amount shown by the two experimenters), therefore preferring the experimenter which was framing the "trade" as a gain.

In a subsequent study (Lakshminarayanan et al., 2011), capuchins were tested with a similar paradigm, presenting them choices between a risky option and a safe option yielding the same average payoff (of two food items) but in two conditions: (i) *Losses*: both experimenters initially displayed three food items, but the first experimenter always delivered two food items, whereas the second experimenter delivered either one or three food items with equal probability; and (ii) *Gains*: both experimenters initially displayed one food item, but the first experimenter always delivered two food items, whereas the second experimenter delivered either one or three food items with equal probability. Overall, capuchins showed a clear-cut evidence of the "reflection effect" since they were risk-seeking when options were framed as losses, and risk averse (although to a lesser extent) when options were framed as gains. Again, decisions appear to be made by subjects relative to their initial reference point.

In sum, the above findings suggest that humans and capuchin monkeys share the reflection effect, as is reported with other behavioral biases (Chen et al., 2006; Lakshminarayanan et al., 2008). However, a very recent "up-linkage" replication of Lakshminarayanan et al. (2011), in which adult humans were tested with exactly the same procedure employed with capuchin monkeys, failed to find a reflection effect (Silberberg et al., 2013). Nonetheless, it should be noted that such a replication may have had a low ecological validity for cognitively sophisticated adult humans, especially because of the repeated interactions with the experimenters, which the participants may have found boring or embarrassing. Future studies should investigate biases in decisions under risk in closely-related non-human primate species with different ecologies (Clutton-Brock and Harvey, 1979; Rosati and Stevens, 2009; Rosati and Hare, 2012) in order to understand whether these behavioral patterns are maladaptive, suboptimal, or instead "ecologically rational" (Todd and Gigerenzer, 2000).

### **RISK ATTITUDES, ENVIRONMENTAL UNCERTAINTY AND ADDICTIVE BEHAVIOR: PERSPECTIVES FROM COMPUTATIONAL NEUROSCIENCE AND EVOLUTIONARY ROBOTICS**

Computational models are a new way of doing science which can be very useful for theorizing about extremely complex systems like vertebrate organisms and their brains. The usefulness of computational models comes largely from two factors: (i) they express hypotheses in a formal, precise, and unambiguous way, so that from those hypotheses a number of detailed predictions can be unequivocally derived which can then be tested through empirical experimentation; (ii) they allow for a degree of direct manipulation on all relevant variables which is unparallelled by naturalistic methods.

The vast majority of computational models deal with the normal functioning of the brain and normal cognitive phenomena, but since the 1990s a number of models have been proposed that address psychiatric and neurological disorders, and recently these models have been raising increasing interest, so that several scholars started to discuss the prospects, challenges, and limitations of *computational psychiatry* (Maia and Frank, 2011; Montague et al., 2012; Huys, 2013). There are many ways in which computational models may help research on decision-making in general and pathological gambling more in particular. Here, we will focus on three different kinds of models: (1) normative (algorithmic) models; (2) neural models; and (3) evolutionary robotics models.

#### **NORMATIVE (ALGORITHMIC) MODELS**

A first class of relevant models is what we can call "normative" or "algorithmic" models. These models derive from the computational reinforcement learning literature (Sutton and Barto, 1998) and are normative because they are based on machine learning algorithms, which prescribe how an agent *should* behave in order to maximize its payoff with future rewards. They became famous in the mid 1990s when it was discovered that the dynamics of dopamine, which is highly involved in motivation and learning (Wise, 2004; Schultz, 2006; Berridge, 2007), as well as in drug addiction, could be modeled by the *reward prediction error* signal postulated in Temporal Difference (TD) reinforcement learning (Barto, 1995; Schultz et al., 1997). The reward prediction error of TD learning is a signal that quantifies "surprise", that is, the difference between expected and actual rewards, and it is used in reinforcement learning models as the learning signal that drives action learning. In a nutshell, the theory holds that an agent continually evaluates the current states (situations) with respect to the reward that it expects to achieve in those states. If it gets more reward than expected, then a prediction error signal is generated that is used to update both its prediction and its action policy, that is the way the animal selects its actions. The idea is that the probability to select an action again, in a given context, is increased if that action leads to more rewards than expected and is decreased if it leads to less reward than expected. Dopamine behaves just as the reward prediction error: its release is triggered by unexpected rewards or unexpected stimuli that predict reward but it is not released when the reward is perfectly predictable and it is inhibited (a deep in dopamine levels occurs) when an expected reward is omitted. This has led to conclude with the hypothesis that dopamine plays the same function of the reward prediction error, within phenomena of reinforcement. Phasic dopamine release would have the role of making the agent learn (1) the value ("saliency") of the stimuli and (2) which are the actions ("strategies") to be deployed in each circumstance in order to maximize future rewards. In mammals, these two roles are attributed to mesolimbic vs. nigrostriatal dopamine pathways, respectively. This theory has guided an enormous amount of empirical research and has received so much empirical support that it is now an important tenet of contemporary neuroscience, and it has become one of the most successful examples of using computational models in the behavioral and brain sciences (e.g., Montague et al., 2004; Ungless, 2004; Wise, 2004; Sugrue et al., 2005; Graybiel, 2008; Glimcher, 2011).

What is most interesting for our purposes is that the reward prediction error hypothesis for dopamine has not only been used to predict and explain behavioral and brain dynamics in normal conditions, but also to explain pathological phenomena.

Moreover, a seminal work by David Redish (2004) used a TD model to explain drug addiction. In particular, the model explained addiction as the consequence of the pharmacological effect that certain drugs of abuse, like amphetamines, cocaine or nicotine, may have on forebrain dopamine circuits. Indeed, these drugs are known to increase dopamine levels upon acute administration. According to Redish's model, the addictive effect of these drugs is associated to specific consequences, due to the dopamine elevation produced by the drug. With natural rewards, a phasic release of dopamine is present only when the reward is not predicted, unexpected. In this perspective, the normal process of reinforcement, produced by any reward, can be cancelled out by accurate predictions. On the contrary, the model postulates that drugs of abuse generate also a pharmacologically-induced dopamine release, a term that cannot be compensated by predictions. Since, in this way, the dopamine prediction error never disappears, as if drug-related pleasure is always "unexpected", the subjective values of the drug related internal states will keep on increasing indefinitely, and the actions that lead to the drug consumption keep on being reinforced, hence becoming a strong habit and thus ultimately resulting in the development of addiction. This model explains several aspects of addiction including, for example, the fact that both drugs and natural rewards are sensitive to effort-related cost, but the reward provided by drugs is much less sensitive than that given by natural rewards. However, one of the key predictions of the theory has been falsified by subsequent research. In particular, the theory predicted that drugs should prevent blocking, i.e., the phenomenon for which a stimulus that predicts a reward, if paired with a new stimulus before presenting the reward, prevents the second stimulus to be conditioned as it stops the learning-inducing dopamine prediction error from occurring. If a drug always produced a dopamine prediction error, as postulated by Redish's model, then the conditioning of the second stimulus should occur, but it does not (Panlilio et al., 2007).

Building on this computational interpretation of drug addiction, Redish et al. (2007) proposed a model that provides a possible explanation of pathological gambling. This model adds to the basic TD prediction error model, which learns the values of states and actions, a second "situation recognition" system that learns to categorize the states. In particular, this system learns to categorize as different states all those situations in which, after having received high rewards, those rewards are not present anymore. Noteworthy, this addition was done to accommodate in the TD framework basic reinforcement learning phenomena related to the extinction of behaviors and their

renewal. However, it provides also an explanation of gambling. Indeed, many pathological gamblers became addict after having experienced an unlikely sequence of wins or a single very high win (Custer, 1984; but see Kassinove and Schare, 2001, for empirically founded doubts on the strength of this big win effect). The model assumes that, when the gambler experiences such a huge success (or the feeling to have almost succeeded, the so called "near miss" effect; Kassinove and Schare, 2001), he forms a very strong and unrealistic expectation that he can win again (or finally; on the similarity in neural processing of wins and near misses, see Chase and Clark, 2010; Winstanley et al., 2011). When the gambler starts to loose, instead of unlearning and cancelling this (false) expectation, by negative reinforcement, his situation recognition system starts to create new "associative" states, namely looking for cues that are supposed to distinguish the winning situation against the loosing ones. Hence, according to this model, pathological gambling results from a misclassification of the situation, with the irrational belief that there are contingencies in which the gambler can win as different from those where he looses. This explanation can account also for two related phenomena: (1) the "hindsight bias" effect, where gamblers analyze their losses and (*post-hoc*) identify which are the cues that differed from the situation when they won, as well as (2) the "illusion of control" phenomenon, in which they believe that they can control an otherwise random situation by identifying and following the right cues that, in their mind, distinguish winning from losing situations (Custer, 1984; Wagenaar, 1988). The most common superstitions of pathological gamblers are thus accounted for.

One limitation of this model is that it tries to explain pathological gambling as a unitary phenomenon with a unique cause, while it is likely that there might be several different causes that underlie this complex behavior, both in the same individual and across different individuals. For example, many pathological gamblers keep on gambling even if they report knowing that they will loose, something that is in contrast with the model (but see the results on cue-induced secondary rewards in rodents and their potential implications for human gambling, discussed in Section Assessment of Gambling Proneness: Clinical and Preclinical Approaches). However, the most important limit of this kind of normative, algorithmic models is that they provide abstract explanations on what computations may go awry in pathological conditions, but they do not explain which are the actual brain mechanisms that may underlie these phenomena: hence the range of phenomena that they can account for and predict is limited. In order to investigate the details of the brain processes that are the basis of the phenomena of study, we need models that simulate those details. This is the province of neural models.

#### **NEURAL MODELS**

Neural models explain a cognitive phenomenon by simulating (with a variable degree of abstraction) neurons and their connections, and making the simulated neural network reproduce the phenomenon. The first models of this kind were called "connectionist" models (McClelland and Rumelhart, 1989): they included very simple neural networks, which were supposed to perform computations in a brain-like manner, but whose structure was not meant to replicate the structure of real brains. More recently, much more biologically realistic models have been developed in computational neuroscience. In these models, different groups of nodes are meant to represent neurons belonging to different parts of the brain, and the connections between the different groups correspond to the connections between those brain areas. The architecture and functioning of the model are thus based on the anatomy and physiology of the same brain areas that are known to be relevant for the phenomenon under study. If the model is able to reproduce the phenomenon, this would give us a detailed explanation on what brain mechanisms may be responsible for it. The plausibility of such an explanation rests on two foundations: (i) how many anatomical and physiological constrains are considered, and how much they are respected; and (ii) how many different phenomena the model is able to account for. Furthermore, the model can be used to derive a number of predictions that can then be tested in humans as well as animal models, through further empirical experiments.

To the best of our knowledge, no neural models have been developed so far to explain pathological gambling, although there is evidence of a role of midbrain dopamine in the coding of reward uncertainty (Fiorillo et al., 2003), thus suggesting an influence of the dopaminergic system on risk-taking behavior.  Briefly reviewing these existing models can provide useful suggestions on how to apply the same methods to the investigation of pathological gambling.

Most of these models deal with the dopaminergic system and its interactions with the basal-ganglia-thalamo-cortical circuits that implement action selection. A notable example is the work of Frank and colleagues on modeling several aspects of Parkinson disease (e.g., Frank et al., 2004, 2007a; Moustafa et al., 2008). Parkinson disease is known to depend on the degeneration of nigro-striatal dopamine cells. This work is based on a detailed model of the basal ganglia-thalamo-cortical circuit that is assumed to implement action selection and reinforcement learning (e.g., Frank et al., 2001). The main idea behind the model is that two sub-systems, a Go and a no-Go system, are present in the basal ganglia, which together implement action selection. In particular, neurons in the basal ganglia are supposed to allow the release of actions in the cortex by selectively disinhibiting a certain action (through the Go system) while inhibiting the others (through the no-Go system). Furthermore, a third structure of the basal ganglia (the subthalamic nucleus) is supposed to dynamically exert a global inhibitory role and to modulate the threshold at which actions are selected depending on the level of cortical conflict. Importantly, neurons belonging to the different systems have different dopamine receptors distributions, with Go neurons having receptors which make dopamine excite the neuron and no-Go neurons that have receptors which make dopamine inhibit the neuron. Through such a model, Frank and colleagues have been able to reproduce and explain a number of detailed behavioral and neural data, and to predict new data that have been empirically verified, such as the effects of dopaminergic medication and of deep brain stimulation of the subthalamic nucleus (a procedure that is known to improve motor symptoms) on different cognitive tasks in Parkinson patients (Frank et al., 2007a), and why medication can lead those patients to develop pathological gambling (Dodd et al., 2005).

In order to explain other facets of this complex behavior and its neural basis, many more details should be added to these models. For example, pathological gambling is known to be associated with dysfunction not only of dopamine, but also of other neuromodulators like serotonin (e.g., Nordin and Eklundh, 1999) and noradrenaline (e.g., Meyer et al., 2004). For this reason, the role of these two neuromodulators should be modeled in future research, possibly by incorporating findings from other computational models that deal with the interactions between these neuro-modulators and dopamine (e.g., Daw et al., 2002). Furthermore, beyond the anomalies in the basal-ganglia and in associated fronto-cortical areas, recent evidence suggests that also deficits in amygdala functioning may be responsible for gambling behavior by significantly reducing loss aversion (De Martino et al., 2010). For this reason, modeling pathological gambling may require modeling the interactions between the amygdala and the basal-ganglia, as done in recent neuro-robotic models of the role of amygdala in conditioning (Mannella et al., 2007, 2008, 2010; Mirolli et al., 2010).

Finally, also factors related to *intrinsic motivations* (i.e., motivations related to novelty, surprise, and competence acquisition: Ryan and Deci, 2000; Baldassarre and Mirolli, 2013) may play a role in pathological gambling. For example, Parkinson patients that develop pathological gambling are distinguished from those that do not in tests that measure impulsivity and novelty seeking (Voon et al., 2007). Recent computational models assume that intrinsic motivations work by hijacking the neural brain systems that underlie also extrinsic motivations, and in particular the dopaminergic system and the action selection system in the basalganglia (e.g., Kakade and Dayan, 2002; Mirolli et al., 2013). Some of these models are detailed neural models very similar to the ones discussed above on dopamine in Parkinson, including basalganglia-thalamo-cortical circuits, the dopaminergic system, and other relevant areas (e.g., Baldassarre et al., 2013; Fiore et al., 2014). Merging the two kinds of models may be a promising way to further understand the brain mechanisms underlying pathological gambling.

#### **EVOLUTIONARY ROBOTICS MODELS**

Evolutionary robotics provide a valuable platform to test evolutionary hypotheses on the ecological pressures behind the emergence of specific behaviors and traits. Such hypotheses, like those already discussed in Sections Rodents as an Animal Model of Gambling Behavior and Risky Choices in Nonhuman Primates: Implications for Human Pathological Gambling with respect to risk attitudes, are often plausible, but also hard to verify directly. They rely on key assumptions about the environment in which the evolution of a given species occurred, and yet it is typically hard to observe with precision the effects of a given ecological variable (e.g., dangers of predation) on the behavior under study (e.g., risk proneness/aversion). Moreover, these assumptions refer to ancestral environments, not present-day ecologies: while there are methods to acquire data on living conditions in ancestral times (e.g., through paleobiology and primate archeology; Haslam et al., 2009), they are bound to deliver incomplete information at best, in spite of substantial research efforts. Recent work has demonstrated the viability and fruitfulness of computational methods, e.g., experimental evolutionary robotics: the basic idea is to let populations of simulated robots evolve under specific ecological pressures, and then observe their behavior with the aim of drawing implications for the understanding of processes in natural organisms faced by similar, uncertainty-based tasks (Da Rold et al., 2011; Saglimbeni and Parisi, 2011). This approach allows to observe how several forms of risk introduced in the evolutionary environment affect choice behavior, both in ecology and in experimental settings.

Moreover, robots are controlled by simple neural networks, whose evolution and effects on behavior can be studied with extreme precision and flexibility: not only recording their activity during behavior, but also "lesioning" a well-adapted neural network and observing the impact on risk-related choices, hence drawing new insights into pathological gambling. These are all key advantages of computational evolutionary models, as opposed to purely mathematical and game-theoretical approaches, for putting forward hypotheses regarding the evolution of certain aspects of risk attitudes in uncertain environments (e.g., McNamara et al., 2013). While mathematical and theoretical models certainly provide valuable contributions to breach the gap between laboratory studies and ecological observations, they lack the opportunities for direct manipulation and experimental observation granted instead by robotics platforms, be they purely simulated or physically implemented.

To the best of our knowledge, no evolutionary (computational) model of pathological gambling have yet been proposed. However, there are several interesting simulations on how risk attitudes in general might have evolved: some of these works have already important implications for our understanding of gambling behavior, and points towards promising research directions. For instance, Niv et al. (2002) used evolutionary computation techniques to evolve near-optimal neuronal learning rules in a simple neural network model of reinforcement learning in bumblebees foraging for nectar. This resulted in a replication of two well-documented choice strategies in these animals: risk aversion and probability matching. Moreover, risk aversion evolved even in a completely risk-less environment. These results suggest that risk-aversion may be a direct consequence of nearoptimal reinforcement learning, with no need to assume further evolutionary constraints, such as the existence of a nonlinear subjective utility function for rewards. Their results were also demonstrated in real-world situations, using experiments in a Kephera wheeled robot, and they dovetail nicely with the evidence on the role of the reward prediction error in determining various choice behaviors (see Section Normative (Algorithmic) Models).

Other models do not explicitly focus on any particular species, but rather try to address general issues pertaining the evolution of risk attitudes. Arbilly et al. (2011) used agent-based evolutionary simulations to investigate an important connection between environmental features, risk-aversion, and the evolution of social learning. They started from the observation that, in environments with significant risks associated to higher value rewards (e.g., an ecology in which the most valuable food is rare and difficult to obtain), the possibility of acquiring such rewards is most likely to require a certain number of failed attempts, before success is achieved. In these circumstances, risk-aversion would lead to neglect such rewards, even if doing so may be sub-optimal in the long run (Real, 1991). However, Arbilly and colleagues noted that this situation also create an important (and often overlooked) evolutionary advantage to social learning over individual learning, since social learners can by-pass the problem of risk aversion by learning where to forage from individuals that have already found food. The results of their evolutionary simulations, which combined a producer–scrounger game with explicit individual and social learning rules for associating different food patch types with experienced reward, confirmed the key role of social learning in similar situations, as an antidote to the adverse effects of risk-aversion in this type of environment. Incidentally, this also provides an explanation to why many species, humans included, continue to rely heavily on social learning even when it produces disastrous effects, e.g., in escape panic scenarios (Helbing et al., 2000). And it also illustrates how this reliance on social learning can be used to produce "contagious gambling": this is precisely what happens when conartists and casinos employ confederates who (falsely) win huge sums, in order to lure unsuspecting potential gamblers into the game.

While the number of computational evolutionary models of risk attitudes is still too limited to permit any universal conclusions on the evolution of this complex suite of behaviors, some important methodological implications stand out, and are worth noticing. This methodology has in fact both advantages and limitations, but what matters is that they tend to be complementary to those exhibited by naturalistic methods. Thus, integrating evolutionary simulations with naturalistic studies has the potential for huge scientific payoffs. With respect to experimental evolutionary robotics (Da Rold et al., 2011; Saglimbeni and Parisi, 2011), advantages of this method include the following ones. First, full observability means that robots' behavior can be observed in extreme detail both "in the wild" (i.e., in the ecological setting where robots evolve), and "in the lab" (i.e., under specific test conditions). Second, there is full control, meaning that all variables can be easily and precisely manipulated, regarding both ecology and test conditions, including the possibility of "counterfactual experiments" (that is, studying how ecological pressures for which no natural correlate is known might affect behavior). Third, there is neurocomputational transparency, in that also the internal dynamics of the robots' control system (e.g., a neural network) are precisely measured (which is not entirely the case for natural, alive organisms). Fourth, individual differences emerge, since robots differ in how they cope with their ecology and in their level of proficiency (also, opening the way to the study of artificial pathologies). Interestingly, non-deterministic responses are present, since evolutionary robots are typically responding in a non-deterministic way, with respect to external stimuli, facilitating comparison with natural, alive organisms (who also do not react always in the same way to identical inputs from the environment). Finally, a potential exists for embodied implementation, since simulated robots are based on simulators of real physical platforms, thus allowing easy implementation in realworld scenarios.

In contrast, the method is mostly vulnerable to the following problems and limitations. First, abstraction, since both the ecology and the artificial laboratory are much simpler than most natural counterparts (and the same is true for the structure of the robot's body and its control system). Second, there is much arbitrariness, since a huge variety of parameters needs to be set by the experimenter, concerning both the ecology, the robot's structure, and the test conditions (and these are likely to have some impact on the resulting behavior). Last, there is need to start small; however, given the number of variables directly controlled by the experimenter and the amount of data obtained, a scalar approach is unavoidable (to understand the results). As mentioned, however, most of these drawbacks can be easily overcome, by allying computational evolutionary models with naturalistic studies (see Sections Risky Choices in Nonhuman Primates: Implications for Human Pathological Gambling and Risk Attitudes, Environmental Uncertainty and Addictive Behavior: Perspectives from Computational Neuroscience and Evolutionary Robotics).

#### **CONCLUSIONS**

In this review, we first discussed how the development of refined operant protocols, to reproduce and to evaluate the gambling proneness phenotype in animal models, is fundamental to increase our understanding of the neurobiological determinants underlying the etiology of pathological gambling and/or to develop new treatment strategies. Then, we surveyed the role of comparative studies on choice behavior in other species, in particular in nonhuman primates, for informing us on the evolutionary origins and cognitive underpinnings of human attitudes towards risk and uncertainty. Finally, we summarized various ways in which computational models can be of assistance in the study of gambling behaviors: while results in this area are still preliminary, we were able to point out several substantial indications originated from combining naturalistic observations and artificial modeling.

Reviewing such diverse studies together is meant to impact on the methodology of future gambling research: while looking at each of these three rich areas of research in isolation is certainly useful, the potential emerging benefits are only compounded by integrating all these methods together. What one learns from an animal model (about the neurobiological underpinnings of pathological gambling) should immediately be verified via computational techniques, and the further predictions generated by that computational model should be tested empirically in natural, alive organisms. Similarly, any evolutionary hypothesis on what adaptive pressures shaped risk attitudes, and generated (possibly as a by-product) gambling behavior, should be verified via computational evolutionary models, which in turn should be informed by naturalistic data coming from ethological studies. Only by bringing to the table both human and nonhuman gamblers, we shall understand what makes us so vulnerable to such a self-destructive behavioral pattern.

#### **ACKNOWLEDGMENTS**

Fabio Paglieri and Elsa Addessi received funding for this research by an ISTC-CNR intramural grant and by an American Society of Primatologists General Small Grant. Elsa Addessi also gratefully acknowledge the support of the PNR/CNR Aging Program 2012-2014. Walter Adriani received support for this research as Principal Investigator of the ERA-net "NeuroGenMRI", and Giovanni Laviola and Walter Adriani received funding by the project "GAMBLING—Fattori psicobiologici alla base di comportamenti di ricerca del rischio, disturbi nel controllo degli impulsi e gioco d'azzardo patologico" from the Department of Antidrug Policies, Presidency of the Council of Ministers (Italy).

#### **REFERENCES**

computational models of striatal-cortical dysfunction. *Biol. Psychiatry* 62, 756– 764. doi: 10.1016/j.biopsych.2006.09.042

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

#### *Received: 30 November 2013; paper pending published: 07 January 2014; accepted: 22 January 2014; published online: 11 February 2014.*

*Citation: Paglieri F, Addessi E, De Petrillo F, Laviola G, Mirolli M, Parisi D, Petrosino G, Ventricelli M, Zoratto F and Adriani W (2014) Nonhuman gamblers: lessons from rodents, primates, and robots. Front. Behav. Neurosci. 8:33. doi: 10.3389/fnbeh.2014.00033*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience.*

*Copyright © 2014 Paglieri, Addessi, De Petrillo, Laviola, Mirolli, Parisi, Petrosino, Ventricelli, Zoratto and Adriani. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Getting a grip on problem gambling: what can neuroscience tell us?

#### **Anna E. Goudriaan 1\*, Murat Yücel 2 and Ruth J. van Holst 1,3**

1 Department of Psychiatry and Amsterdam Institute for Addiction Research, Academic Medical Center, University of Amsterdam, Amsterdam, Netherlands 2 Monash Clinical and Imaging Neuroscience (MCIN) Laboratory, Monash Biomedical Imaging and School of Psychological Sciences, Monash University, Monash, VIC, Australia

3 Centre for Cognitive Neuroimaging, Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, Netherlands

#### **Edited by:**

Paul Vezina, The University of Chicago, USA

#### **Reviewed by:**

Douglas L. Delahanty, Kent State University, USA Susan Ferguson, University of Washington, USA Matthias Brand, University Duisburg-Essen, Germany Marco Leyton, McGill University, Canada

#### **\*Correspondence:**

Anna E. Goudriaan, Department of Psychiatry and Amsterdam Institute for Addiction Research, Academic Medical Center, University of Amsterdam, PO Box 22660, 1100 DD Amsterdam, Meibergdreef 5, 1105 AZ Amsterdam, Netherlands e-mail<EMAIL_ADDRESS>
In problem gamblers, diminished cognitive control and increased impulsivity is present compared to healthy controls. Moreover, impulsivity has been found to be a vulnerability marker for the development of pathological gambling (PG) and problem gambling (PrG) and to be a predictor of relapse. In this review, the most recent findings on functioning of the brain circuitry relating to impulsivity and cognitive control in PG and PrG are discussed. Diminished functioning of several prefrontal areas and of the anterior cingulate cortex (ACC) indicate that cognitive-control related brain circuitry functions are diminished in PG and PrG compared to healthy controls. From the available cue reactivity studies on PG and PrG, increased responsiveness towards gambling stimuli in fronto-striatal reward circuitry and brain areas related to attentional processing is present compared to healthy controls. At this point it is unresolved whether PG is associated with hyper- or hypo-activity in the reward circuitry in response to monetary cues. More research is needed to elucidate the complex interactions for reward responsivity in different stages of gambling and across different types of reward. Conflicting findings from basic neuroscience studies are integrated in the context of recent neurobiological addiction models. Neuroscience studies on the interface between cognitive control and motivational processing are discussed in light of current addiction theories.

**Clinical implications**: We suggest that innovation in PG therapy should focus on improvement of dysfunctional cognitive control and/or motivational functions. The implementation of novel treatment methods like neuromodulation, cognitive training and pharmacological interventions as add-on therapies to standard treatment in PG and PrG, in combination with the study of their effects on brain-behavior mechanisms could prove an important clinical step forward towards personalizing and improving treatment results in PG.

**Keywords: pathological gambling, disordered gambling, reward sensitivity, impulsivity, cue reactivity, response inhibition, review, addictive behaviors**

#### **GAMBLING, COGNITIVE CONTROL, AND IMPULSIVITY: ON GAMBLING AND THE CONCEPT OF SELF-CONTROL**

Pathological gambling (PG) has a relatively stable prevalence in western countries, with estimations ranging from 1.4% (lifetime prevalence) in the USA, to 2% in Canada (Welte et al., 2002; Cox et al., 2005). Prevalence rates are comparable and relatively stable between countries and across survey instruments (Stucki and Rihs-Middel, 2007), with a cumulative rate around 3% for PG and problem gambling (PrG) together.

Diminished cognitive control over the urge to engage in addictive behaviors is a central characteristic of PG. It is central to the *phenomenology* of PG as defined in several of the diagnostic criteria of PG (e.g., unsuccessful efforts to control, cut back, or stop gambling). Defined from a neurocognitive perspective, the overarching notion of cognitive control can be defined as the ability to control one's actions. Cognitive control can be divided in several (sub) processes such as the ability to inhibit automatic responses (referred to as response inhibition, measured by tasks like the stop signal task) and the ability to ignore irrelevant interfering information (referred to as cognitive interference measured by tasks such as the Stroop task). In terms of the verbal representation of cognitive control, the term "impulsivity" is used regularly, to indicate a tendency to act on a whim, to display behavior that is characterized by little or no forethought, reflection, or consideration of the consequences (Daruna and Barnes, 1993). Impulsivity is a multi-faceted construct that often is deconstructed into the concept of "impulsive action", characterized by diminished motor inhibition and "impulsive choice", represented by a propensity to favor immediate rewards over delayed, larger, or more beneficial rewards in decision-making processes (Lane et al., 2003; Reynolds, 2006; Reynolds et al., 2006; Broos et al., 2012). Impaired response inhibition is thought to predispose for impulsive behavior, and diminished cognitive control has been implicated as an endophenotypic vulnerability marker for addictive disorders in recent years.

Numerous self-report and neurocognitive studies in PG indicate increased impulsivity on measures such as the Barratt Impulsiveness Scale, or Eysenck's and Impulsiveness Questionnaire (Eysenck et al., 1985) and diminished cognitive control as evidenced in diminished response inhibition, cognitive interference, and delay discounting tasks (for reviews see: Goudriaan et al., 2004; Verdejo-Garcia et al., 2008; van Holst et al., 2010a,b). Clinically, the diminished control over one's own behavior could lead to a higher vulnerability to develop PrG or PG, since for instance a diminished control to inhibit responses (response inhibition) could be associated with a more fast progression into PrG due to the diminished ability to stop gambling when one's money runs out. Similarly, a diminished cognitive interference ability could lead to a diminished ability to ignore cues for gambling in the environment. For example, experiencing high cognitive interference could lead to a higher responsivity towards gambling advertisements, which could lead to a higher likelihood of engaging in gambling, whereas diminished cognitive control could result in diminished ability to stop gambling despite high losses.

Several reviews have already been published with a focus on cognitive control or impulsivity studies in PG (van Holst et al., 2010a,b; Conversano et al., 2012; Leeman and Potenza, 2012). This review therefore focuses on more recent neurocognitive and neuroimaging studies that have been published in PG and PrG. Specifically, this review also focuses on neuroimaging studies of motivational aspects (e.g., cue reactivity), cognitive functions (e.g., impulsivity), and on neuroimaging studies addressing the interaction between cognitive and motivational processes.

Whereas a clear definition of PG is present, fulfillment of the (usually latest version of the) DSM diagnostic criteria for PG, there is no clear definition for PrG. Usually, PrG refers to a less severe form of PG, or is used when no clinical diagnosis can be determined, due to the administration of questionnaires instead of structured clinical interviews. Some studies define PrG by a score of 5 or higher on the South Oaks Gambling Screen (SOGS) or by a score of 3 or higher on a short version of the SOGS (Slutske et al., 2005). In other studies gamblers who are in treatment for problematic gambling, and fulfill up to four criteria of the PG criteria, are defined as problem gamblers (Scherrer et al., 2005), or the entire studied group is defined as "problem gamblers" when not all of the participants who are in treatment fulfill five or more of the PG criteria (e.g., de Ruiter et al., 2012). In this review therefore, PrG is used, when no information is given on DSM diagnosis of PG, but when questionnaire data indicate that PrG is present.

As concluded in Conversano et al. (2012), several studies indicate diminished cognitive control in PG as evidenced in stopsignal tasks, Go-NoGo tasks, and also in Stroop task performance. Ledgerwood et al. (2012) however assessed response inhibition with a Stroop and stop signal task, and reported no differences between pathological gamblers and controls on these tasks, but differences were present in planning tasks (Tower of London) and in cognitive flexibility (Wisconsin Card Sorting Test). As the sample included both community-recruited pathological gamblers (not in treatment) and treatment-seeking pathological gamblers, differences with other studies may be related to a less severe cognitive profile in non-treatment seeking pathological gamblers. Indeed, in another study by the same group lower impulsivity scores (Barratt impulsivity Scale), lower past-year illegal behaviors, lower depression and dysthymic disorders, and lower preoccupation with gambling were present in community-recruited pathological gamblers vs. pathological gamblers in treatment (Knezevic and Ledgerwood, 2012).

Despite the number of neuropsychological studies indicating diminished cognitive control, the number of neuroimaging studies focusing on the neural mechanisms underlying diminished cognitive control is very limited and therefore all neuroimaging studies on cognitive control are discussed here. In a study by Potenza et al. a Stroop task was administered in an fMRI study in 14 pathological gamblers and 13 healthy controls (HCs) (Potenza et al., 2003a). Diminished BOLD responsivity in the left ventromedial PFC and in the superior OFC was reported in pathological gamblers compared to HCs, despite a lack of behavioral differences. This lack of behavioral differences may have been related to the modified version of the Stroop that was used: silent naming of the colors of the letters and behavioral performance measured by self-report of the participants after performing the Stroop task. In a recent study by de Ruiter et al. (2012), diminished neural responsivity after failed inhibitions was found in the anterior cingulate cortex (ACC) in 17 problem gamblers compared to 17 HCs. Of note, reduced activity was also observed following successful inhibitions in similar regions (right dorso-medial PFC bordering on ACC) HCs. In this study similar to the study by Potenza et al.—no behavioral differences were found for the PrG group compared to the HCs, which may be related to power issues due to the smaller sample sizes of fMRI studies in PrG and PG compared to neuropsychological studies. Both these fMRI studies on cognitive control in PG and PrG show that diminished functioning of several prefrontal areas and of the ACC indicate that cognitive-control related brain circuitry functions are diminished in PG and PrG compared to HCs. These results implicate that diminished frontal functions may contribute to the pathophysiology of PG and PrG, in which diminished control over gambling behavior is central.

Another line of studies shows that impulsivity also plays an important role as vulnerability factor for the development of PrG. Several longitudinal studies in adolescents and adults from a research group from Montreal in Canada show that level of impulsivity is a predictor of both gambling and of PrG (Vitaro et al., 1997, 1999; Wanner et al., 2009; Dussault et al., 2011). Specifically, increasing impulsivity levels were associated with higher levels of PrG (Vitaro et al., 1997). In one of the more recent studies, a positive predictive link between impulsivity at age 14 and depressive symptoms and gambling problems at age 17 was present (Dussault et al., 2011). In another study using two male community samples, behavioral disinhibition and deviant peers were related to PrG, but also to substance use and delinquency, indicating similar risk factors for vulnerability to several externalizing problem behaviors (Wanner et al., 2009). These studies focused on adolescents and the predictive role of impulsivity for PrG; very recently two large-scale longitudinal birth cohort studies, investigated the role of impulsivity in early childhood and PrG during adulthood. In one of these studies (Shenassa et al., 2012), psychologists rated impulsive and shy/depressed behaviors at age 7, and related this to life-time self-reported PrG as adults, in a follow-up. Whereas impulsive behavior at age 7 predicted PrG, shy/depressed behavior did not predict PrG in adulthood, in this US based cohort of 958 offspring from the Collaborative Perinatal Project. In a large birth cohort study from Dunedin, New Zealand, temperament was assessed at age 3, and disordered gambling was assessed in this cohort when aged 21 and 32. Remarkably, children with (behaviorally and emotionally) undercontrolled temperament when aged 3 years, were more than twice as likely to evidence disordered gambling in adulthood, compared to children who were well-adjusted at age 3. This relationship was even stronger in boys compared to girls (Slutske et al., 2012). Several other studies show that impulsivity is also a vulnerability marker for engaging in gambling (Pagani et al., 2009; Vitaro and Wanner, 2011).

In conclusion, from this line of studies, there is strong evidence that impulsivity and diminished behavioral control play an important promoting role from the engagement in gambling to the development and persistence of at-risk gambling and PrG.

Given this crucial role of cognitive control in promoting gambling and PrG, evidenced from the birth cohort studies, neurocognitive studies, more neuroimaging studies in PrG and PG should focus on cognitive control, in order to illucidate what neurophysiological mechanisms may underly diminished cognitive control in problematic gambling. Thus, studying interactions between (novel) psychological, pharmacological, or neuromodulation interventions in PG, and their effect on the neurocircuitry of cognitive control in PG, is a very relevant venue for future neuroimaging and clinical intervention studies in PG (detailed in the Discussion section).

#### **RIGHT ON CUE? CUE-REACTIVITY STUDIES IN PROBLEM GAMBLING**

Compared to the small number of neuroimaging studies on cognitive control or impulsivity in PG and PrG, the topic of the neural mechanisms of cue-reactivity in PG and PrG is relatively well-studied. Five neuroimaging studies on cue-reactivity in PG and PrG (Potenza et al., 2003b; Crockford et al., 2005; Goudriaan et al., 2010; Miedl et al., 2010; Wölfling et al., 2011) and several studies focusing on cue reactivity relating to subjective craving and/or peripheral physiological responses in PrGs are present (Freidenberg et al., 2002; Kushner et al., 2007; Sodano and Wulfert, 2010). For the purpose of this review, we focus on the neuroimaging findings.

Of the five neuroimaging studies in PG and PrG related to cue reactivity, the first (Potenza et al., 2003b) used a cue reactivity paradigm consisting of videos designed to evoke emotional and motivational antecedents to gambling. In these videos, actors mimicked emotional situations (e.g., happy, sad), after which the actor described driving to or walking through a casino and experiencing the feeling of gambling. In this study, timeframes in which the participants experienced craving were analyzed for 10 pathological gamblers compared to eleven HCs. In all cases, this was before actual gambling cues were present and in response to the actors' descriptions of the emotional situation (i.e., gambling scenarios). Less activation in the cingulate gyrus, (orbito) frontal cortex (OFC), caudate, basal ganglia, and thalamic areas was present in the 10 pathological gamblers compared to the 11 HCs. In another study using gambling-related videos to elicit cuereactivity, 10 pathological gamblers and 10 HCs were compared on brain responsivity to these gambling-related videos compared to watching nature-related videos (Crockford et al., 2005). Higher activation in dorsal prefrontal areas, inferior frontal areas, the parahippocampal areas, and occipital lobe was found in pathological gamblers compared to HCs. In a subsequent fMRI cuereactivity study, Goudriaan et al. (2010) found elevated activity of similar regions when comparing 17 pathological gamblers vs. 17 HCs using gambling-related and gambling unrelated photos. In this last study, a positive relationship was found between subjective craving for gambling in pathological gamblers and activity of the frontal and parahippocampal regions when viewing gambling pictures vs. neutral pictures. In an EEG study by Wölfling et al. (2011), 15 pathological gamblers were compared to 15 HCs on EEG responsivity to gambling pictures compared to neutral, positive and negative emotional pictures. Compared to HCs, pathological gamblers showed significantly larger late positive potentials (LPPs) induced by gambling stimuli when compared to neutral stimuli, but displayed comparable LPPs towards negative and positive emotional pictures. In contrast, in HCs there was a larger response towards positive and negative stimuli compared to both neutral and gambling stimuli. Higher LPPs were present in the parietal, central, and frontal electrodes in PGs compared to HCs, interpreted as a higher overall psychophysiological responsivity towards gambling stimuli in pathological gamblers.

Finally, in an fMRI study comparing brain responsivity towards high-risk vs. low-risk gambling situations in 12 problem gamblers vs. 12 HCs, problem gamblers showed an increased BOLD response in thalamic, inferior frontal, and superior temporal regions during high-risk trials, whereas a signal decrease in these regions during low-risk trials was present. The opposite pattern was observed in the non-problem gamblers (Miedl et al., 2010). The authors argue that this frontal-parietal activation pattern during high-risk trials compared to low-risk trials in problem gamblers reflects a cue-induced addiction memory network, triggered by gambling-related cues. The findings of this study implicate that high-risk wagers may be attractive to problem gamblers, eliciting cue-reactivity and craving, whereas low-risk wagers, representing a high chance to win a smaller amount of money may elicit higher reward expectations in nonproblem gamblers. A possible interpretation of the diminished responsiveness to low-risk wagers in the problem gamblers, may be that this is due to a diminished reward sensitivity due to a blunted brain response to low-risk monetary rewards.

When summarizing the neuroimaging studies on cuereactivity in PG and PrG, a convergent picture emerges regarding the studies that employ gambling pictures or gambling movies in which actual gambling scenes are included. In these studies, increased responsiveness in fronto-striatal reward circuitry and brain areas related to attentional processing towards gambling stimuli is present in pathological gamblers/problem gamblers compared to HCs (Crockford et al., 2005; Goudriaan et al., 2010; Miedl et al., 2010; Wölfling et al., 2011). In contrast, in the one study employing stress-provoking situations, followed by verbal descriptions of wanting to engage in gambling, diminished responsiveness in fronto-striatal circuitry was found (Potenza et al., 2003b). These findings imply that cue-reactivity elicited by gambling stimuli engages reward- and motivation related circuitry thus potentially enhancing the chance of engaging in gambling. On the other hand, negative mood states induced by stressful situations may induce a relatively diminished activity in the same reward- and motivation related circuitry in pathological gamblers, which in turn may elicit craving for gambling, in order to relieve this depletion in reward experience (or anhedonia). The one finding of diminished fronto-striatal reactivity (Potenza et al., 2003b) relates to the "allostatic" negative emotional state (e.g., dysphoria, anxiety, irritability) reflecting a motivational withdrawal syndrome state as hypothesized by Koob and Le Moal and as recently integrated in a review by Koob and Volkow (2010). The remainder of the neuroimging findings in response to gambling cues relate to the preoccupation and anticipation of engaging in addictive behavior, characterized by craving. Thus, both increased responsivity in the brain's reward system to gambling cues as well as decreased responsivity of the reward system to stress-provoking cues in anticipation of gambling could lead to craving and (relapse in) gambling. This combination is also consistent with a behavioral study by Kushner et al. (2007), in which diminished cue reactivity was reported after negative mood induction.

Together, these cue-reactivity studies and addiction theories indicate that an important area to investigate in PG and PrG is the link between positive mood states and negative mood states/ stress reactivity, and both craving for gambling and gambling behavior. From the studies comparing gambling stimuli to neutral stimuli, increased frontal-striatal reactivity relating to increased cue-reactivity is evident. However, the role of the amygdala and negative emotional mood states (i.e., as a "motivational withdrawal syndrome") in inducing craving and relapse in PG and PrG should receive additional research attention.

The "withdrawal/negative affect" part of the addiction cycle, which consists of re-engagement in addictive behaviors due to withdrawal effects or negative affect, in order to diminish withdrawal and/or negative affect (Koob and Volkow, 2010) can be linked to the emotionally vulnerable problem gambler, one of the three subtypes of problem gamblers, as proposed by Blaszczynski and Nower (2002) and characterized by stress reactivity and negative mood as a pathway to PrG (Blaszczynski and Nower, 2002). The "preoccupation/anticipation" part of the addiction cycle, which is characterized by enhanced attention and cuereacitivity towards addiction-relevant cues, links to the "antisocial, impulsivist" subgroup of problem gamblers as defined by Blaszczynski and Nower (2002). They describe the latter subgroup of problem gamblers as characterized by higher impulsivity, and clinical impulsive behaviors such as ADHD and substance abuse, which promote and fasten processes of classical and operant conditioning in developing PrG (Blaszczynski and Nower, 2002).

So far, these three subtypes of pathological gamblers have hardly been studied empirically: Ledgerwood and Petry investigated these three gambling subtypes within a group of 229 pathological gamblers, which were based on self-report questionnaires. Although the subtypes differed on PrG severity, subtyping did not predict a differential treatment response. Several behavioral studies indicate differences between problem gamblers and HCs in stress reactivity. For instance, in a recent study (Steinberg et al., 2011), uncontrollable noise (stress induction) led to diminished craving for gambling in problem gamblers, whereas it increased craving for alcohol use in problem gamblers, alcohol use disordered participants and HCs. This finding, although in a small sample (12 participants in each clinical group), indicates that differential changes in craving for different addictive behaviors may result from stress (here: gambling vs. alcohol use). In a selfreport study (Elman et al., 2010) the only measure positively related to gambling urges in problem gamblers was a daily stress inventory, indicating a positive relation between stress and craving for gambling. Interestingly, in a recent pilot-study with a pharmacological challenge with yohimbine, significant left amygdala activation in response to yohimbine across all four PG subjects was observed, whereas this effect was not present in the five HCs, suggesting pharmacologically induced stress sensitization in the brain of pathological gamblers. Thus, studies focusing on the relation between stress reactivity and gambling cues, gambling urges, and gambling behavior are needed, in order to elucidate the etiology of both the withdrawal/negative affect (stress reactivity) and the motivation/anticipation (cue reactivity) part of the addiction cycle in PG and PrG. Based on the results of these behavioral and physiological studies, and the negative finding from the one study focusing on the three subtypes of pathological gamblers (Ledgerwood and Petry, 2010), it is clear that more (neuro)biological research is needed into subtyping of PG. It may well be that one problem gambler subtype is identified for whom gambling urges emerge through negative affect (with amygdala circuit abnormalities as a neural mechanism) and another problem gambler subtype where gambling urges emerge through gambling cues (with a hyperactive orbitofrontostriatal circuitry as underlying neural mechanism). This subtyping of pathological gamblers based on endophenotype (negative affect/stress reactivity vs. positive affect/gambling cue reactivity) could then be compared to the three subtypes as defined by Nower and Blaszczynski (2010): behaviorally conditioned, emotionally vulnerable and antisocial-impulsive.

Although a minimal number of neuroscience studies on stress reactivity in PG and PrG exist, a related issue is the presence of either increased or decreased reward sensitivity in neuroimaging studies in PG and PrG, and these studies will be discussed next.

A popular hypothesis of addiction is that substance dependent persons suffer from a reward deficiency syndrome, which makes them pursue strong reinforcers (i.e., drugs) to overcome this deficiency (Comings and Blum, 2000). The first fMRI studies in PG focusing on reward processing have reported results consistent with such decreased reward sensitivity. For example, in response to monetary gains compared to monetary losses pathological gamblers showed blunted activation of the ventral striatum and ventral prefrontal cortex (Reuter et al., 2005). Similarly attenuated activation of ventral prefrontal cortices was present in with a cognitive switching paradigm where problem gamblers could win or lose money dependent on their performance (de Ruiter et al., 2009).

Recently, more detailed studies investigating *different phases of reward processing* have been conducted. Using a modified monetary incentive delay (MID) task (Knutson et al., 2000) in which subjects have to make speeded responses to acquire points/money or to prevent losing points/money, pathological gamblers showed attenuated ventral striatal responses during reward anticipation as well as in response to monetary wins (Balodis et al., 2012; Choi et al., 2012). Whereas results from these two studies are consistent with the reward deficiency hypothesis, other fMRI studies have found increased responses in anticipation of reward or after receiving rewards in fronto-striatal reward related brain areas.

For instance, using a probabilistic choice game to model anticipatory processing, pathological gamblers showed greater dorsal striatum activity during anticipation of large rewards compared to small rewards (van Holst et al., 2012c). In addition, pathological gamblers compared to controls showed higher activity in the dorsal striatum and OFC for gain-related expected value. Hyperreactivity after receiving monetary rewards in high risk bets was also found in the medial frontal cortex with an ERP study using a black jack task (Hewig et al., 2010). In a fMRI study by Miedl et al. (2012) subjective value coding for delay discounting and probability discounting in pathological gamblers and HCs was investigated. The subjective value for each task was computed for each participant individually and correlated with brain activity in the ventral striatum. Compared to controls, pathological gamblers showed a greater subjective value representation in the ventral striatum on a delay discounting task, but a reduced subjective value representation during the probabilistic discounting task. This indicates that pathological gamblers evaluate values and probabilities differently than controls. These results suggest that abnormal choice behavior with regard to future delayed rewards in problem gamblers could be related to different value coding.

At this point it is unresolved whether PG is associated with hyper- or hypo-activity in the reward circuitry in response to monetary cues, a similar issue that consists in the substance dependence literature (Hommer et al., 2011). Several methodological issues could explain the hyper- or hypo-activity findings in the reward circuitry found in the above mentioned studies. For example, in the MID task subjects have to respond as quickly as possible to a target to obtain a reward whereas in the task used by van Holst et al. (2012c) subjects have no influence on their wins or losses. This difference in control over the task outcomes could have influenced the striatal responses during the task. Furthermore, the graphic designs of the two studies also differed markedly; the MID task used in the study by Balodis et al. (2012) used non-monetary abstract pictograms, the task by van Holst et al. (2012c) featured familiar playing cards and Euro coins and bills. These gambling associated cues may elicit cue reactivity responses leading to hyperresponsivity in the striatal regions (see for a discussion: Leyton and Vezina, 2012; van Holst et al., 2012c,d). This hypothesis regarding diminished reactivity of the striatum in the absence of addiction relevant cues, and an overactivity of the striatum in the presence of addiction relevant cues was recently reviewed in depth by Leyton and Vezina (2013).

The reward deficiency hypothesis of addiction has received considerable support from PET studies measuring dopamine functioning, consistently showing lower dopamine D2/D3 receptor binding potential in drug dependent subjects (Martinez et al., 2004, 2005, 2011; Volkow et al., 2004, 2008; Lee et al., 2009). Whether this D2/D3 receptor binding potential underlies PG is still unclear because PET techniques have only recently been utilized in PG. Currently, no significant differences in baseline DA binding in pathological gamblers compared to HCs seems to be present (Linnet et al., 2010; Joutsa et al., 2012; Boileau et al., 2013) but other studies indicate a positive correlations between DA binding and gambling severity and impulsivity (Clark et al., 2012; Boileau et al., 2013). In addition, A PET study measuring DA activity during the Iowa gambling task found that DA release in pathological gamblers was related to excitement (Linnet et al., 2011a) and poor performance (Linnet et al., 2011b). Overall these results do suggest a role for abnormal DA binding in PG but not to the same extent as that found in drug addiction in which clear diminished binding potentials are consistently reported (Clark and Limbrick-Oldfield, 2013). Missing from the literature are studies measuring more stable baseline DA synthesis capacity: existing studies have only focused on aspects related to highly state dependent DA D 2/3 receptor availability. Studies measuring DA synthesis capacity could test the hypothesis of a higher DA synthesis capacity in PG and PrG. Higher DA synthesis could lead to higher dopaminergic *reactivity* when confronted with addiction related cues (e.g., games, money, risk). Furthermore, PG studies directly manipulating DA and measuring fMRI BOLD responses during reward processing could provide important information about the causal role of DA in PG.

An alternative hypothesis, next to the reward deficiency hypothesis for PG and PrG is that, similar to substance use disorders (SUDs; Robinson and Berridge, 2001, 2008), pathological gamblers and problem gamblers suffer from an enhanced incentive salience for gambling related cues. This enhanced incentive salience for gambling cues could be so strong that it overrides incentive salience of alternative sources of reward, leading to an imbalance in incentive motivation. To test whether pathological gamblers would suffer from an overall reward deficiency or from an imbalance in incentive salience, Sescousse et al. (2013) compared neural responses to both financial gains and to primary rewards (erotic pictures) in pathological gamblers and HCs. In line with the latter hypothesis, hypo-reactivity was observed for the erotic cues, in contrast with normal-reactivity to the financial rewards, indicating an imbalanced incentive salience attribution in PG. Taken all the above studies together, at this point it seems most likely that pathological gamblers are not suffering from a reward deficiency in general but that pathological gamblers have a different appraisal of gambling related stimuli, presumably caused by enhanced incentive salience of gambling related stimuli.

Recently fMRI studies have focused on specific gambling related cognitive biases. This is important because problem gamblers often display a number of cognitive biases regarding gambling games (Toneatto et al., 1997; Toneatto, 1999; Clark, 2010; Goodie and Fortune, 2013). For example, gamblers are known to falsely believe that they can influence outcome probabilities of games ("illusion of control") (Langer, 1975). Various intrinsic features of games of chance promote these biases (Griffiths, 1993), as for example "near-miss" events (Kassinove and Schare, 2001). These near-wins or near-miss outcomes (which are actually losses) occur when two reels of a slot machine display the same symbol and the third wheel displays that symbol immediate above or below the pay-off line. A study investigating near-miss effects in problem gamblers found that brain responses during near-miss outcomes (compared to full-miss outcomes) activated similar brain reward regions such as the striatum and insular cortex as during win outcomes (Chase and Clark, 2010). Habib and Dixon (2010) found that near-miss outcomes lead to more win-like brain responses in pathological gamblers, whereas HCs activated brain regions associated with losses to a larger extent. These studies contribute to a better understanding of the addictiveness of gambling games and its underlying neuronal mechanism.

#### **CAN ENHANCED SALIENCE FOR GAMBLING RELATED STIMULI LEAD TO LOSS OF CONTROL OVER BEHAVIOR?**

An influential and empirically grounded neurobiological model for substance dependence, the Impaired Response Inhibition and Salience Attribution (I-RISA) model, postulates that repeated drug use triggers a series of adaptations in neuronal circuits involved in memory, motivation, and cognitive control (Volkow et al., 2003). If an individual has used drugs, memories of these events are stored as associations between the stimulus and the elicited positive (pleasant) or negative (aversive) experiences, facilitated by dopaminergic activation caused by the drug of abuse. This results in an enhanced (and long-lasting) salience for the drug and its associated cues at the expense of decreased salience for natural reinforcers (Volkow et al., 2003). In addition, the I-RISA model assumes loss of control (disinhibition) over drugs due to enhanced salience and pre-existing deficiencies (as discussed in part 1 of the review), which renders individuals suffering from addictive disorders vulnerable to relapse into addictive behavior.

In addictive disorders including PG, there is evidence that both affective and motivational systems are more sensitive to addiction relevant material. For example, studies have shown that addiction related cues attract more attention than other salient stimuli, a phenomenon known as "attentional bias" (McCusker and Gettings, 1997; Boyer and Dickerson, 2003; Field and Cox, 2008). As discussed in the "cue reactivity" section of this review, in problem gamblers, enhanced brain responsiveness towards gambling related cues ("cue reactivity") has also been found in brain areas related to motivational processing and cognitive control (amygdala, basal ganglia, ventrolateral prefrontal cortex and dorsolateral prefrontal cortex; Crockford et al., 2005; Goudriaan et al., 2010).

As discussed in the first section of this review, PG is associated with impaired cognitive control. However how cognitive control interacts with motivational processes is still subject of investigation. Just recently, studies have started to test the interaction between cognitive control and salience attribution in PG. In one of our recent studies, we employed a modified Go/NoGo task by including affective stimulus blocks (gambling, positive and negative), in addition to the standard affectively neutral block in problem gamblers and HCs (van Holst et al., 2012b). Subjects were requested to respond or withhold a response to specific types of pictures with a different emotional loading, allowing the investigation of the interaction between motor inhibition and salience attribution. Whereas we found no behavioral differences on neutral response inhibition trials, problem gamblers compared to controls showed greater dorsolateral prefrontal and ACC activity. In contrast, during gamble and positive pictures problem gamblers made less response inhibition errors than controls and showed reduced activation of the dorsolateral prefrontal and ACC. This study indicated that pathological gamblers rely on compensatory brain activity to achieve similar performance during neutral response inhibition. However, in a gambling-related or positive context response inhibition appears to be *facilitated*, as indicated by lower brain activity and fewer response inhibition errors in pathological gamblers. Data from this Go/NoGo study was further analyzed to test the effect of affective stimuli on functional connectivity patterns during the task (van Holst et al., 2012a). As expected, adequate response inhibition was related to functional connectivity within the sub-regions of the dorsal executive system as well as on functional connectivity between the dorsal executive and the ventral affective system in both HCs and problem gamblers. Compared to HCs, problem gamblers showed a stronger positive correlation between the dorsal executive system and task accuracy during inhibition in the gambling condition. These findings suggest that increased accuracy in pathological gamblers during the gambling condition was associated with increased connectivity with the dorsal executive system (van Holst et al., 2012a). It seems likely that DA function plays an important role in these findings. Salient stimuli enhance DA transmission in the mesolimbic system (Siessmeier et al., 2006; Kienast et al., 2008) and DA is known to modulate prefrontal cortex functioning (Robbins and Arnsten, 2009).  More research is needed to further clarify the interaction between motivation, DA and cognitive control in PG. In the earlier mentioned review by Leyton and Vezina (2013), a model is proposed that integrates the influence of these opposite striatal responses on the expression of addictive behaviors. Central to his model is the idea that low striatal activity leads to an inability to sustain focussed goal-directed behavior, whereas in the presence of high striatal activity (when drug cues are present) a sustained focus and drive to obtain rewards is present. The findings reviewed above (van Holst et al., 2012a,b) fit this model well: better performance was present in problem gamblers in the positive and gambling conditions, and more functional connectivity was found with the dorsal executive system in problem gamblers in the gambling condition. This could be an indication of normalization in probleml gamblers of the underactive striatal system, in the presence of salient motivational cues in the positive and gambling Go/NoGo conditions.

It is clinically relevant to further investigate whether increased activity in the reward system indeed has the effect of transiently restoring prefrontal cortex functioning in problem gamblers. This could be tested by pharmacological challenges or by enhancing activity in the reward system more locally, for example by using real time-fMRI neurofeedback (deCharms, 2008) or Transcranial Magnetic Stimulation (TMS; Feil and Zangen, 2010). However, we suggest that enhanced salience to rewarding stimuli could also lead to *impaired* task performance. For example, when too much attention is allocated to salient stimuli, this can result in attenuated executive control recourses (Pessoa, 2008). Enhanced reward seeking behavior and enhanced responsiveness to potential rewards could therefore be an important concept in understanding why especially on tasks with contingencies gamblers show diminished cognitive performance (Brand et al., 2005; Goudriaan et al., 2005, 2006; Labudda et al., 2007; Tanabe et al., 2007; de Ruiter et al., 2009).

#### **SUMMARY NEUROIMAGING FINDINGS: SELF-CONTROL, CUE-REACTIVITY, REWARD SENSITIVITY AT DIFFERENT STAGES OF GAMBLING, AND THE INTERACTION BETWEEN SELF-CONTROL AND MOTIVATIONAL URGE**

When trying to reach an overarching conclusion with regard to the studies reviewed, it is clear that for some topics, consistent findings have been established over the years. For instance, the notion of increased impulsivity in PG and PrG is firmly established and the first neuroimaging studies show that this heightened impulsivity is accompanied by diminished prefrontal and ACC functioning. It is clear that the field of cognitive functions in PG needs more neuroimaging studies to investigate what cognitive functions are most affected. Neuroimaging cuereactivity studies indicate that when gambling cues are present, the motivational system of the brain is overactive in PG and PrG, as evidenced in higher parahippocampal, amygdala, basal ganglia, and OFC activation. With regard to either enhanced neural reward sensitivity or diminished reward sensitivity, the first studies seem to indicate that whereas enhanced activation of the brain's reward circuitry is present in *anticipation of* winning or in experiencing risky gamble situations, diminished reward responsiveness is present in this same circuitry *after* winning and/or losing money. Finally, the interaction of cue-reactivity and cognitive control suggests that the activation of the cognitive control system in problem gamblers may be enhanced by activating the motivational circuit. However, this finding is in need of replication, and the role of DA in facilitating or diminishing cognitive control in PG deserves further study.

#### **CLINICAL IMPLICATIONS**

Cognitive behavioral therapy (CBT) for problem gamblers focuses on behavioral and cognitive interventions to curb the motivational lure of gambling behavior and has been shown to be effective in the treatment of PG (Petry, 2006; Petry et al., 2006), although relapse is still high, ranging around 50–60% in treatment studies, with rates of continuous abstinence for a year as low as 6% (Hodgins et al., 2005; Hodgins and el Guebaly, 2010). Thus, there is still room for major improvement in treatment results for PG/PrG. CBT focuses on enhancement of cognitive control over gambling, and a change in the behavior of engagement in gambling due to encountering gambling cues or experiencing craving. Specific techniques used in CBT for PG and PrG include learning coping strategies, applying stimulus control strategies, and handling high risk situations by implementing behavioral strategies, for instance on emergency cards. Thus, in CBT for PG and PrG, a substantial part of the intervention depends on engagement of executive functions by implementing behavior and emotion regulation strategies. In other psychiatric disorders, neuroimaging studies have shown that differences in pre-treatment brain functioning can predict CBT treatment effects. For instance, better frontal-striatal brain functions during a response inhibition task resulted in better response to CBT in post traumatic stress disorder (Falconer et al., 2013). Increased activity at baseline in the ventromedial PFC as well as valence effects in emotional tasks (e.g., social threat tasks) in the (anterior) temporal lobe, ACC and DLPFC promote treatment success in major depressive disorder (Ritchey et al., 2011) and in social anxiety disorder (Klumpp et al., 2013). These findings not only suggest that brain functions may be important new biomarkers for indicating the chance for treatment success with CBT, but also point to the potential value of new interventions targeting neurobiological vulnerabilities of PG and PrG. By studying brain functions that are biomarkers for CBT success in PG and subsequently improving these brain functions by neuromodulation or pharmacological interventions, treatment results for PG and PrG may improve.

Several interventions targeted at neurobiological vulnerabilities of PG and PrG are promising and may result in additional treatment effects by interacting and improving the functions that are a prerequisite for CBT success. Recently, neuromodulation interventions have gained interest in addiction research. Specifically, neurostimulation methods such as repeated Transcranial Magnetic Stimulation (rTMS) and transcranial Direct Current Stimulation (tDCS) were evaluated in a meta-analysis (Jansen et al., 2013). From this meta-analysis, a medium-effect size was found for neurostimulation with either rTMS or tDCS to reduce craving for substances or high-palatable food. In a study with multiple sessions of rTMS in 48 heavy smokers, 10 daily sessions of active rTMS over the DLPFC resulted in diminished cigarette consumption and nicotine dependence, compared to a control condition of sham rTMS (Amiaz et al., 2009). Related to neurostimulation, EEG neurofeedback in SUDs has recently gained renewed interest, with some pilot studies showing positive results of EEG neurofeedback training in cocaine dependence (Horrell et al., 2010) and opiate dependence (Dehghani-Arani et al., 2013). Thus, interventions with neurostimulation or neurofeedback in PG and PrG are warranted as well, to investigate whether neurostimulation interventions also hold promise in this behavioral addiction.

As a potential non-pharmacological intervention, changes in the motivational system in PG could be targeted by "attentional retraining" (MacLeod et al., 2002; Wiers et al., 2006). During attentional retraining patients are trained to reverse their attentional bias by performing computer tasks, thus aiming to reduce cue reactivity and to change habitual behaviors. A related intervention is retraining of automatic action tendencies, in which approach behavior towards addiction related stimuli is retrained to avoidance behavior (Wiers et al., 2006, 2010; Schoenmakers et al., 2007). In alcohol use disorders, results from the suggested interventions are promising (Wiers et al., 2006, 2010). However, these interventions have not yet been tested in PG and long-term effects of attentional and action tendency retraining are not yet available and need to be assessed in future research.

#### **PHARMACOLOGICAL INTERVENTIONS**

In addition to the potential of neurostimulation, neurofeedback and attentional retraining interventions, a number of promising pharmacological interventions for the treatment of PG have been reported (for a review see van den Brink, 2012). Neurobiological findings indicate a pivotal role of the mesolimbic pathway, comprising the ventral striatum, and ventromedial prefrontal cortex (VMPFC) in PG. Because the VMPFC is a structure that mainly depends on DA projections that communicate with limbic structures to integrate information, dysfunctional DA transmission could be the underlying deficit causing the VMPFC dysfunctions in PG. However, numerous other neurotransmitter systems are probably also engaged and may interact during the processing of positive and negative feedback. For example, opiates are known to increase DA release in the reward pathway, and the opiate antagonists naltrexone and nalmefene, which are known to decrease DA release, have been found to reduce reward sensitivity and probably increase punishment sensitivity as well (Petrovic et al., 2008). Moreover, treatment with opiate antagonists has been shown to be effective in PG and to diminish gambling urges (Kim and Grant, 2001; Kim et al., 2001; Modesto-Lowe and Van Kirk, 2002; Grant et al., 2008a,b, 2010b).

Whereas in substance addictions, drugs and drug-associated stimuli may elicit DA release in the ventral striatum and reinforce drug intake during the acquisition of a substance use disorder, *chronic* drug intake is associated with neuroadaptation of glutamatergic neurotransmission in the ventral and dorsal striatum and limbic cortex (McFarland et al., 2003). In addition, cue exposure has been found to depend on projections of glutamatergic neurons from the prefrontal cortex to the nucleus accumbens (LaLumiere and Kalivas, 2008). Blocking the release of glutamate has prevented drug seeking behavior in animals as well as in human substance dependent persons (Krupitsky et al., 2007; Mann et al., 2008; Rösner et al., 2008). Therefore, the first promising results from pilot studies with N-acetyl cysteine (Grant et al., 2007) and memantine (Grant et al., 2010a), which modulate the glutamate system, warrant larger studies that investigate the effects of these glutamate regulating compounds in the treatment of PG.

Besides the focus on improving cognitive functions and diminishing craving by neuromodulation or pharmacological techniques, recently, interest in the influence of protective factors has grown. For instance, low impulsivity and active coping skills have been linked to a more positive outcome for SUDs. Thus, not only a focus on risk factors, but also on the role of protective factors and environmental variables that promote them may foster our understanding of the brain-behavior relationships and the pathways in developing and recovering from PG and PrG. A potential application of a focus on both risk and protective factors may be to monitor cognitive-motivational and brain functions during treatment, investigate which functions spontaneously normalize, and which functions need additions from novel interventions such as cognitive training, neuromodulation, or pharmacological interventions.

#### **CONCLUSIONS**

PG and PrG are clearly associated with cognitive and motivational differences in neuropsychological and brain functioning. Specifically, higher impulsivity and impaired executive functioning is present, which is associated with diminished functioning of the cognitive control circuitry in the brain, such as the ACC and dorsolateral prefrontal cortex. In addition, motivational functions are affected, which are associated with differential functioning in medial frontal areas and in the thalamo-striatal circuitry, linking to the frontal cortex. More research is needed to investigate the interaction between cognitive and motivational functions, as the combination of gambling cues in cognitive tasks sometimes also improves cognitive functions. Investigating the efficacy of novel interventions that target these neurobiological mechanisms, such as neuromodulation, cognitive training, and pharmacological interventions, is needed in order to investigate its potential to improve treatment outcome. In addition, research focusing on protective factors and the spontaneous recovery of risk factors could indicate which mechanisms to target in order to improve the course of PG.

#### **AUTHOR CONTRIBUTIONS**

Anna E. Goudriaan, Murat Yücel, and Ruth J. van Holst contributed to the design of the review, Anna E. Goudriaan and Ruth J. van Holst drafted parts of the manuscript, Anna E. Goudriaan, Ruth J. van Holst, and Murat Yücel revised this work critically for important intellectual content. Final approval of the version to be published was given by all authors and all authors agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

#### **REFERENCES**

Langer, E. J. (1975). The illusion of control. *J. Pers. Soc. Psychol.* 32, 311–328.

Ledgerwood, D. M., Orr, E. S., Kaploun, K. A., Milosevic, A., Frisch, G. R., Rupcich, N., et al. (2012). Executive function in pathological gamblers and healthy controls. *J. Gambl. Stud.* 28, 89–103. doi: 10.1007/s10899-010-9237-6

Pessoa, L. (2008). On the relationship between emotion and cognition. *Nat. Rev. Neurosci.* 9, 148–158. doi: 10.1038/nrn2317

longitudinal study of a complete birth cohort. *Psychol. Sci.* 23, 510–516. doi: 10. 1177/0956797611429708

to young adulthood: additive and moderating effects of common risk factors. *Psychol. Addict. Behav.* 23, 91–104. doi: 10.1037/a0013182

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 30 November 2013; accepted: 08 April 2014; published online: 20 May 2014*.

*Citation: Goudriaan AE, Yücel M and van Holst RJ (2014) Getting a grip on problem gambling: what can neuroscience tell us? Front. Behav. Neurosci. 8:141. doi: 10.3389/ fnbeh.2014.00141*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience*.

*Copyright © 2014 Goudriaan, Yücel and van Holst. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms*.

**OPINION ARTICLE** published: 02 December 2013 doi: 10.3389/fnbeh.2013.00182

### What motivates gambling behavior? Insight into dopamine's role

#### *Patrick Anselme1 \* and Mike J. F. Robinson2,3*

*1 Département de Psychologie, Université de Liège, Liège, Belgium*

*2 Department of Psychology, University of Michigan, Michigan, MI, USA*

*3 Department of Psychology, Wesleyan University, Connecticut, CT, USA*

*\*Correspondence<EMAIL_ADDRESS>
#### *Edited by:*

*Bryan F. Singer, University of Michigan, USA*

#### *Reviewed by:*

*Nichole Neugebauer, University of Chicago, USA*

**Keywords: dopamine, motivation, gambling, loss, reward uncertainty**

It is commonly believed that monetary gain is the cause of gambling behavior in humans. Mesolimbic dopamine (DA), the chief neuromediator of incentive motivation, is indeed released to a larger extent in pathological gamblers (PG) than in healthy controls (HC) during gambling episodes (Linnet et al., 2011; Joutsa et al., 2012), as in other forms of compulsive and addictive behavior. However, recent findings indicate that the interaction between DA and reward is not so straightforward (Blum et al., 2012; Linnet et al., 2012). In PG and HC, DA release seems to reflect the unpredictability of reward delivery rather than reward *per se*. This suggests that the motivation to gamble is strongly (though not entirely) determined by the inability to predict reward occurrence. Here we discuss several views of the role of DA in gambling, and attempt to provide an evolutionary framework to explain its role in uncertainty.

#### **TRADITIONAL VIEW: MONEY DRIVES GAMBLING**

Common sense suggests that if gambling at casinos is attractive for many people, it is because it offers an opportunity to win money (Dow Schüll, 2012). Of course, a "big win" is rare, but the random component behind most games and the publicizing of big winners lets people believe that the chance of winning a lot is not so unlikely. In this traditional view, money is a gambler's primary motivation, and randomness in games allows the gambler to hope that the gains will overcome the losses.

This view is compatible with the evidence that DA released in the nucleus accumbens, a mesolimbic region in the brain, magnifies the attractiveness of rewards and conditioned cues (Berridge, 2007). Mesolimbic DA transforms neutral cues into conditioned cues when they come to reliably predict reward delivery (Melis and Argiolas, 1995; Peciña et al., 2003; Flagel et al., 2011). Money is certainly a strong conditioned cue, which has been associated with abundance and power in all human civilizations. As with other reward sources, money is known to enhance mesolimbic DA levels in the human striatum during gambling episodes, suggesting that money is what motivates gamblers (Koepp et al., 1998; Zald et al., 2004; Zink et al., 2004; Pessiglione et al., 2007). For example, Joutsa et al. (2012) showed that DA is released in the ventral striatum during instances of high- but not low-reward, in both PG and HC, and that the severity of symptoms in PG is associated with larger DA responses.

#### **THE ATTRACTIVENESS OF LOSSES**

Although the traditional view is in agreement with neuroscientific data, it fails to explain why people often describe gambling as a pleasant activity rather than as an opportunity to gain money. During gambling episodes, PG report euphoric feelings comparable to those experienced by drug users (van Holst et al., 2010), and the more PG lose money, the more they tend to persevere in this activity a phenomenon referred to as loss-chasing (Campbell-Meiklejohn et al., 2008). Such results are hardly compatible with the traditional view. Animal and human studies indicate that the role of DA in reward is, at least in gambling, more complex than initially believed (Linnet, 2013).

Determining the exact timing of subjective feelings or how losses spur on a gambler's desire to play during gambling episodes is difficult because different emotions and cognitions constantly overlap. Nevertheless, Linnet et al. (2010) were able to measure mesolimbic DA release in PG and HC winning or losing money. Unexpectedly, they found no difference in dopaminergic responses between PG and HC who won money. Dopamine release in the ventral striatum, however, was more pronounced for the losses in PG relative to HC. Given the motivational impact of mesolimbic DA, Linnet and colleagues argue that this effect could explain losschasing in PG. In addition, they point out that "PG are not hyperdopaminergic *per se*, but have increased DA susceptibility toward certain types of decisions and behavior" (p. 331). This finding that DA release is higher in PG losing money than in PG winning money is consistent with the evidence that "near misses" enhance the motivation to gamble and recruit the brain reward circuit more than "big wins" (Kassinove and Schare, 2001; Clark et al., 2009; Chase and Clark, 2010). Possibly related to this phenomenon is the evidence that, compared with gains, the amount of monetary losses has limited effect on the extent to which probabilistic (and delayed) losses are discounted in humans (Estle et al., 2006). This suggests that a lower probability (and a longer delay) reduces a gambler's motivation less when losses rather than gains are involved. In contrast, the big win hypothesis suggests that pathological gambling develops in individuals that initially experienced large monetary gains, but the attempts to demonstrate this effect on persistence of gambling have failed (Kassinove and Schare, 2001; Weatherly et al., 2004). Current evidence therefore suggests that losses contribute to motivate gambling more than gains.

#### **THE ATTRACTIVENESS OF REWARD UNCERTAINTY**

One of the main underlying factors to the phenomenon of loss-chasing may relate to the importance of reward uncertainty. Studies have shown that reward uncertainty rather than reward *per se*, will magnify mesolimbic DA, both in monkeys (Fiorillo et al., 2003; de Lafuente and Romo, 2011) and healthy human participants (Preuschoff et al., 2006). In PG, accumbens DA is maximal during a gambling task when the probability of winning and losing money is identical a 50% chance for a two-outcome event representing maximal uncertainty (Linnet et al., 2012). Although non-dopaminergic neurons might also be involved in the coding of reward uncertainty (Monosov and Hikosaka, 2013), these results based on electrophysiological and neuroimaging techniques indicate that DA is crucial for the coding of reward uncertainty. This suggestion is corroborated by a large number of behavioral studies, showing that mammals and birds respond more vigorously to conditioned cues predicting uncertain rewards (Collins et al., 1983; Anselme et al., 2013; Robinson et al., under review) and tend to prefer an uncertain food option over a certain food option in dualchoice tasks (Kacelnik and Bateson, 1996; Adriani and Laviola, 2006), sometimes despite a lower reward rate (Forkman, 1991; Gipson et al., 2009). According to Greg Costikyan, an award-winning game designer, games cannot hold our interest in the absence of uncertainty—which can take many forms, occurring in the outcome, the game's path, analytical complexity, perception, and so on (Costikyan, 2013). Discussing the game of *Tic-Tac-Toe*, Costikyan (p. 10) notes that this game is dull for anyone beyond a certain age because its solution is trivial. The reason why children play this game with enjoyment is that they do not understand that the game has an optimal strategy; for children, the game of *Tic-Tac-Toe* produces an uncertain outcome. A predictable game is dull, just like a detective novel for which the identity of the murderer is known in advance. Based on this assumption, Zack and Poulos (2009) note that several payoff schedules (slot machines, roulette, and dice game of craps) have a probability of winning close to 50%, so that they are expected to elicit maximal DA release and, therefore, reinforce the act of gambling.

The evidence that uncertainty itself appears to be a source of motivation is visible in the growing trend of pathological gambling that involves extended play at video poker or slot machines (Dow Schüll, 2012). Individuals are playing to play rather than to win, and monetary wins are conceived as the opportunity to extend the duration of play, rather than the game's main objective. In addition, game programmers have uncovered a profitable trend toward larger and larger number of bets per round of a given game (in Australia, *>*100 bets on a given roll), with smaller and smaller amounts (going as low as one cent), giving rise to a "losses disguised as wins" effect, where players win less than they wagered (Dixon et al., 2010). It is almost as if players were drawn to placing bets or trying to uncover the algorithm that determines the wins and losses (this is often reported in players, see Dow Schüll, 2012). Recently, we have shown in adult rats that an initial exposure (8 days) to conditioned cues predicting highly uncertain rewards sensitizes responding to those cues in the long term (for at least 20 days) despite a gradual reduction in the level of uncertainty (Robinson et al., under review). No behavioral sensitization was apparent following a later exposure to high uncertainty (rewards were provided with certainty during the first 8 days). This result is compatible with other findings showing that persistent gambling behavior is more likely to occur in individuals that experience unpredictable environments and gambling situations early in life (Scherrer et al., 2007; Braverman and Shaffer, 2012).

#### **A POSSIBLE EVOLUTIONARY ORIGIN OF GAMBLING BEHAVIOR**

Since wins are rare and often small during gambling episodes, it is unlikely that they are sufficient to motivate people to persevere in the task. The fact that losses can motivate gambling more than gains is also difficult to understand. So, why do people gamble? Pathological gambling is certainly maladaptive behavior, but the attractiveness of uncertain rewards is so widespread in the animal kingdom that this tendency should have an adaptive origin. Here we suggest a hypothesis—referred to as the compensatory hypothesis—developed by one of the authors, that describes gambling-like behavior in an evolutionary framework (Anselme, 2013).

In nature, animals are subject to a lack of cognitive control in many circumstances; they are often unable to predict what is going to happen. This essentially occurs for two reasons. First, the distribution of natural resources is random, so that a large number of responses must be produced before finding vital resources. Second, the reliability of conditioned cues is often imperfect—e.g., for some species, fruit-trees may act as conditioned cues because of their association with reward (the presence of fruits), but this association is unreliable since fruit-trees have no fruits for most of the year. Given this lack of cognitive control about objects and events, it can be argued that if reward uncertainty were not a source of motivation, most behaviors would extinguish because of the high failure rate (and energy loss) experienced by animals. The compensatory hypothesis suggests that, when a significant object or event's predictability is low, motivational processes are recruited to compensate for the inability to make correct predictions; motivation would act as a mechanism to delay extinction (Anselme, 2013). In other words, allowing an animal to persevere in a task is only possible if its behavior is motivated by the lack of predictability (i.e., uncertainty) rather than by reward itself. The compensatory hypothesis could explain why losses are so important in motivating human gamblers: without the opportunity of receiving no reward, gains become predictable and hence most games become dull (Costikyan, 2013). In addition, this hypothesis provides an interpretation to the evidence that, like physiological deprivations (Nader et al., 1997), psychosocial deprivations such as a lack of maternal care enhance mesolimbic DA release and, correlatively, incentive motivation to seek food (Lomanowska et al., 2011). Psychosocial deprivations also seem to be a cause of gambling-like behavior in both pigeons and humans (van Holst et al., 2010; Pattison et al., 2013). In fact, all forms of deprivation result from the inability to predict how to find/obtain appropriate stimuli whether food, social relationships, opportunities to work and play, etc. In most cases, this inability is a consequence of environmental poverty. On account of this, poor environments resemble unpredictable environments and the compensatory hypothesis suggests that, in both cases, a higher motivation is recruited to persevere in the laborious task of finding resources.

Assuming that this interpretation is correct, gambling behavior in humans could be phylogenetically inherited from older mammalian species whose members motivated by reward uncertainty had a better chance of survival in complex, dynamic environments. Pathological gambling might be the exaggeration of a natural tendency exploited by casinos and games of chance. Of course, uncertaintydriven motivation is no longer required to survive within most western cultures. However, gambling might be hijacking an evolutionary system designed to resolve uncertainty by spurring pulses of motivation, despite or because of repeated losses. How could pathological gambling be addressed? We think that this psychopathology should certainly be treated on a case-by-case basis, depending on the vulnerability of each PG. For example, favoring enrichment of a PG's daily environment by varying leisure activities and social relations may reduce his desire to seek a surplus of stimulation. At a societal level, one approach allowing to address pathological gambling might be that gamblers at casinos can win more often than they lose but only very small gains (similar to the wagered amounts) in order to render gambling persistence less attractive. More thorough investigations are needed to identify the parameters underpinning the addictive power of games and to promote the development of games which do not exploit our phylogenetic vulnerability.

#### **REFERENCES**

Adriani, W., and Laviola, G. (2006). Delay aversion but preference for large and rare rewards in two choice tasks: implications for the measurement of self-control parameters. *BMC Neurosci.* 7:52. doi: 10.1186/1471-2202-7-52

a twin cohort. *J. Nerv. Ment. Dis.* 195, 72–78. doi: 10.1097/01.nmd.0000252384.20382.e9

striatal responses to monetary reward depend on saliency. *Neuron* 42, 509–517. doi: 10.1016/S0896- 6273(04)00183-7

*Received: 20 October 2013; accepted: 12 November 2013; published online: 02 December 2013.*

*Citation: Anselme P and Robinson MJF (2013) What motivates gambling behavior? Insight into dopamine's role. Front. Behav. Neurosci. 7:182. doi: 10.3389/fnbeh. 2013.00182*

*This article was submitted to the journal Frontiers in Behavioral Neuroscience.*

*Copyright © 2013 Anselme and Robinson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Neurobiological underpinnings of reward anticipation and outcome evaluation in gambling disorder

#### **Jakob Linnet 1,2,3,4\***

1 Research Clinic on Gambling Disorders, Aarhus University Hospital, Aarhus, Denmark

2 Center of Functionally Integrative Neuroscience, Aarhus University, Aarhus, Denmark

3 Division on Addiction, Cambridge Health Alliance, Cambridge, MA, USA

4 Department of Psychiatry, Harvard Medical School, Harvard University, Cambridge, MA, USA

#### **Edited by:**

Bryan F. Singer, University of Michigan, USA

#### **Reviewed by:**

Alexis Faure, Centre Neurosciences Paris Sud (CNPS), CNRS, France Jonathan David Morrow, University of Michigan, USA

#### **\*Correspondence:**

Jakob Linnet, Research Clinic on Gambling Disorders, Aarhus University Hospital, Nørrebrogade 44, Building 30, DK-8000 Aarhus C, Denmark e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Gambling disorder is characterized by persistent and recurrent maladaptive gambling behavior, which leads to clinically significant impairment or distress. The disorder is associated with dysfunctions in the dopamine system. The dopamine system codes reward anticipation and outcome evaluation. Reward anticipation refers to dopaminergic activation prior to reward, while outcome evaluation refers to dopaminergic activation after reward. This article reviews evidence of dopaminergic dysfunctions in reward anticipation and outcome evaluation in gambling disorder from two vantage points: a model of reward prediction and reward prediction error by Wolfram Schultz et al. and a model of "wanting" and "liking" by Terry E. Robinson and Kent C. Berridge. Both models offer important insights on the study of dopaminergic dysfunctions in addiction, and implications for the study of dopaminergic dysfunctions in gambling disorder are suggested.

**Keywords: anticipation, reward prediction error, reward prediction, incentive salience, dopamine, gambling disorder, pathological gambling**

#### **NEUROBIOLOGICAL UNDERPINNINGS OF REWARD ANTICIPATION AND OUTCOME EVALUATION IN GAMBLING DISORDER**

Gambling disorder is characterized by persistent and recurrent maladaptive gambling behavior, which leads to clinically significant impairment or distress (American Psychiatric Association [DSM 5], 2013). Gambling disorder was recently reclassified from "pathological gambling" (an impulse control disorder) to a "behavioral addiction" under the substance use classification, which emphasizes the association between gambling disorder and other types of addiction.

Gambling disorder is associated with dysfunctions in the dopamine system. The dopamine system is sensitive to behavioral stimulation related to monetary reward, particularly in the ventral striatum (Koepp et al., 1998; Delgado et al., 2000; Breiter et al., 2001; de la Fuente-Fernández et al., 2002; Zald et al., 2004). Dopaminergic dysfunctions in the ventral striatum are linked to gambling disorder (Reuter et al., 2005; Abler et al., 2006; Linnet et al., 2010, 2011a,b, 2012; van Holst et al., 2012; Linnet, 2013).

The dopamine system codes *reward anticipation* and *outcome evaluation*. Reward anticipation refers to dopaminergic activation prior to reward, while outcome evaluation refers to dopaminergic activation after the reward. This article reviews evidence on dopaminergic dysfunctions in reward anticipation and outcome evaluation in gambling disorder from two vantage points: a model of reward prediction and reward prediction error by Schultz et al. (Fiorillo et al., 2003; Schultz, 2006; Tobler et al., 2007; Schultz et al., 2008), and a model of "wanting" and "linking" by Robinson and Berridge (Robinson and Berridge, 1993, 2000, 2003, 2008; Berridge and Aldridge, 2008; Berridge et al., 2009). It is suggested that gambling disorder may provide a "model disorder" of addiction for the two approaches, which is not confounded by ingestion of exogenous substances.

The ventral striatum and the nucleus accumbens (NAcc) play a central role in both models, which is consistent with findings of dopamine dysfunctions in the ventral striatum in gambling disorder. Therefore, this review focuses on the ventral striatum in relation to gambling disorder. Other relevant areas include the prefrontal cortex (e.g., orbitofrontal cortex) and other areas of the basal ganglia (e.g., the putamen, nucleus or caudate).

#### **REWARD PREDICTION AND REWARD PREDICTION ERROR**

Reward prediction refers to the anticipation of reward, while reward prediction error refers to the outcome evaluation. Reward prediction and reward prediction error are associated with the learning of reward properties of stimuli. According to Wolfram Schultz (2006), reward prediction and reward prediction error derive from Kamin's *blocking rule* (Kamin, 1969), which suggests that a reward that is fully predicted does not contribute to learning. A stimulus that can be entirely predicted contains no new information, and the reward prediction error rate is therefore zero. Rescola and Wagner described the so-called *Rescola-Wagner learning rule* (Rescola and Wagner, 1972), which states that learning slows progressively as the reinforcer becomes more predicted.

In random binary outcome conditions, e.g., reward vs. noreward, the *expected value* (EV) is the average value that can be expected from a given stimulus, which is a linear function of reward probability. In contrast, *uncertainty*, which can be defined as the variance (σ 2 ) of a probability distribution (Schultz et al., 2008), is the mean squared deviation from the EV, which is an inverse U-shaped function. Midbrain and striatal dopamine coding of EV and uncertainty follow linear and quadratic functions of reward prediction similar to their mathematical expressions (Fiorillo et al., 2003; Preuschoff et al., 2006; Schultz, 2006). The dopamine system also codes deviations in outcome from the reward prediction, i.e., reward prediction error: ". . .dopamine neurons emit a positive signal (activation) when an appetitive event is better than predicted, no signal (no change in activity) when an appetitive event occurs as predicted, and a negative signal (decreased activity) when an appetitive event is worse than predicted. . .[and] dopamine neurons show bidirectional coding of reward prediction errors, following the equation Dopamine response = Reward occurred−Reward predicted" (Schultz, 2006, pp. 99–100).