Inverse decision-making using neural amortized Bayesian actors
Dominik Straub Tobias F. Niehues Jan Peters Constantin A. Rothkopf

GoT-CQA: Graph-of-Thought Guided Compositional Reasoning for 
 Chart Question Answering
    Lingling Zhang1# Muye Huang1# Qianying Wang2Corresponding author. # These authors contributed to the work equally. Yaxian Wang1 Wenjun Wu1 Jun Liu1

Xi’an Jiaotong University1   Lenovo Research2

{huangmuye, wyx1566, nickjun98}@stu.xjtu.edu.cn   wangqya@Lenovo.com


{zhanglling, liukeen}@xjtu.edu.cn

    September 9, 2024
=================================================================================================================================================================================================================================================================================================================

§ ABSTRACT 
Bayesian observer and actor models have provided normative explanations for many behavioral phenomena in perception, sensorimotor control, and other areas of cognitive science and neuroscience. They attribute behavioral variability and biases to different interpretable entities such as perceptual and motor uncertainty, prior beliefs, and behavioral costs. However, when extending these models to more complex tasks with continuous actions, solving the Bayesian decision-making problem is often analytically intractable. Moreover, inverting such models to perform inference over their parameters given behavioral data is computationally even more difficult. Therefore, researchers typically constrain their models to easily tractable components, such as Gaussian distributions or quadratic cost functions, or resort to numerical methods. To overcome these limitations, we amortize the Bayesian actor using a neural network trained on a wide range of different parameter settings in an unsupervised fashion. Using the pre-trained neural network enables performing gradient-based Bayesian inference of the Bayesian actor model's parameters. We show on synthetic data that the inferred posterior distributions are in close alignment with those obtained using analytical solutions where they exist. Where no analytical solution is available, we recover posterior distributions close to the ground truth. We then show that identifiability problems between priors and costs can arise in more complex cost functions. Finally, we apply our method to empirical data and show that it explains systematic individual differences of behavioral patterns.




§ INTRODUCTION

Explanations of human behavior based on Bayesian observer and actor models

have been widely successful, 
because they structure the factors influencing behavior into interpretable components 
<cit.>. They have explained a wide range of phenomena in perception <cit.>, motor control <cit.>, other domains of cognitive science <cit.>, and neuroscience <cit.>. 
Bayesian actor models assume that an actor receives uncertain sensory information about the world. This sensory information is uncertain because of ambiguity of the sensory input or noise in neural responses <cit.>. To obtain a belief about the state of the world, the actor fuses this information with prior knowledge according to Bayes' rule. But humans do not only form beliefs about the world, they also act in it. In Bayesian actor models, this is expressed as the minimization of a cost function, which expresses the actor's goals and constraints. An optimal actor should minimize this cost function while taking their belief about the state of the world into account, i.e. minimize the posterior expected cost. However, there is not only uncertainty in perception, but also in action outcomes, due to the inherent variability of the motor system <cit.>. If an actor wants to perform the best action possible <cit.>, this uncertainty should also be incorporated into the decision-making process by integrating over the distribution of action outcomes.


One problem, which makes solving the Bayesian decision-making problem hard in practice, is that the expected cost is often not analytically tractable, because it involves integrals over the posterior distribution and the action distribution. Consequently, optimization of the expected cost is also often intractable.
Certain special cases, especially Gaussian distributions and quadratic cost functions, admit analytical solutions. Applications of Bayesian models have typically made use of these assumptions.
However, empirical evidence shows that the human sensorimotor system does not conform to these assumptions. 
Noise in the motor system depends on the force produced <cit.> and variability in the sensory system follows a similar signal-dependence known as Weber's law <cit.>. Cost functions other than quadratic costs have been shown to be required in sensorimotor tasks <cit.>.
If we want to include in a model what is known about the human sensorimotor system or treat more naturalistic task settings, we need to incorporate these non-Gaussian distributions and non-quadratic costs.

Another challenge is that Bayesian actor models often have free parameters. It is common practice to set these parameters by hand, e.g. by choosing parameters of the prior distribution to match the statistics of the experiment or the natural environment or by choosing the cost function to express task goals defined by the experimenter. Predictions of the Bayesian model are then compared to behavioral data to assess optimality.
This practice is problematic because priors and costs might not be known beforehand and can be idiosyncratic to individual participants.
For example, an actor's cost function might not only contain task goals, e.g. hitting a target, but also internal factors. These can include cognitive factors such as computational resources <cit.>, but also more generally cognitive and physiological factors including biomechanical effort <cit.>. An actor's prior distribution might neither match the statistics of the task at hand nor those of the natural environment perfectly <cit.>. 
In the spirit of rational analysis <cit.>, this has motivated researchers to invert Bayesian actor models, i.e. to use Bayesian actor models as statistical models of behavior and infer the free parameters from behavior.
This approach has come to be known in different application areas under different names, including doubly-Bayesian analysis <cit.>, cognitive tomography <cit.>, inverse reinforcement learning <cit.> and inverse optimal control <cit.>.

Because the forward problem of computing optimal Bayesian actions for a given perception and decision-making problem is already quite computationally expensive, the inverse decision-making problem, i.e.

performing inference about the parameters of such a Bayesian actor model given behavioral data, is an even more demanding task. For every evaluation of the probability of observed data given the actor's parameters, 

one typically needs to solve the Bayesian actor problem. If only numerical solutions are available, this can be prohibitively expensive and makes computing gradients with respect to the parameters for efficient optimization or sampling difficult.

Here, we address these issues by providing a new method for inverse decision-making, i.e. Bayesian inference about the parameters of Bayesian actor models in sensorimotor tasks with continuous actions. Such tasks are widespread in cognitive science, psychology, and neuroscience 
and include so called production, reproduction, magnitude estimation and adjustment tasks. 
First, we formalize such tasks with Bayesian networks, both from the perspective of the researcher and from the perspective of the participant.
Secondly, we approximate the solution of the Bayesian decision-making problem with a neural network, which is trained in an unsupervised fashion using the decision problem's cost function as a stochastic training objective. Third, using the pre-trained neural network as a stand-in for the Bayesian actor within a statistical model enables efficient Bayesian inference of the Bayesian actor model's parameters given a dataset of observed behavior. 
Fourth, we show on simulated datasets that the posterior distributions obtained using the neural network recover the ground truth parameters very closely to those obtained using the analytical solution for various typical response patterns like undershoots or regression to the mean behavior. Fifth, the identifiability between priors and costs of Bayesian actor models is investigated, which is now possible based on our proposed method.
Finally, we apply our method to human behavioral data from a bean-bag throwing experiment and show that the inferred cost functions explain the previously mentioned typical behavioral patterns not only in synthetically generated but also empirically observed data.



§ RELATED WORK

Inferring priors and costs from behavior has been a problem of interest in cognitive science for many decades. In psychophysics, for example, signal detection theory is an early example of an application of a Bayesian observer model used 
to estimate sensory uncertainty and a criterion, which encompasses prior beliefs and a particular cost function <cit.>. Psychologists and behavioral economists have developed methods to measure the subjective utility function from economic decisions <cit.>. More recently, Bayesian actor models have been used within statistical models
to infer parameters of the observer's likelihood function <cit.>, the prior <cit.>, and the cost function <cit.>. The inference methods are often bespoke tools for the specific model considered in a study. They are also typically limited to discrete decisions and cannot be applied to continuous actions. 

There are two notable exceptions. 
<cit.> presented an inference framework for 
Bayesian observer models using mixtures of Gaussians. While their approach only takes perceptual uncertainty into account in the decision-making process, here we assume that the agent also considers action variability.
Furthermore, instead of inverted Gaussian mixture cost functions, our method allows for arbitrary, parametric cost functions that are easily interpretable.
<cit.> introduced the idea of
approximating Bayesian decision-making with neural networks. They
trained neural networks in a supervised fashion 
using a dataset of numerically optimized actions.
We extend this approach in three ways. First, we train the neural networks directly on the cost function of the Bayesian decision-making problem without supervision, overcoming the necessity for computationally expensive numerical solutions.
Second, in addition to cost function parameters and motor variability, we also infer priors and sensory uncertainty. Finally, we leverage the differentiability of neural networks in order to apply efficient gradient-based Bayesian inference methods, allowing us to draw thousands of samples from the posterior over parameters in a few seconds.

Our method is related to amortized inference <cit.>. A conceptual difference is that we amortize the solution of a Bayesian decision-making problem faced by a subject. This allows us to solve the Bayesian inference problem from the perspective of a researcher using efficient gradient-based inference techniques, without the need to use amortized likelihood-free inference.



§ BACKGROUND: BAYESIAN DECISION-MAKING


We start with the standard formulation of Bayesian decision theory <cit.>, illustrated in <ref> A.
An actor receives a stochastic observation  generated from a latent state , according to a generative model p(). Since the actor has no direct access to the true value of the state , they need to infer it by combining a prior distribution p() with the likelihood p() using Bayes' rule

    p() ∝ p()  p().

This Bayesian inference process describes the actor's perception. Based on their perception, the actor's goal is to perform an optimal action ^*, which represents the intended motor response. This is commonly framed as a decision-making problem 
based on a cost function ℓ(, ). The optimal action ^* for the actor is the action  that minimizes the expected cost under the posterior distribution 

    ^*    = __p()[(, )] = _a∫(, ) p() d.

Often, the loss is not defined directly in terms of the performed action , but in terms of the expectation over some stochastic version of it ∼ p(), modeling variability in responses with the same intended action .
Taking the expectation over the posterior distribution and the response distribution lets us expand <ref> as

    ^*    = __p()[ _p()[ (, ) ] ] = _∫∫(, )  p()  p()  d  d,

which changes the problem conceptually from computing an estimate of a latent variable to performing an action that is subject to motor noise.



§ METHOD

The Bayesian decision-making problem in <ref> describes the situation faced by a subject performing a task. 
For example, consider a subject in an experiment throwing a ball at a target. There is uncertainty in the perception of the target, which increases with the distance to the target. The subject has a prior belief about target locations, which might bias their perception. Additionally, there is uncertainty in the motor actions, such that repeated throws aimed at the same location will be variable. The task can be described by a cost function. For example, this cost function could consist of two parts. One part could be a function of the distance between the ball and the target. The other part could relate to motor effort, with throws at a larger distance being more costly. This is only an example and our framework in general allows for arbitrary parametric families of cost functions. As a Bayesian actor, the subject then chooses the action that maximizes the expected cost, taking into account the uncertainty in perception and action.

From the researcher's perspective, want to infer the parameters of the subject's perception-action system. In the ball throwing example, we would be interested in the subject's perceptual uncertainty and their prior belief about the target location, their action variability, and the effort cost of throwing the ball. These parameters constitute a parameter vector . Formally, we want to compute the posterior p(𝒟) for a set 𝒟 of behavioral data, as shown in <ref> B.

Our proposed method consists of two parts. First, we approximate the optimal solution of the Bayesian decision-making problem with a neural network (<ref>). A forward-pass of the neural network is very fast compared to the computation of numerical solutions to the original Bayesian decision-making problem, and gradients of the optimal action w.r.t. the parameters of the model (uncertainties, priors, costs) can be efficiently computed. In the second part of our method, this allows us to utilize the neural network within a statistical model of an actor's behavior to perform inference about model parameters (<ref>). 



 §.§ Amortizing Bayesian decision-making using neural networks

Because the Bayesian decision-making problem stated in <ref> is intractable for general cost functions, we approximate it using a neural network
    ^* ≈_(, ), which
takes the parameters of the Bayesian model  and the observed variable  as input and is parameterized by learnable parameters ψ.
It can then be used as a stand-in for the computation of the optimal action ^* in down-stream applications of the Bayesian actor model, in our case to perform inference about the Bayesian actor's parameters.



  §.§.§ Unsupervised training

We train the neural network in an unsupervised fashion by using the cost function of the decision-making problem as an unsupervised stochastic training objective.
After training, the neural network implicitly solves the Bayesian decision-making problem.

More specifically, we use the expected posterior loss 

    ℒ()    = 
        _p()[ _p(_(, ))[ (, ) ] ]
 
as a training objective. 
Because the inner expectation depends on the parameters of the neural network, w.r.t. which we want to compute the gradient, we need to apply the reparameterization trick <cit.> and instead take the expectation over a distribution that does not depend on 

    ℒ() = _p()[ _p(ϵ)[ (, ) ] ],
 
where r = g(_(, ), ϵ) with some appropriate transformation g. For example, in the perceptual decision-making model used later (<ref>), p() is log-normal with scale parameter σ_, so we can sample ϵ∼𝒩(0, 1) and use the reparameterization g(ϵ, ) = exp(log() + ϵσ_).


Now, we can easily evaluate the gradient of the objective using a Monte Carlo approximation of the two expectations,

    ∇_ℒ() ≈1/K1/N∑_k=1^K∑_n=1^N∇__(_n, _k),

which makes it possible to train the network using any variant of stochastic gradient descent.
In other words, the loss function used to train the neural network that approximates the optimal Bayesian decision-maker is simply the loss function of the underlying Bayesian decision-making problem. All that we require is a model in which it is possible to draw samples from the posterior distribution _k ∼ p() and the response distribution _n ∼ p().
This allows us to train the network in an unsupervised fashion, i.e. we only need a training data set consisting of parameters  and inputs m, without the need to solve for the optimal actions beforehand. The procedure is summarized in <ref>. See <ref> for the prior distributions used to generate parameters during training.


We used the RMSProp optimizer with a learning rate of 10^-4, batch size of 256, and N = M = 128 Monte Carlo samples per evaluation of the stochastic training objective. The networks were trained for 500,000 steps, and we assessed convergence using an evaluation set of analytically or numerically solved optimal actions (see <ref>).



  §.§.§ Network architecture

We used a multi-layer perceptron with 4 hidden layers and 16, 64, 16, 8 nodes in the hidden layers, respectively. 
We used swish activation functions at the hidden layers <cit.>. As the final layer, we used a linear function with an output 𝐲∈ℝ^3, followed by a non-linearity
    ^* = softplus(y_1  ^ y_2 + y_3),
where  is the observation received by the subject. This particular non-linearity is motivated by the functional form of the analytical solution of the Bayesian decision-making problem for the quadratic cost function as a function of  and  (<ref>) and serves as an inductive bias (<ref>).



 §.§ Bayesian inference of model parameters

The graphical model in <ref> B illustrates the generative model of behavior in an experiment. Our goal is to infer p(𝒟), where 𝒟 = {_i, _i: i = 1, …, n} is a dataset of stimuli and responses.
We assume that for every stimulus  presented to the subject, they receive a stochastic measurement ∼ p(). They then solve the Bayesian decision problem given above, i.e. they decide on an action . We used the neural network ^* ≈(, ) to approximate the optimal action.
The chosen action is then corrupted by action variability to yield a response ∼ p().
To sample from the researcher's posterior distribution over the subject's model parameters p(𝒟), we use the Hamiltonian Monte Carlo algorithm NUTS <cit.>. This gradient-based inference algorithm can be used because the neural network is differentiable with respect to the parameters  and sensory input . The procedure is summarized in <ref>.



 §.§ Implementation

The method was implemented in  <cit.>, using the packages  <cit.> for neural networks and  <cit.> for probabilistic models and sampling. Our implementation is provided in the supplementary material and will be made available on GitHub upon publication. Our software package enables the user to define new, arbitrary parametric families cost functions and train neural networks to approximate the decision-making problem.

No special high-performance compute resources were needed, since all evaluations for this paper were run on standard laptop computers (e.g. with Intel Core i7-8565U CPU). Training a neural network for 500,000 steps took 10 minutes, and drawing 20,000 posterior samples for a typical dataset with 60 trials took 10 seconds.



§ RESULTS

We evaluate our method on a perceptual decision-making task with log-normal prior, likelihood and action distribution (<ref>), which we later combine with several cost functions.
For certain cost functions, this Bayesian decision-making problem is analytically solvable. We evaluate our method's posterior distributions against those obtained when using the analytical solution for the optimal action (<ref>).
Our evaluations allowed us to find possible identifiability problems between prior and cost parameters inherent to Bayesian actor models, which we analyze in more detail (<ref>).
Finally, we apply our method to real data from a sensorimotor task performed by humans and show that it explains the variability and biases in the data (<ref>).



 §.§ Perceptual decision-making model

We now make the decision-making problem more concrete by introducing a log-normal model for the perceptual and action uncertainties, which captures the ball throwing example sketched in <ref> and shown in <ref>.
From the actor's perspective, we assume that sensory measurements are generated from a log-normal distribution ∼(, σ),
or equivalently that ln∼𝒩(ln, σ). This assumption is motivated by Weber's law, i.e. that the variability scales linearly with the mean <cit.>, and by Fechner's law, i.e. that perception takes place on an internal logarithmic scale <cit.>. Thus, this model can be applied to a wide range of stimuli, such as time <cit.>, space <cit.>,
sound <cit.>, numerosity <cit.>, and others. 





  
Posterior distribution
Assuming a log-normal prior ∼(μ_0, σ_0) and a log-normal likelihood ∼(, σ), the posterior is

    p() = (μ_post, σ_post)

with σ_post^2 = (1/σ_0^2 + 1/σ^2)^-1 and μ_post = exp(σ_post^2 (lnμ_0/σ_0^2 + ln/σ^2)). This can be shown by using the equations for Gaussian conjugate priors for a Gaussian likelihood in logarithmic space and then converting back to the original space.



  
Response distribution
Motivated by the idea of signal-dependent noise in actions <cit.>, we again use a log-normal distribution ∼(, σ_r)
as a probability distribution describing the variability of responses r given an intended action . This assumption results in a linear scaling of response variability with the mean, which has been observed empirically <cit.>.



  
Analytical solution for quadratic costs
For certain cost functions, this formulation allows analytical solutions for the optimal action. Specifically, for quadratic costs (, ) = ( - )^2, the optimal action is
   ^* = μ_post exp(1/2(σ_post^2 - 3 σ_r^2)).
For a derivation, see <ref>. This allows us to validate our method against a special case in which the optimal action can be computed analytically.



 §.§ Evaluation using synthetic data


We first evaluate our method on a case for which we know the analytical solution for the optimal action: the quadratic cost function (<ref>).
We generated a dataset of 60 pairs of stimuli s_i and responses r_i using the analytical solution for the optimal action with ground truth parameters μ_0 = 1.5, σ_0=0.15, σ=0.2, σ_r=0.1. The simulated data are shown in <ref> A, with a characteristic pattern of signal-dependent increase in variability, an overshot for low stimulus values and an undershot for higher stimulus values due to the prior. 
We then computed posterior distributions for the model parameters using the analytical solution for the optimal action and using the neural network. We drew 20,000 samples from the posterior distribution in 4 chains, each warmed up with 5,000 burn-in steps. Note that, because the concrete values of σ and σ_0 are unidentifiable even when using the analytical solution for the optimal action (see <ref>), we kept σ fixed to its true value during inference. This was done to evaluate our method on a version of the model, for which the analytical solution as a gold standard produces reliable results.
 <ref> B shows that both versions recover the true parameters, and the contours of both posteriors align well. 
The posterior predictive distribution generated from the neural network posterior reproduces the pattern of variability and bias in the data (shaded region in <ref> A). 



To ensure and quantitatively assess that the method works for a wide range of parameter settings, we then simulated 100 sets of parameters sampled uniformly (see <ref> for the choice of prior distributions). For each set of parameters, we simulated a dataset consisting of 60 trials. We then computed posterior distributions for each dataset in two ways: using the analytical solution for the optimal action and using the neural network to approximate the optimal action. 
In both cases, we drew 20,000 samples (after 5,000 warm-up steps) from the posterior distribution in 4 chains and assessed convergence by checking that the R-hat statistic <cit.> was below 1.05. The mean squared errors between the posterior mean and the ground truth parameter value in <ref> C show that the inference method using the neural network recovers the ground truth parameters just as well as the analytical version. <ref> A&B additionally show the error as a function of the ground truth parameter value and <ref> shows the results from <ref> C numerically.



 §.§ Limits of identifiability of costs and priors



We now apply our method to new cost functions, for which analytical solutions are not (readily) available. For example, we consider cost functions that incorporate the cost of the effort of actions. It is more costly to throw a ball at a longer distance due to the force needed to produce the movement, or it is more effortful to press a button for a longer duration. This can be achieved using a weighted sum of the squared distance to the target stimulus and the square of the response:

    (, ) = β ( - )^2 + (1-β) ^2.

This cost function introduces another source of biases besides perceptual priors: people might undershoot stimuli at larger distances more (see <ref>).
Using our method, we can turn to investigating whether we can tease these different sources of biases apart.
<ref> A shows a pattern of behavior with an undershot with 
ground truth parameters μ_0 = 2.95, σ_0=0.19, σ=0.14, σ_r=0.23, β=0.72.
The posterior distribution <ref> B shows that the undershot can either be attributed to a subject trying to avoid the mental or physical strain of larger effort, or to biased perception due to a low prior mean. This is difficult to disentangle, and shows in a correlated posterior for the effort cost parameter β and the prior mean μ_0. Over 100 simulated datasets with a range of different ground truth parameter values, the MSE between the inferred posterior mean and the ground truth is high when both μ_0 and β are unknown. Once we fix one of the confounding parameters at their true value and exclude them from the set of inferred parameters, we observe a considerable increase in accuracy of the inferred parameters (see <ref> C). Again, <ref> C&D show the error as a function of the ground truth parameter value and <ref> shows the results from <ref> C numerically.

Nevertheless, this unidentifiability is a property of the model itself and not a shortcoming of the inference method. In fact, our method opens up the possibility to investigate these properties of Bayesian actor models in the first place.
To demonstrate this, we derived an analytical solution for the quadratic cost with quadratic effort (see <ref>) and showed that the posteriors obtained using the neural network match those obtained with the analytical solution (see <ref> B).

We performed the same analysis for an asymmetric quadratic cost function, which can penalize overshots more than undershots, or vice versa:

    (, ) = 2 |α - 1( - )| ( - )^2   with  1(x) = 
           1      if   x ≥ 0
    
           0      else
    .

<ref> D shows an example of behavior generated using this cost function with μ_0 = 4.47, σ_0=0.17, σ=0.22, σ_r=0.23 and α = 0.29, which exhibits an undershot.
There is no analytical solution for the Bayesian decision-making problem with this cost function known to us. Nevertheless, ground truth parameters are accurately inferred (see <ref> E) and the same pattern of identifiability issues as for the previous cost function is observed (see <ref> F and <ref>). 





 §.§ Inference of human costs and priors


We applied our method to data of human participants from a previously published study of a bean bag throwing task <cit.>. 20 participants were asked to throw a bean bag at five different target distances from 3 to 11 feet (0.9 to 3.4 meters) with 2 feet increments (0.6 meters). The subjects received neither visual nor verbal feedback.
<ref> A shows data from two example participants from the study. The first participant's behavior is characterized by a tendency to undershoot far targets.
The second participant hits the target accurately on average, but has a higher variability.


We fit the model using the quadratic cost function with quadratic effort to the data by drawing 20,000 samples (after 5,000 warm-up steps) from the posterior distribution for both participants. The posterior distributions (<ref> B) show that the first participant's behavior can be either explained by an effort cost or a prior belief about targets (either low β or low μ_0),
but we cannot conclusively say which of the two is the case (cf. <ref>).
The second participant's relatively unbiased behavior is attributed to a comparatively small effort cost, with β being very close to 1. The action variability σ_r is higher compared to the first participant. 

The posterior mean cost function is visualized in the two-dimensional target-response space in <ref> C, where the color indicates the cost of performing a response r when the target is s. For the first participant, the cost function exhibits an asymmetry: for targets that are further away, the minimum of the cost function is shifted towards shorter responses. The second participant's cost function, on the other hand, is symmetric around the target.



§ DISCUSSION


First, we have presented a new framework for Bayesian inference about the parameters of Bayesian actor models. Computing optimal actions in these models is intractable for general cost functions and, therefore, previous work has often focused on cost functions with analytical solutions, or derived custom tools for specific tasks. We propose an unsupervised training scheme for neural networks to approximate Bayesian actor models with general parametric cost functions.
The approach is extensible to cost functions with other functional forms suited to particular tasks. 
Second, performing inference about the parameters of Bayesian actor models given behavioral data is computationally very expensive because each evaluation of the likelihood requires solving the decision-making problem. By plugging in the neural network approximation, we can perform efficient inference.
Third, a very large number of tasks involve continuous responses, including economic decision-making (‘how much would you wager in a bet?’), psychophysical production (‘hit the target’), magnitude reproduction (‘reproduce the duration of a tone’), sensorimotor tasks (‘reproduce the force you felt’), and cross-modality matching (‘adjust a sound to appear as loud as the brightness of this light’). Therefore we see very broad applicability of our method in the behavioral sciences including cognitive science, psychology, neuroscience, sensorimotor control, and behavioral economics.
Fourth, over the last few years, a greater appreciation of the necessity for experiments with continuous responses has developed <cit.>. Such decision problems are closer to natural environments than discrete forced-choice decisions, which have historically been the dominant approach. We provide a statistical method for analyzing continuous responses.
Finally, by inferring what a subject’s decisions were optimal for instead of postulating optimality, we conceptually reconcile normative and descriptive models of decision-making. Thus, we model the researcher’s uncertainty about the subject’s decision-making parameters explicitly.


Once we move beyond quadratic cost functions, identifiability issues between prior and cost parameters can arise. As recognized in other work as well <cit.>, these identifiability issues in Bayesian models have implications for how experiments should be designed. We have shown that, when either priors or costs are known, the identifiability issue vanishes. Based on our results, we recommend experiments with multiple conditions, between which one can assume either priors or costs to stay fixed, in order to disentangle their effects.
Our methodology should prove particularly useful for investigating task configurations that lead to or avoid such unidentifiabilities.




  
Limitations The strongest limitation we see with the present approach is that we require a model of the perceptual problem that allows drawing samples from the observer's posterior distribution, which works for stimuli that can be described well by these assumptions, e.g. magnitude-like stimuli with log-normal distributions considered in our experiments. Ideally, we would like a method that can be applied to perceptual stimuli for which these assumptions do not hold (e.g. circular variables) or more complex cognitive reasoning tasks. We see a potential to extend our method by learning an approximate posterior distribution together with the action network. This idea is closely related to loss-calibrated inference <cit.>, an approach that learns variational approximations to posterior distribution adapted to loss functions. We will explore this connection more in future work.



  
Broader impacts Methods for inferring beliefs and costs from behavioral data have potential for both positive and negative societal impact. We see a great benefit for behavioral research from methods that provide estimates of people's perceptual, motor, and cognitive properties from continuous action data, particularly in clinical applications where such tasks are easier for patients than classical forced-choice tasks. On the other hand, methods to infer people's uncertainties and costs could potentially be used in harmful ways, especially if they are used without the subjects' consent. The inference methods presented here are applicable to controlled experiments, 
and are far from scenarios with harmful societal outcomes. Still, applications of these methods should of course take place with the subjects' informed consent and the oversight of ethical review boards.



We thank Nils Neupärtl for initial work on this project idea.
We acknowledge the suggestion by an anonymous NeurIPS reviewer to try unsupervised training.
We thank Zili Liu and Chéla Willey for sharing their beanbag throwing data.
This research was supported by the European Research Council (ERC; Consolidator Award “ACTOR”-project number ERC-CoG-101045783)
and
by the 'The Adaptive Mind', funded by the Excellence Program of the Hessian Ministry of Higher Education, Science, Research and Art.
We gratefully acknowledge the computing time provided to us on the high-performance computer Lichtenberg at the NHR Centers NHR4CES at TU Darmstadt. 




unsrtnat



figuresection
tablesection



§ ALGORITHM










§ DERIVATIONS OF OPTIMAL ACTIONS



 §.§ Quadratic cost


One cost function for which the Bayesian decision-making problem under a log-normal observation model and a log-normal response model (<ref>) can be solved in closed form is the quadratic function. In that case, we can write the expected loss as 

    [(, )]    = [( - )^2] 
       = [^2] - 2[][] + [^2],

assuming independence between  and .

Using the moment-generating function of the log-normal distribution, we can evaluate these expectations as

    [(, )]    = e^2 lnμ_post + 2 σ_post^2
    	- 2 e^lnμ_post + σ_post^2/2 e^ln + σ_^2/2 + e^2 ln + 2 σ_^2.

By differentiating with respect to 

    ∂/∂[(, )] = 2 e^2 σ_^2 - 2 μ_post e^1/2 (σ_^2 + σ_post^2)

and setting to zero, we obtain the optimal action

    ^* = μ_post exp(1/2(σ_post^2 - 3 σ_r^2)).

Inserting the posterior mean (<ref>), we obtain

    ^*    = exp(σ_post^2 (lnμ_0/σ_0^2 + ln/σ^2))  exp(1/2(σ_post^2 - 3 σ_r^2)) 
       = μ_0^σ_post^2/σ_0^2exp(1/2(σ_post^2 - 3 σ_r^2))   m^σ_post^2/σ^2




 §.§ Quadratic cost with quadratic action effort

We now consider a class of cost functions of the form

    (, ) = β ( - )^2 + (1 -β) ^2.

This allows us to write the expected cost as

    [(, )]    = [β( - )^2 + (1-β) ^2] 
       = β[( - )^2] + (1-β) [^2]

Using the previously obtained result of <ref>, we can write this as

    [(, )]    = β(e^2 lnμ_post + 2 σ_post^2
    	- 2 e^lnμ_post + σ_post^2/2 e^ln a + σ_r^2/2 + e^2 ln a + 2 σ_r^2) + (1-β) e^2 ln a + 2σ_r^2.

Differentiating

    ∂/∂ a[(, )] = 2 e^2σ_r^2a - 2 βμ_post e^1/2 (σ_r^2 + σ_post^2)

and setting to zero as well as inserting the posterior mean (<ref>) yields

    a    = βμ_postexp(1/2(σ_post^2 - 3 σ_r^2)) 
       = βexp(w_0 lnμ_0 + σ_post^2/2 - 3σ_r^2/2)   m^w_m

with w_0 = σ_post^2/σ_0^2 and w_m = σ_post^2/σ^2.



§ HYPERPARAMETERS AND OTHER METHODS DETAILS



 §.§ Parameter prior distributions

We use relatively wide priors to generate the training data for the neural networks to ensure that they accurately approximate the optimal action over a wide range of possible parameter values:

    
  * σ∼𝒰(0.01, 0.5)
    
  * σ_r ∼𝒰(0.01, 0.5)
    
  * σ_0 ∼𝒰(0.01, 0.5)
    
  * μ_0 ∼𝒰(0.1, 7.0)


During inference, we use narrower, but still relatively uninformed prior distributions, to avoid the regions of the parameter space on which the neural network has not been trained:

    
  * σ∼Half-Normal(.25)
    
  * σ_r ∼Half-Normal(.25)
    
  * σ_0 ∼Half-Normal(.25)
    
  * μ_0 ∼𝒰(0.1, 5.0)


To evaluate the inference procedure, we use parameters sampled from priors, which correspond to the actual parameter values that we would in expect in a behavioral experiment:

    
  * σ∼𝒰(0.1, 0,25)
    
  * σ_r ∼𝒰(0.1, 0,25)
    
  * σ_0 ∼𝒰(0.1, 0,25)
    
  * μ_0 ∼𝒰(2.0, 5.0)


In contrast to the sensorimotor parameters, we kept the same priors for cost parameters during training of the neural network, inference and evaluation:


    
  * Quadratic cost with linear effort: β∼𝒰(0.5, 1.0)
    
  * Quadratic cost with quadratic effort: β∼𝒰(0.5, 1.0)
    
  * Asymmetric quadratic cost: α∼𝒰(0.1, 0.9)




 §.§ Evaluation dataset

To assess convergence of the neural networks, we generated an evaluation dataset consisting of 100,000 parameter sets and optimal solutions of the Bayesian decision-making problem for each cost function. If an analytical solution was available (e.g. quadratic cost), we computed the optimal action analytically. If there was no analytical solution known to us, we solved the Bayesian decision-making problem numerically. Specifically, we computed Monte Carlo approximations of the posterior expected loss

    ℒ()    = 
        _p()[ _p())[ (, ) ] ] 
       ≈1/K1/N∑_k=1^K∑_n=1^N(_n, _k)

with N = K = 10,000 and used the BFGS optimizer (implemented in ) to solve for the optimal action ^*.



 §.§ Inductive bias for the neural network

We know the closed-form solution for the optimal action ^* as a function of the parameters  for the quadratic cost function (<ref>). Therefore, we assume that the parametric form of the optimal action as a function of the sensory measurement  will not be substantially different for other cost functions, although the specific dependence on the parameters will vary.
We rewrite the optimal action for the quadratic loss (<ref>) as 

    ^* = f_1() m^f_2() + f_3().

To allow for additive biases as well, we have added an additive term, which is zero for the quadratic cost function.






§ ADDITIONAL RESULTS



 §.§ Inference with both perceptual and prior uncertainty as free parameters

During evaluation of our method, we found identifiability issues inherent to Bayesian actor models between the prior uncertainty σ_0 and the sensory variability σ.
In order to produce the same behavior as the one observed, only the ratio σ_0/σ needs to be inferred correctly (see diagonal correlation in the joint posterior of the two parameters in <ref> A) – the absolute magnitude of each of the two parameters does not substantially influence the resulting behavior and thus the solution to the true values conditioned on the observed behavior is an underdetermined problem. This intuitively makes sense, since the resulting behavior is largely shaped by how much influence prior and sensory information have, which is weighted by σ_0 and σ, respectively.

We were able to considerably improve accuracy in the inference of these two confounding parameters by fixing one of them at their true value (see <ref> B). Therefore, we decided to keep the perceptual uncertainty σ fixed when probing our method since this corresponds to psychophysically measuring σ prior to applying our method and fixing it at the measure value.





 §.§ Accuracy of inference with amortized optimal actions



  §.§.§ Comparison of posterior means and standard deviations

To assess the accuracy of the inference with amortized optimal actions, we used the data sets also shown in <ref> C and <ref> C. We compared means and standard deviations of the posteriors obtained with the analytical solutions for the optimal actions ^* or with the neural network as approximation for the subject's decision-making. The neurally amortized inference accurately produces very similar posteriors to the ones obtained using the analytical solution, as shown in <ref>.





  §.§.§ Mean squared errors

We additionally summarize the mean squared errors visualized in <ref> C (<ref>), <ref> C (<ref>), and <ref> F (<ref>).













