Evidence for Composite Cost Functions in Arm Movement Planning: An Inverse Optimal Control Approach
[['Numerous experimental studies have demonstrated that biological motion exhibits invariant features , i . e . parameters that do not significantly change with movement size , speed , load and direction [1]–[4] .', 'A number of these features was described for point-to-point ( e . g . reaching , see [5] ) and continuous ( e . g . drawing and handwriting , see [6] ) movements of the upper limb .', 'Therefore , despite the infinite number of motor strategies compatible with most of these tasks , regularities characterize human voluntary movements , suggesting that the central nervous system ( CNS ) overcomes the redundancy of movement accomplishment by following some specific rules or principles .', 'Many authors investigated these principles in the framework of deterministic optimal control theory .', 'This theory assumes that biological movements are optimal in the sense that they minimize some performance criteria or cost/loss functions .', 'In this regard , a plethora of optimal control models have been proposed in the literature [7 , 8 , for reviews] and most of them were found to fit well the experimental data .', 'Therefore , the exact relationship between different mathematical cost functions and the movement variables actually represented in the brain still remains unclear and this seems due to multiple reasons .', 'The first one is methodological: in many cases , models based on divergent assumptions and minimizing different costs can yield similar arm trajectories [9] , [10] .', 'If one considers the range of human motor variability and the consequences of model approximations , several cost functions can perform well enough to be considered valid .', 'For example , the minimum hand jerk [11] , the minimum torque change [12] but also the minimum variance models [13] make fully acceptable predictions for point-to-point arm movements performed in the horizontal plane ( i . e . quasi-straight hand paths with bell-shaped time-courses ) .', 'The second reason is conceptual: seeking a single and universal cost function might be useless [10] , in particular if the CNS is capable of optimizing a weighted combination of costs depending on the features of the task [14]–[17] .', 'Thus , a part of the present collection of models may represent constituent pieces of one motor plan .', 'It is already well-known for instance that the weight given to objective ( e . g . task-related ) and subjective ( e . g . body-related ) cost functions can be modulated by the CNS .', 'Increasing the accuracy requirements of a pointing task while keeping the movement time constant leads to an increase of muscle co-contraction ( and thus of metabolic energy expenditure , see [18] ) .', 'Conversely , experimentally induced fatigue leads to a reweighting of accuracy and energy economy requirements in the sensorimotor control of fast elbow flexions [19] .', 'Hence , cost functions would result in any case from the combination of external task demands with internal constraints .', 'In contrast to this well-identified objective/subjective costs trade-off ( see also [20] ) it has not been established yet whether or not the CNS actually combines subjective costs ( e . g . neural or mechanical energy expenditures , hand/joint/torque jerk , amount of torques/forces etc . ) .', 'In order to test this cost combination hypothesis , our approach was two-fold .', 'First , we wanted to stress the differences between the predictions of different classical models already existing in literature .', 'To this aim , we designed a pointing task with target redundancy .', 'Precisely , we reduced the external constraints of the task by asking subjects to reach to a vertical bar .', 'Thus no accuracy requirement was present in the vertical axis , which had the interesting advantage of discriminating better between different cost functions than during classical point-to-point experiments ( see Figure 1 for a proof-of-concept ) .', 'Second , we developed a framework permitting us to examine simultaneously several existing models/costs , as well as any linear combination of them , by means of an automated inverse optimal control method .', 'Inverse optimal control is a mathematical approach in which inference about the cost function is made automatically from experimental data , which are assumed to be optimal [21] .', 'Using such a method , we were able to link the recorded data to an infinite number of potential ( composite ) cost functions , in contrast to the a priori choice of one single cost function characterizing most of the previous investigations .', 'In this way we could automatically uncover which single cost or mix of costs fit best with the average behavior of subjects .', 'Direct optimal control was then used to strengthen the results provided by the inverse method and to compare directly the recorded and simulated data .', 'The experimental results show that participants adopted a consistent behavior although the final point was not imposed by the experimenter .', 'Inverse optimal control reveals that their average behavior mainly relied on a composite cost function , combining the minimization of mechanical energy expenditure ( here the absolute work of torques ) with the maximization of joint smoothness ( here the integrated squared acceleration ) .', 'Further analyses demonstrate that this mix-of-cost model replicated the most important features of arm movements and performed better than any other single cost function on which our method was based .', 'Results provided therefore support the cost combination hypothesis and , in particular for this task , emphasize two complementary and subjective costs .'], ['We used quantile-quantile plots to visually check that the data were normally distributed ( qqplot Matlab function ) .', "Shapiro-Wilk's test was used to quantify these observations for some relevant parameters .", 'One-way ANOVAs were also performed to analyze the effects of the initial posture on certain parameters .', "Post-hoc tests were conducted with Scheffé's test when necessary and appropriate ( the chosen threshold was ) .", 'Previous models of optimal control for arm movements were originally designed by their respective authors on the basis of some particular assumptions and restrictions .', 'In order to compare several different costs proposed in the literature and to apply the inverse optimal control technique described thereafter , we consider a homogeneous framework , compatible with most existing models .', 'The next subsections describe the musculoskeletal model , the inverse and direct optimal control techniques that we employed .', 'Details are deferred to the supplementary documents Text S1 and Text S2 .'], ['Reaching to objects involving target redundancy is a very common task in everyday life .', 'For instance , grasping a small ball can be achieved through many task-equivalent solutions , depending on how one chooses to put his fingers on it .', 'In such a case , like for the bar , target point discriminability is greatly reduced and , therefore , decision confidence in the brain decreases [47] .', 'Decision making in such a motor planning context [48] , [49] can be essentially driven by optimal control [8] .', 'Indeed , resolving the indeterminacy of action selection through optimal control implies that a specific cost function must be selected .', 'Whereas inverse optimal control was considered as a promising tool to characterize automatically the cost function in motor control [21] , very little has been done in the context of goal-directed arm movements .', 'Successful applications of inverse methods have been reported in sensorimotor learning [50] , [51] , human prehension [52] , pointing movements [34] .', 'To test the cost combination hypothesis for arm movement planning we decided to use a more generic method [25] .', 'The extrinsic redundancy of the task reduced the risk that several classical cost functions ( and thus , several combinations of them ) might replicate well the recorded data , which may occur if divergent models could not be sufficiently disambiguated .', 'Indeed , being able to discriminate between different cost functions was precisely a pre-requisite to test whether the CNS combines several cost functions .', 'Figure 1 illustrates that the bar reaching paradigm possesses this property .', 'Inverse optimal control gave us the possibility to drastically enlarge the number of a priori functions that are hypothetically minimized by the CNS , which is usually restricted to few candidate functions in classical studies relying on direct optimal control .', 'In a direct approach , a small number of costs is generally compared and the best one is assumed to be actually optimized by the brain .', 'The weakness is the lack of evidence that another cost , with a different biological meaning , could not perform as well or even better .', 'Although our method did not consider every possible cost function , it improved direct approaches by drastically expanding the search space .', 'Certain limitations however remain such as the uniqueness of the solution and the problem of local minima , which are hardly avoidable in the context of complex non-linear optimal control .', 'Uniqueness of the solution has been addressed recently in static inverse optimization [52] , [53] , in the context of additive cost functions and linear constraints .', 'Previous theoretical work on inverse methods was developed in other contexts such as ( linear ) control theory [54] and reinforcement learning [55] , [56] .', 'Here , the present problem was so complex that we tackled it empirically by testing multiple restarts of the algorithm and check a posteriori the effectiveness of the solution compared to basis cost functions .', 'The specific set of eight candidate cost functions has been chosen among a set of costs which could be physiologically interpreted .', 'In this sense , other cost functions such as polynomials could have been included to fit the experimental data but understanding the meaning of such abstract costs would have resulted impossible .', 'Instead we exploited the fact that many costs were already proposed in the literature of arm movement planning .', 'The presence of noise and variability in the observed data is an additional source of difficulty for identifying a unique cost using inverse optimization and only “best fitting” approximations can be found in practice .', 'Here we tested two different metrics in the space of trajectories , based on the Cartesian position of markers ( a particularly reliable measure in motion capture systems ) .', 'Actually , which metric to use to compare human and simulated trajectories remains unresolved [17] .', 'Here , the two metrics we chose allowed to greatly minimize the consequence of noise measurement and inter-trial variability , in contrast to other metrics that may try to fit directly the state vector ( including more noisy derived signals , e . g . velocities , torques or accelerations ) .', 'While these quantities are of course crucial to fully specify a motor plan , attempting to replicate those features and introducing additional uncertainty in the data set may not improve the efficiency of the inverse method .', 'Finally , differences across subjects are rarely addressed in optimal control studies because a single cost , valid for all subjects is generally sought .', 'Inverse optimal control can theoretically reveal if the same costs but weighted differently are actually optimized by different subjects or if the cost ingredients are simply not the same .', 'Inverse optimal control results showed that most subjects ( 15/20 ) adopted a behavior which essentially corresponded to a strict mixture of two subjective costs ( absolute work of torques and angular acceleration energy ) .', 'More precisely , mixing these two costs was found to fit better the observed hand paths than other linear combinations of the eight candidate costs we considered .', 'Each subject could use a different weighting of those two costs but on average their contribution to the total movement cost was roughly the same ( about 40% of the total movement cost ) .', 'These findings were quite robust as confirmed by the results when using an alternative metric ( Text S2 , Section 4 ) .', 'Further evidence for mixing energy and smoothness optimality criteria was provided by the direct optimal control analysis .', 'The bar reaching experiment revealed that several previously proposed costs did not generalize well to the present task .', 'In general , it was relatively easy to discriminate between different models .', 'Clearly , the most discrepant model was the minimum torque model , which assumes that the total amount of ( squared ) torques needed to drive the movement has to reach a minimum .', 'This model was mainly influenced by the maximum exploitation of gravity to reach the bar .', 'The minimum torque change model , which maximizes smoothness in the dynamic space , also predicted non-biological paths since even the movement direction was poorly predicted in most cases .', 'Similarly , the minimum effort model , optimizing the amount of neural input to control the movement , was unable to predict some basic features of the recorded arm trajectories .', 'Other simulations showed that neither modeling agonist/antagonist muscles as low-pass filters nor separating the control of static ( gravitational ) and dynamic forces ( speed-related ) could improve drastically the model predictions for this task ( large errors on the movement directions were still clear , see Text S2 , Section 1 ) .', 'To remove the problem of gravity integration , we also considered the same task but performed in the horizontal plane ( Text S2 , Section 2 ) .', 'We tested the behavior of 2 subjects when reaching to an horizontal target bar and the results suggest that those models were still less accurate than the energy , hybrid or geodesic models .', 'Maximizing smoothness at the level of the hand was also found to be generally irrelevant with respect to the geometry of the paths .', 'The minimum hand jerk model predicted to follow the shortest Euclidean path to reach the bar .', 'It is worth mentioning that this model had been validated originally for horizontal movements performed with a robotic device [11] , which could have induced this specific motor strategy [57] .', 'We found differently that the geodesic model , which predicts the shortest paths in joint space using the kinetic energy metric , generalized quite well to the current task .', 'This model is elegant and parameter-free and , therefore , it may be considered to be simpler than the composite cost model that we have identified .', 'One can wonder whether the gain of performance using the hybrid cost is worth its complexity .', 'Whatever the answer , it seems that the cost combination hypothesis would still be supported .', 'Indeed , Biess and collaborators demonstrated recently that “geodesic paths in the Riemannian configuration manifold have been identified as least-effort paths [where effort is defined as the amount of torques that are acting on the arm] as well as the optimal solution of the one-parameter family of MSD [Minimum Squared Derivatives] costs in Riemannian space . Hence , these costs do not only maximize smoothness , but simultaneously minimize movement effort and , thus , encode two performance indices [ . . . ]” [58] .', 'It is interesting to note that the geodesic model had been initially validated for unconstrained 3D point-to-point movements [31] .', 'These movements involved redundancy but the specification of the exact target to reach in space combined with the musculoskeletal architecture limiting the joint mobility , significantly reduced the space of admissible behaviors .', 'Consequently , only small differences were observed in many cases between the geodesic model and a model simply predicting straight paths in angle space .', 'The task we presented in this study enlarged the differences between these models ( as illustrated in Figure 1 ) .', 'While the geodesic model was quite efficient in predicting path curvature , the minimum angle jerk/acceleration models ( predicting straight lines in angle space ) captured very precisely the final point on the bar .', 'When reaching to a bar , the actual final postures thus corresponded quite accurately to the final point given by the shortest path in intrinsic space equipped with the Euclidean metric .', 'Other movement features however implied that joint co-variation was not the general rule for motor planning .', 'In particular , for certain starting postures , the only means to replicate the shape of finger paths was to include the minimization of the absolute work of torques into the cost .', 'Interestingly , minimizing this mechanical energy expenditure also resulted in final hand positions that were comparable to the real ones .', 'In agreement with the inverse optimal control results , relevant features of the bar reaching task were better reproduced by a composite cost involving two complementary functions .', 'This complementarity revealed itself quite clear with respect to parameters such as hand path curvature and joint coupling .', 'The matching between the hybrid cost model and the real data was however not perfect , notably with respect to the endpoint location .', 'This discrepancy could be due to the role of vision , which may partly influence the endpoint selection process , but this remains to be investigated .', 'We nevertheless checked the predictions of the models with known endpoints in the Text S2 ( Section 3 ) and showed that the hybrid model accurately predicts the trajectories in the case of point-to-point reachings .', 'Another explanation could be related to the fact that the vector is actually not fixed across conditions but varies depending on the initial posture .', 'This possibility may be suggested by Figure 5B where different cost contributions are obtained for different experimental conditions .', 'The brain may nevertheless prefer to keep constant the respective contributions of complementary costs because of their physical meaning ( rather than preserving the way they are combined ) .', 'This would require adjusting the weighting vector during the planning process to ensure that the resulting movement equally takes into account the different performance criteria , which is a testable hypothesis .', 'It is undeniable that a theory of motor planning assuming that the CNS is able to combine different objectives depending on the task would be very powerful for explaining almost every experimental fact and could be unfalsifiable [10] .', 'Without any prior expectation on the costs that the CNS may combine , it is likely that such a theory would be inappropriate to identify the variables represented by the brain .', 'However , to reduce such a drawback , we propose a more structured view .', 'It is worth noting that the combination of energy and smoothness costs was revealed by a task with reduced external constraints on the target .', 'By extension , we suggest that these costs emerged more clearly because we focused on natural/unconstrained movements .', 'The present results , however , raise a fundamental question: why a combination of energy and smoothness ?', 'First , since every movement consumes energy , minimizing its expenditure seems to be an appropriate strategy to keep the musculoskeletal system close to its nominal state .', 'For instance , muscle fatigue alters the execution of actions which might be decisive for species survival .', 'Accordingly , such an optimal behavior may have arisen from natural selection [59] , [60] .', 'Second , self-injuring the musculoskeletal system can have dramatic consequences so that pulling a muscle or slipping a joint could have undesirable consequences .', 'Maximizing smoothness therefore contributes also to keep the system close to its operational state .', 'The functional meaning of such costs thus appears related to homeostasis , that is to the process that maintains the internal state of biological systems within bounds [61] , [62] .', 'Accordingly , the relevance of such subjective costs had been previously reported for different species and motion , but most of the times these studies focused either on energy or on smoothness .', 'Emphasizing on the mechanical energy , [63] reported evidence that energy was a primary constraint for legged insect locomotion .', 'In a previous study [34] , we showed that particular temporal and electromyographic features of vertical pointing movements reflected mechanical energy minimization ( i . e . absolute work of torques ) .', 'Focusing on joint smoothness , [27] showed that a cost function based on the angular acceleration fit well with point-to-point movements in the horizontal plane .', 'Part of the few studies considering composite costs ( but using direct optimal control ) , [64] reported strong evidence for simultaneous multiple performance objectives including the angular acceleration and the mechanical energy expenditure during human locomotion .', 'The fact that , in this study , energy and smoothness were jointly optimized in roughly similar proportions further supports the relevance of combining subjective costs: minimizing only energy may be detrimental to smoothness and vice-versa .', 'The complementarity of cost functions has been rarely discussed in the motor control literature , even though it constitutes the main motivation for mixing different goals in the same motor plan .', 'For energy and smoothness , the complementarity is evident .', 'However , other costs turn out to be more correlated , in the sense that minimizing the one can imply a decrease of the other .', 'For example , minimizing the amount of motor command ( effort cost ) may result in a “not so large” torque change cost .', 'Due to nonlinearities , it is nevertheless difficult to establish general rules .', 'In the same vein , the similarity of joint acceleration and jerk costs is the reason why , in this study , we only conclude about the optimization of a quite generic “joint smoothness” term .', 'In general , objective costs are also optimized for the task achievement per se and are thus complementary to subjective costs .', 'Consider for instance the task of drawing a straight line on a sheet on paper .', 'In this case , optimizing the jerk at hand would be the best solution to produce such a path .', 'Energy and joint smoothness costs could however be integrated in the motor plan to determine the remaining degrees of freedom ( i . e . joint angles , muscles activities . . . ) .', 'Conversely , when trying to jump at a maximal height , it is likely that the weight given to the energy cost is decreased .', 'Joint smoothness should instead remain still present to avoid injuries and fulfill goal achievement .', 'We propose therefore that planning is a dynamic process weighting flexible objective costs ( e . g . pointing accuracy , path tracking , via-point etc . ) with more deeply anchored subjective costs .', 'This combination of cost would crucially yield the necessary flexibility for the sensorimotor system to achieve a variety of tasks , which agrees with other recent results obtained in the stochastic optimal control context [20] .', 'We still remain ignorant about the detailed neural mechanisms underlying such flexible combinations of cost functions .', 'We may suggest however that subjective cost functions are encoded at a low level of the CNS , while objective cost functions are determined at a higher level .', "Autonomic motor system that control basic involuntary function through the sympathic system dealing with body's resources might regulate the selection and combination of costs .", 'In other words , we speculate that hypothalamus , reticular formation and spinal cord , which ensure the regulation of internal body states contributing to overall physiological balance , would control the optimization process , however remaining under the influence of descending pathways .', 'Such a hierarchical view of motor planning and control is reminiscent of the theory proposed in [65] where it was suggested that the role of the low-level controller is to compute energy-efficient motor commands that conform to the higher-level variables encoding the constraints of the task itself .', 'Most of the time , external constraints are task-dependent ( hand accuracy , speed , center of mass position etc . ) , while internal constraints may be embodied in the nervous system as subjective constraints resulting from evolutionary , hereditary and learning processes .', 'This proposal needs however to be investigated more deeply .', 'Testing whether the complementary costs we have found are still present when external constraints and explicit rewards strongly shape the motor output could contribute to answer this unresolved question .']]
['Introduction', 'Materials and Methods', 'Discussion']
['An important issue in motor control is understanding the basic principles underlying the accomplishment of natural movements .', 'According to optimal control theory , the problem can be stated in these terms: what cost function do we optimize to coordinate the many more degrees of freedom than necessary to fulfill a specific motor goal ?', 'This question has not received a final answer yet , since what is optimized partly depends on the requirements of the task .', 'Many cost functions were proposed in the past , and most of them were found to be in agreement with experimental data .', 'Therefore , the actual principles on which the brain relies to achieve a certain motor behavior are still unclear .', 'Existing results might suggest that movements are not the results of the minimization of single but rather of composite cost functions .', 'In order to better clarify this last point , we consider an innovative experimental paradigm characterized by arm reaching with target redundancy .', 'Within this framework , we make use of an inverse optimal control technique to automatically infer the ( combination of ) optimality criteria that best fit the experimental data .', 'Results show that the subjects exhibited a consistent behavior during each experimental condition , even though the target point was not prescribed in advance .', 'Inverse and direct optimal control together reveal that the average arm trajectories were best replicated when optimizing the combination of two cost functions , nominally a mix between the absolute work of torques and the integrated squared joint acceleration .', 'Our results thus support the cost combination hypothesis and demonstrate that the recorded movements were closely linked to the combination of two complementary functions related to mechanical energy expenditure and joint-level smoothness .']
['To reach an object , the brain has to select among a set of possible arm trajectories that displace the hand from an initial to a final desired position .', 'Because of the intrinsic redundancy characterizing the human arm , the number of admissible joint trajectories toward the goal is generally infinite .', 'However , many studies have demonstrated that the range of actual trajectories can be limited to those that result from the fulfillment of some optimal rules .', 'Various cost functions were shown to be relevant in the literature .', 'A peculiar aspect of most of these costs is that each one of them aims at optimizing one specific feature of the movement .', 'The necessary motor flexibility of everyday life , however , might rely on the combination of such cost functions rather than on a single one .', 'Testing this cost combination hypothesis has never been attempted .', 'To this aim we propose a reaching task involving target redundancy to facilitate the comparisons of different candidate costs and to identify the best-fitting one ( possibly composite ) .', 'Using a numerical inverse optimal control method , we show that most participants produced movements corresponding to a strict combination of two subjective costs linked to the mechanical energy consumption and the joint-level smoothness .']
['motor systems', 'control theory', 'mathematics', 'applied mathematics', 'biology', 'neuroscience']