882 vIII. EMERGING  ISSuES AND  NEW  DIRECTIONS
to be a more objective measure of student quality 
than high school grades (Lemann, 1999).
18. See Almlund et al. (2011) and Borghans et al. 
(2008) for comparisons of some of these different 
taxonomies.
19. See Costa and McCrae (1992).
20. See Borghans et al. (2008) and Almlund et 
al. (2011) for comparisons of these taxonomies.
21. See, for example, Cloninger, Dragan, Svra -
kic, Bayon, and Przybeck (1999).
22. A deeper question, as yet not systematically 
investigated in the literature in economics or psy -
chology, is whether the “traits” captured by the al -
ternative measurement systems are the expression 
of a more basic set of preferences or goals. McAd -
ams (2006) adds goals to the list of possible skills. 
Almlund et al. (2011) and Heckman et al. (2012) 
develop a model in which preferences and endow -
ments of skills determine the effort applied to tasks. 
As shown in the next section, performance on tasks 
is the source of any measurement of a skill. Hence, 
in their framework, measures of skills are deter -
mined, in part, by preferences
23. A growing body of research shows that 
performance on tasks is moderated by a variety of 
behavioral biases (for recent summaries, see Chap -
man, Dean, Ortoleva, Snowberg, & Camerer, 2018; 
Stango & Zinman, 2019).
24. See Almlund et al. (2011).
25. The idea of using behaviors to measure non- 
cognitive skills is old. Ralph Tyler (1973; also see 
Rothstein, Jacobsen, & Wilder, 2008) suggested 
using measures of behavior to capture noncognitive 
skills in his first proposal for the National Assess -
ment of Educational Progress tests. This idea is pur -
sued in the recent literature (Heckman, Humphries, 
& Veramendi, 2018; Jackson, 2018). See Kautz and 
Zanoni (2019) for a recent application of this idea. 
We discuss this approach more extensively in a later 
section.
26. See Borghans et al. (2011a), Almlund et al., 
2011, and Heckman and Kautz (2012) for discus -
sions of this problem.
27. See Borghans et al. (2008), Clingman and 
Fowler (1976), Edlund (1972), Ayllon and Kelly 
(1972), Breuning and Zella (1978), Holt and Hobbs 
(1979), Larson, Saccuzzo, and Brown (1994), 
and Segal (2012). This evidence is summarized in 
Borghans et al. (2008) and Almlund et al. (2011).
28. See, for example, Nisbett (2009).
29. The correlation between DAT and the wide -
ly used Armed Forces Qualification Test (AFQT) 
scores in the National Longitudinal Study of Youth 
1979 (NLSY79) is .75 (Borghans, Golsteyn, Heck -
man, & Humphries, 2016). Friedman and Streicher 
(1985) estimate correlations between .65 and .82 in 
a sample of high school sophomores and juniors. 
Kettner (1976) estimates correlations between DAT 
and the AFQT subtests of .76–.89 in a sample of 
juniors and seniors.30. In the Stella Maris data, openness to experi -
ence is strongly correlated with IQ. See Borghans et 
al. (2011b).
31. These are called Likert (1932) scales.
32. Reference bias is also problematic in health 
surveys that use self-  reported, subjective health as -
sessments (see Groot, 2000).
33. r = –0.07 ( p = 0.73).
34. See the studies summarized in Almlund et al. 
(2011).
35. For further discussion of reference bias, see 
Duckworth (2012), Goldammer (2010), Peng, Nis -
bett, and Wong (1997), Heine et al. (2001), Heine, 
Lehman, Peng, and Greenholtz (2002), Heine, Bu -
chtel, and Norenzayan (2008), Naumann and John 
(2011), and Schmitt et al. (2007).
36. In an attempt to address reference bias, some 
psychologists measure skills using behaviors. Heine 
et al. (2008) examine cross-  country differences in 
conscientiousness using objective measures, includ -
ing walking speed, postal workers’ speed, and the ac -
curacy of clocks in public banks. To measure walk -
ing speed, researchers timed how long it took for a 
random sample of people to walk 60 feet in public 
areas. Postal workers’ speed was assessed by measur -
ing how long it took for postal workers to sell stamps.
37. See Heckman, Humphries, and Kautz (2014).
38. The measure of risky and reckless behavior 
is based on whether adolescents engaged in any of 
the following behaviors: stealing from a store, pur -
posefully damaging property, taking something 
worth less than $50, or conning someone.
39. See the discussion in Pratt and Cullen (2000) 
and Benda (2005).
40. See Golsteyn and Schildberg-  Hörisch (2017) 
for a review.
41. A mechanism is said to be incentive compat -
ible if it provides individuals with an incentive to 
truthfully and fully reveal their preferences. As of 
now, there is no existing theory regarding incen -
tive compatibility in making hypothetical choices. 
However, there is some evidence that people have 
a preference for being honest and for being seen as 
honest (Abeler, Nosenzo, & Raymond, 2019).
42. There is evidence that the level of incentiv -
ization (stakes) also matters (see Holt & Hobbs, 
1979).
43. Harrison and Swarthout (2016) showed how 
appropriately designed experimental elicitation can 
be used to empirically test competing theories of 
decision making under risk that allow for various 
biases.
44. Hypothetical bias  refers to differences in re -
sponse between settings in which consequences are 
hypothetical and real.
45. The responses from a hypothetical setting 
are corrected for “hypothetical bias” so as to match 
those which would have been obtained in an incen -
tivized setting (where choices would have had real 
consequences).