 42. Some Contributions of Economics to the Study of Personality 859
working and persistent. The bars display the 
average number of hours that people work 
in the country.
At first glance, the results are surprising. 
South Korea ranks second to last in terms 
of conscientiousness but also ranks first in 
the number of hours worked. South Korea 
is not an anomaly. Country-  level reports 
of Big Five conscientiousness are unrelated 
to the number of hours worked. The rank 
correlation between hours worked and con -
scientiousness across countries is negative, 
though statistically insignificant.33 This 
finding contrasts with studies showing that 
noncognitive skills tend to be positively re -
lated with labor supply within individual 
countries.34
Anchoring vignettes improve the perfor -
mance of surveys. For example, the PISA 
2012—an instrument that assesses student 
achievement across countries, also included 
a teacher support scale based on subjective 
student ratings of their teachers and cor -
responding anchoring vignettes. Before ad -
justing responses using vignettes, the cor -
relation across countries of achievement 
and (positive) teacher support suggested a 
negative relationship between the two vari -
ables (Kyllonen & Bertling, 2013). This re -
sult is surprising, since teacher support and 
achievement are typically positively related. 
After adjusting the scale using the vignettes, 
the correlation between the two variables in -
creased and became positive—  as would be 
expected. The unadjusted negative relation -
ship was the misleading result of reference 
bias.
These findings are relevant for construct -
ing measures of noncognitive skills. Grant -
ing that achievement tests miss important 
skills, would the self-  reported Big Five be 
a useful supplement to school evaluations? 
The possibility of reference bias suggests 
that it might not. For example, self-  reports 
of conscientiousness might measure differ -
ent things for different schools.
This issue is especially relevant given that 
some school districts are using self-  reported 
measures of noncognitive skills in their ac -
countability systems (West, Buckley, Krach -
man, & Bookman, 2018). One obvious 
concern with accountability is that teachers 
or principals might coach students to give 
favorable responses on noncognitive skill surveys. The previously discussed evidence 
of Chen et al. (2020) and the potential for 
reference bias suggests that such surveys are 
susceptible to more subtle biases.
Psychologists have addressed this prob -
lem.35 Some surveys include vignette-  based 
questions that attempt to standardize for 
aspects of the culture or situation. They at -
tempt to frame questions so that the people 
in the survey answer within a common situ -
ation. However, this approach might not 
work well for evaluating schools, especially 
if teachers have incentives to coach children 
on taking these tests, so that they score bet -
ter and give answers perceived to be posi -
tive. Direct use of standard psychological 
measures can be problematic.36
Studies Measuring Skills Using  Behaviors
Ralph Tyler, who pioneered the development 
of achievement tests, recognized their limi -
tations.37 He suggested using measures of 
behavior such as performance, participation 
in student activities, and other observations 
by teachers and school administrators to 
complement achievement tests when evalu -
ating students and schools. Several recent 
papers demonstrate that this is a promising 
approach. Heckman et al. (2013) showed 
that teacher ratings of elementary school -
children’s behaviors are strong predictors 
of adult outcomes, and that early childhood 
interventions promote the noncognitive 
skills measured by these ratings. Heckman, 
Humphries, and Kautz (2014) estimated 
the causal effect of cognitive and socioemo -
tional (noncognitive) skills on a variety of 
outcomes. They measured socioemotional 
(noncognitive) skills using risky and reck -
less behaviors measured in the adolescent 
years.38 They developed and applied meth -
ods to use high school grades to measure 
both cognitive and noncognitive skills. They 
showed that noncognitive skills promote ed -
ucational attainment, beneficial labor mar -
ket outcomes, and health.
Jackson (2018) studied the effect of teach -
ers on students’ cognitive and noncognitive 
skills. In a fashion similar to Heckman et 
al. (2018), Jackson measured cognitive skills 
using achievement test scores, while mea -
suring noncognitive skills using absences, 
suspensions, grades, and grade progression. 