Around 600 students have a score on an extensive piece of assessment, which can be assumed to have good reliability/validity. The assessment is scored out of 100, and it's a multiple-choice test marked by computer.

Those 600 students also have scores on a second, minor, piece of assessment. In this second piece of assessment they are separated into 11 cohorts with 11 different graders, and there is an undesirably large degree of variation between graders in terms of their 'generosity' in marking, or lack thereof. This second assessment is also scored out of 100.

Students were not assigned to cohorts randomly, and there are good reasons to expect differences in skill levels between cohorts.

I'm presented with the task of ensuring that differences between cohort markers on the second assignment don't materially advantage/disadvantage individual students.

My idea is to get the cohort scores on the second assessment to cohere with cohort scores on the first, while maintaining individual differences within the cohorts. We should assume that I have good reasons to believe that performance on the two tasks will be highly correlated, but that the markers differ considerably in their generosity.

Is this the best approach? If not, what is?

It'd be greatly appreciated if the answerer could give some practical tips about how to implement a good solution, say in R or SPSS or Excel.

  • 3
    $\begingroup$ Great question! Are the final scores for the multiple choice and the essay portions supposed to be comparable (ie the same numerical ranges)? $\endgroup$ – gung Nov 13 '14 at 3:33
  • 2
    $\begingroup$ As I was writing this question I thought it might be up your alley! The final scores are broadly comparable, but a bit different. The mean on the multiple choice section is ~70 with a SD around 15. The mean on the other section is ~85 with a SD around 6. $\endgroup$ – user1205901 Nov 13 '14 at 3:37
  • 7
    $\begingroup$ I would be suspicious of any effort to solve this problem based only on the data you have described, because it would have to rest on the strong (and untestable) assumption that there is no interaction between cohort and performance on the two separate test instruments. If you possibly can, consider the option of conducting a separate small experiment to calibrate the graders. $\endgroup$ – whuber Nov 13 '14 at 4:14
  • 9
    $\begingroup$ To see better where the problem lies, suppose (hypothetically) that (1) the two forms of assessment are multiple choice and essay and (2) your older students tend to do relatively better on essay questions. When you use your data to make the scores "cohere" you will be confounding the grader effects with the age effects and, by making adjustments, thereby systematically disadvantage the older students compared to the younger. No matter how sophisticated an algorithm you choose, it can only paper over this basic problem. You need some additional data to resolve this confounding. $\endgroup$ – whuber Nov 13 '14 at 4:44
  • 3
    $\begingroup$ One thing to consider is how comfortable you'd be explaining the adjustment procedure to students or other stakeholders: many might feel that given a potential issue with the marking, putting some effort into a proper calibration of markers would not be too much to expect if the exam's an important one. $\endgroup$ – Scortchi Nov 17 '14 at 13:40

Knowing how graders differ is good, but still doesn't tell you what to compensate the grades to. For simplicity imagine just two graders. Even if we conclude grader 1 is consistently 5 marks more generous than grader 2, that doesn't tell you what to do with two students who were each graded 70, one by grader 1 and one by grader 2. Do we say that grader 2 was a harsh marker, and uprate that 70 to 75, while keeping the 70 marked by grader 1 unchanged? Or do we assume grader 1 was unduly lenient, knock his student down to 65 marks, and keep grader 2's 70 unchanged? Do we compromise half-way between - extending to your case, based on an average of the 11 graders? It's the absolute grades that matter, so knowing relative generosity is not enough.

Your conclusion may depend on how "objective" you feel the final absolute mark should be be. One mental model would be to propose each student has a "correct" grade - the one that would be awarded by the Lead Assessor if they had time to mark each paper individually - to which the observed grades are approximations. In this model, observed grades need to be compensated for their grader, in order to bring them as close as possible towards their unobserved "true" grade. Another model might be that all grading is subjective, and we seek to transform each observed grade towards the mark we predict it would have been awarded if all graders had considered the same paper and reached some sort of compromise or average grade for it. I find the second model less convincing as a solution even if the admission of subjectivity is more realistic. In an educational setting there is usually someone who bears ultimate responsibility for assessment, to ensure that students receive "the grade they deserve", but this lead role has essentially absolved responsibility to the very graders who we already know disagree markedly. From hereon I assume there is one "correct" grade that we aim to estimate, but this is a contestable proposition and may not fit your circumstances.

Suppose students A, B, C and D, all in the same cohort, "should" be graded as 75, 80, 85 and 90 respectively but their generous grader consistently marks 5 marks too high. We observe 80, 85, 90 and 95 and should subtract 5, but finding the figure to subtract is problematic. It can't be done by comparing results between cohorts since we expect cohorts to vary in average ability. One possibility is using the multiple choice test results to predict the correct scores on the second assignment, then use this to assess variation between each grader and the correct grades. But making this prediction is non-trivial - if you expect different mean and standard deviation between the two assessments, you can't just assume that the second assessment grades should match the first.

Also, students differ in relative aptitude at multiple-choice and written assessments. You could treat that as some kind of random effect, forming a component of the student's "observed" and "true" grades, but not captured by their "predicted" grade. If cohorts differ systematically and students in a cohort tend be similar, then we shouldn't expect this effect to average out to zero within each cohort. If a cohort's observed grades average +5 versus their predicted ones, it is impossible to determine whether this is due to a generous grader, a cohort particularly better-suited to written assessment than multiple-choice, or some combination of the two. In an extreme case, the cohort may even have lower aptitude at the second assessment but had this more than compensated for by a very generous grader - or vice versa. You can't break this apart. It's confounded.

I also doubt the adequacy of such a simple additive model for your data. Graders may differ from the Lead Assessor not just by shift in location, but also spread - though since cohorts likely vary in homogeneity, you can't just check the spread of observed grades in each cohort to detect this. Moreover, the bulk of the distribution has high scores, fairly near the theoretical maximum of 100. I'd anticipate this introducing non-linearity due to compression near the maximum - a very generous grader may give A, B, C and D marks like 85, 90, 94, 97. This is harder to reverse than just subtracting a constant. Worse, you might see "clipping" - an extremely generous grader may grade them as 90, 95, 100, 100. This is impossible to reverse, and information about the relative performance of C and D is irrecoverably lost.

Your graders behave very differently. Are you sure they differ only in their overall generosity, rather than in their generosity in various components of the assessment? This might be worth checking, as it could introduce various complications - e.g. the observed grade for B may be worse than that of A, despite B being 5 point "better", even if the grader's allocated marks for each component are a monotonically increasing function of the Lead Assessor's! Suppose the assessment is split between Q1 (A should score 30/50, B 45/50) and Q2 (A should score 45/50, B 35/50). Imagine the grader is very lenient on Q1 (observed grades: A 40/50, B 50/50) but harsh on Q2 (observed: A 42/50, 30/50), then we observe totals of 82 for A and 80 for B. If you do have to consider component scores, note that clipping may be an issue - I suspect few papers get graded a perfect 100, but rather more papers will be awarded full marks in at least one component.

Arguably this is an extended comment rather than an answer, in the sense it doesn't propose a particular solution within the original bounds of your problem. But if your graders are already already handling about 55 papers each, then is it so bad for them to have to look at five or ten more for calibration purposes? You already have a good idea of students' abilities, so could pick a sample of papers from right across the range of grades. You could then assess whether you need to compensate for grader generosity across the whole test or in each component, and whether to do so just by adding/subtracting a constant or by something more sophisticated like interpolation (e.g. if you're worried about non-linearity near 100). But a word of warning on interpolation: suppose the Lead Assessor marks five sample papers as 70, 75, 80, 85 and 90, while a grader marks them as 80, 88, 84, 93 and 96 so there is some disagreement about order. You probably want to map observed grades from 96 to 100 onto the interval 90 to 100, and observed grades from 93 to 96 onto the interval 85 to 90. But some thought is required for marks below that. Perhaps observed grades from 84 to 93 should be mapped to the interval 75 to 85? An alternative would be a (possibly polynomial) regression to obtain a formula for "predicted true grade" from "observed grade".

  • 1
    $\begingroup$ Unfortunately the nature of assessment 2 makes it impossible for the graders to look at more for calibration purposes. You can think of it as being like an oral poetry recitation that was done once with no recording, and which was assessed immediately afterwards. It would be impractical to schedule new recitations purely for calibration purposes. To answer your other question, Assessment 2 didn't really have clear subcomponents, and we don't need to consider component scores. $\endgroup$ – user1205901 Nov 18 '14 at 10:32
  • 1
    $\begingroup$ This is "not an answer" but in an ideal world I'd have suggested to turn things around and use an example sample (possibly of artificial assignments deliberately designed to be on grade borderlines, rather than by real students) as a way of training the graders to have the same generosity, rather than to deduce and compensate for their generosities. If the assessments are done this is clearly no solution for you, though. $\endgroup$ – Silverfish Nov 18 '14 at 11:23
  • 1
    $\begingroup$ (+1) Very thorough "not an answer". Consistency in rather subjective tests can often be greatly improved by splitting the grading task into components - otherwise one grader might be giving more weight to rhythm, another to projection, &c. $\endgroup$ – Scortchi Nov 18 '14 at 11:36
  • $\begingroup$ It is clear that in addition to submitting a possible adjustment to the person who will ultimately decide the issue, I will also need to submit some explanation of the pros and cons of adjustment. Your response provides a lot of helpful material regarding this. However, I wonder what criteria I can use to make a judgement on whether it's more beneficial to leave everything alone, or to make a change. I look at the cohort grades and my intuition says that the differences between markers are a having a big impact. Intuition is unreliable, but I'm not sure what else I can go on in this case. $\endgroup$ – user1205901 Nov 18 '14 at 11:47
  • 2
    $\begingroup$ One question is whether you have reasonable grounds to believe the "differential task aptitude" effect to be small, particularly when averaged over a cohort, compared to the "grader generosity" effect. If so, you might attempt to to estimate the generosity effect for each cohort - but you risk being confounded. Moreover, there is a Catch 22. I would be most cautious of applying large "corrections" to the observed grades. But if suggested corrections are small, it is plausible they are due to systematic differences in differential task ability between cohorts, not grader generosity at all. $\endgroup$ – Silverfish Nov 18 '14 at 13:19

A very simple model:

Let $s_{1,i}$ be the score of student $i$ on test 1, and $s_{2,i}$ his score on test 2. Let $A_1, \ldots, A_p$ be the partition of the students in the original cohorts.

Each cohort is biased by the strength of its students and the easiness of the grader. Assuming this is an additive effect, we back out of it the following way: we'll subtract the average score of the cohort on the first test, and add the average score of the cohort on the second test.

We compute an adjusted score $s'_1$ as follow

$$\forall j \leq p, \forall i \in A_j, s'_{1,i} = s_{1,i} - \frac{1}{|A_j|} \sum_{i \in A_j} ( s_{1,i} - s_{2,i} )$$

Finally, form a final score $s$ with whichever weighting you find appropriate

$$\forall i, s_i = \alpha s'_{1,i} + (1-\alpha) s_{2,i}$$

The downside is that an individual student might be penalized if the people in his cohort happened to get unlucky on the second test. But any statistical technique is going to carry this potentially unfair downside.

  • 3
    $\begingroup$ As with any other proposal, this one will suffer from the inherent unfairness of being unable to distinguish the grader effect from the group effect. There simply is no way around that. At least your procedure is a little more transparent than some others that have been proposed, by making its arbitrary nature obvious (in the choice of $\alpha$). $\endgroup$ – whuber Nov 17 '14 at 16:47
  • 1
    $\begingroup$ No - the cohorts aren't selected at random. $\endgroup$ – Scortchi Nov 17 '14 at 17:11
  • 1
    $\begingroup$ ... which, as @whuber keeps saying, is confounded with any inherent tendency of the cohort (owing to age or whatever) to do relatively better on one type of test than another. $\endgroup$ – Scortchi Nov 17 '14 at 17:28
  • 2
    $\begingroup$ You cannot eliminate confounding by taking larger cohorts! At best you can come up with ever more precise estimates of uninterpretable values. $\endgroup$ – whuber Nov 17 '14 at 17:31
  • 3
    $\begingroup$ Reasonable, perhaps: but it's untestable given the information available to the OP. The validity of your answer relies on the truth of this implicit assumption. Even worse, its negation (which of course is also untestable) is eminently reasonable, too: because cohorts are self-selected, they may consist of people who perform in common ways on different assessment instruments, suggesting it may actually be likely that differential success will be due in part to the cohort and only partially due to variability among graders. $\endgroup$ – whuber Nov 17 '14 at 17:57

You can't. At least, not without collecting additional data. To see why, read @whuber's numerous upvoted comments throughout this thread.



The problem solved in this answer is that of finding graders who give less points to the students they dislike.

Original post

My approach, which I think is easy to implement, would be as follows:

Let $\mu_{k, i}$ denote the grade student $k$, belonging to cohort $i$ got on assignment 1. Let $y_{k, i}$ denote the grade for the second assignment.


Assume the model

$y_{k, i} = \mu_{k, i} + \alpha + \tau e_{k, i}$ and estimate $\alpha$ over all students. That is, $\alpha$ is estimated without regard to $i$. It is possible that $\alpha$ is zero but students may change their performance for the final exam giving an overall increase/decrease.


Let $G_i$ denote the generosity for the grader in cohort number $i$. Then form $\tilde{y}_{k, i}$ and assume the model

$y_{k, i} - \mu_{k, i} - \alpha = \tilde{y}_{k, i} = G_i + \sigma_i \tilde{e}_{k, i}$

And do 11 individual estimates of $G$ and $\sigma$


Now an unusual observation is one such that the quantity

$T = \vert \frac{\tilde{y} - G_i}{\sigma_i} \vert$ is large. Select the largest of these quantities for every cohort and investigate them.


All $e$'s are assumed to be Gaussian. The grades are not normally distributed so guidelines on the size of $T$ are difficult to give.


Below is the code in R. Note that in your case, both mu and y will be given so the generating rows when they are assigned rnorm-numbers should be ignored. I include them to be able to evaluate the script without data.

mu_0 <- 50; 
alpha <- 5;
tau<- 10; 
# 0 Generate data for first assignment
mu <- matrix(rnorm(605, mu_0, tau), 11) 

# 1 Generate data for second assignment and estimate alpha
G <- rnorm(11, 0)*10;
for(i in 1:11){
    y[i,] <- rnorm(55, 0, sigma) + mu[i,] + alpha + G[i];

alpha_hat <- mean(y-mu)

# 2 Form \tilde{y} and find unsual observations
ytilde <- y - mu - alpha_hat
T <- matrix(0, 11, 55);
for(i in 1:11){
    G_hat <- mean(ytilde[i,]);
    sigma_hat <- sd(ytilde[i,]);
    T[i,] <- order(abs(ytilde[i,] - G_hat)/sigma_hat)
# 3 Look at grader number 2 by
  • 4
    $\begingroup$ You do not seem to have answered the question: you end only with a recommendation to investigate any "unusual observations." How does that solve the problem? $\endgroup$ – whuber Nov 17 '14 at 0:18
  • $\begingroup$ Reading the question again, perhaps I focused too much on the "individual" part. The problem solved in this answer is rather that of finding graders who give less points to the students they dislike. The original question is impossible(!) to solve. As already suggested, it is very likely that students collaborate or otherwise strongly correlate within each cohort. $\endgroup$ – Hunaphu Nov 17 '14 at 19:26

Rephrasing the problem: How best to approach setting a mark of a two part an exam with the conditions requiring that the second part is exposed to greater uncertainty due to the range of Delegated Markers' qualitative assessments.

Where: Master Tester = accountable person for exam Delegated Tester = person (1 of 11) assigned to mark par #2 of the exam Student = the guy that gets the fun of sitting an exam

Goals include: A) Students receive a mark that is reflecting their work B) Manage the uncertainty of the second part to align with the intent of the Master Tester

Suggested approach (answer): 1. Master Tester randomly selects a representative sample set of exams, marks the part #2 and develops correlation with the part #1 2. Utilise the correlation to assess all of the Delegated Markers' data (Part #1 vs #2 score) 3. Where the correlation is significantly different from the Master Tester - significance to be acceptable to the Master Tester - examine the exam as the Master Tester to re-assign the result.

This approach ensures that the Master Tester is accountable for the correlation and the acceptable significance. The correlation could be as simple as the score for part #1 vs #2 or relative scores for questions of test #1 vs #2.

The Master Tester will also be able to set a quality of result for Part #2 based on the "rubbery-ness" of the correlation.


protected by whuber Nov 19 '14 at 19:18


Would you like to answer one of these unanswered questions instead?
