Empirical Legal Studies: What Judges Don't Say: The Challenge to Content Analysis
« Guest Blogger: Carolyn Shapiro | Main | Relationship between Journal Prestige and Citations »
Posted by CarolynShapiro on 26 February 2007 at 09:51 PM in Courts & Judges, Guest Bloggers, Methodology, Scholarship | Permalink
In our paper "Systematic Content Analysis of Judicial Opinions," Ron Wright and I catalogue several instances of different researchers who attempt to grapple with this issue, as follows:
"Researchers can specifically examine the fidelity of reported facts, looking for indications of distortion or incompleteness, to determine if the facts are close enough to reality for use in statistical analysis. Hillman, for instance, in his prominent study of promissory estoppel, read a sample of cases more closely looking for any indications that courts stated there was reliance when there really wasn’t. He reports there was little evidence of misrepresentation of the facts, but he does not explain how he made this determination or why he felt he should be able to detect misrepresentation. Krawiec and Zeiler had students code cases both for whether the judge indicated the factor was present, and whether the student thought it was present, in order to “minimize the impact of conscious or unconscious judicial mischaracterizations of fact.” Only “rarely” did they find disagreement between the judge and the student coder. One interesting technique, used in four studies, is to compare facts reported in an appellate opinion with those reported in either the trial court’s opinion or a dissenting opinion. [Richards & Kritzer; Posner; Ignagni; Lawlor.] These researchers found few instances of disagreement or distortion. Lawlor is especially thorough. Using student coders, he tested for 30 possible facts in two Supreme Court cases that each had three opinions. Out of these 60 observations that provided 180 possibilities for discrepancies to appear, he found only two."
For full citations, go to: http://papers.ssrn.com/sol3/papers.cfm?abstract_id=913336.
I have indeed thought about comparing trial and appellate court opinions, and I think that might be very fruitful. At least I think it would be fruitful in terms of evaluating how reliable the trial courts' reports of the facts are. I think that the problems and challenges are similar to those I mentioned in my third post about whistleblowing dissents. For my purposes, an appellate court opinion that relies on facts the trial court ignored could be seen as a whistleblower too. I don't think, however, that these comparisons would shed as much light on how reliable the appellate court's reporting is, and that is, frankly, what I'm more interested in.
Posted by: CarolynShapiro | 02 March 2007 at 10:51 AM
With regard to the question of which facts are included and which facts are omitted, it might be much easier to find something interesting if one compared trial court decisions with appellate decisions in the same cases -- where the appeals courts should be accepting the facts found below -- rather than looking at trial court decisions -- where judges may well ignore the evidence in the course of rendering a decision. I've experienced both as a litigator. You get a much more sympathetic response complaining about the former than the latter, both because it's more egregious -- clearly eroneous vs. a question of discretion -- but also because to fully appreciate the latter, you kinda had to be there.
Carolyn, surely this has already occurred to you.
Posted by: Robert Mahnke | 01 March 2007 at 01:57 PM
Ah, ye olde non-decision problem! Admittedly this is a bear to contend with. Might I suggest another way to look at the conundrum?
The main problem here is to see what never reaches the table. One way to get around it might be counterfactuals. We are just looking at a fine example of it - Gaventa's Power and the Powerless - in my senior seminar. The way it might work here is to think of the kinds of issues that might arise on appeal in a particular case with existing precedential constraints. Then one can look at two things. First, what issues that arise on appeal are ignored and, second, what issues never arise, despite the expectation that they would, given the facts in play? This tracks Lukes's second and third dimensions of power. The issues that are ignored are obviously ones the justices don't want to make decisions about, for one reason or another. Those that never arise, although one might expect them to, are issues that have been socialized out of consideration.
How to approach these empirically? History is the answer. The first kind of non-decision is usually the product of a set of path dependent cumulative positive feedback loops that lead the judges to feel they can ignore the issue and the attorneys to feel that they at least tried, despite not having much expectation that they would prevail. The second is usually deeper and involves patterns in the professional socialization of the parties and long standing cultural barriers to raising issues in law. Or, to be more exact, so I would suppose.
Well! There's a nice, long, involved research project for someone younger and better funded. I'm always coming up with those.
Posted by: Tracy Lightcap | 28 February 2007 at 07:54 PM
I don't have much evidence to bring here, but I have read a lot of confession cases in the circuit courts, attempting to ascertain whether those judges mention the "right" facts in deciding their cases ("right" meaning the ones the Supreme Court says they should consider), so I thought I'd weigh in. I coded the cases trichotomously to attempt to ascertain whether they are paying heed to the High Court by considering all relevant facts; so, each fact is coded as "mentioned as present," "mentioned as not present," and "not mentioned." In so doing, it seems to me that the lower courts DO mention every conceivable fact that may be relevant (to the Supreme Court) in determining whether the confession is voluntary or not, so they don't appear to be ignoring facts (at least not in confession cases in the circuit courts). This still leaves us with a problem, though -- how do different judges "weight" different facts? It's surely possible (probable?) that they are interpreting the facts while discussing them, emphasizing those they deem to be dispositive, de-emphasizing those anithetical to their preferred disposition.
It seems like these are related problems and I agree that they are important to attempt to understand. Too bad I have no answers. But thanks for the post, Carolyn! I'll look forward to seeing what you come up with.
I agree that this is a great question. We really ought to know if/when judicial opinions are a reliable source for the facts of a case. If they aren't, that would have some profound implications for empirical research.
In courts where cases are randomly assigned to judges, the presence of case characteristics should be uncorrelated with the ideology of the judge. So you could easily compare different kinds of case characteristics to see if this is true. You might also want to think carefully about which kinds of case characteristics would be more or less susceptible to manipulation.
Posted by: Josh Fischman | 27 February 2007 at 02:38 PM
I think you're on to something important. This is potentially amenable to study by examining the briefs (or lower court opinions). That takes a lot of work, though.
An easier study that I'm playing around with examines justice's choices of what precedents to cite, using the briefs as a baseline
Posted by: frank cross | 27 February 2007 at 11:28 AM