It is often the case that scientists will be informed about the competing propositions regarding activities alleged by the parties only at trial, if at all (Risinger, 2013). It is generally understood that the propositions of interest are "(...) set by the specific case circumstances or as indicated by the mandating authority" (Willis et al., 2015, p. 6), but limited cooperation by the defense often represents an obstacle in practice7 . If propositions are not available, or only one proposition is available, scientists should make every effort to obtain relevant information8 regarding the position of each party involved in the process (see Willis et al., 2015 for complete guidance on how to deal with the absence of propositions), because without at least a pair of propositions, it is impossible to evaluate forensic observations in a balanced way.

It is a common misconception that the scientist who is evaluating the observations in light of competing posited activities needs to know every aspect of what has allegedly happened. For example, if it is alleged that the suspect grabbed the victim, aspects of DNA transfer will be important in considering activities. Only rarely will the scientist be provided with an exact recount of the position of grabbing, the force used to grab, the exact length of time the struggle lasted, and so on, to cover all aspects of the alleged encounter. However, there are several aspects that should be considered. Firstly, the manner in which the activities are, or have been, set up in controlled experiments to mimic the activity of interest are likely also to have similar aspects of uncertainty. Therefore, the uncertainty arising from many of the unmeasurable (and unknowable) aspects of the alleged activities will present themselves in the spread of the obtained data. Secondly, controlled experiments can be set up to study the impact that different factors have on the transfer of DNA during the activity in question. It may be that there is a large enough amount of variation in one aspect of the activity, such as the shedder status of the individual, that all others (such as the length or vigor of contact or when the person of interest last showered, etc.) have a negligible effect on the evaluations. In that case, this level of activity resolution is not required. Thirdly, if a number of aspects are found to have a considerable impact, then they can be included in the logical framework used to evaluate the findings. If the actual states of these important factors are not known (or not provided) by either party then they can be incorporated by considering all possible states within the evaluation, weighted by probabilities informed either by data from controlled experiments, supplemented by the analysts' knowledge, which should be available for disclosure and auditing (see also Section 2.2.2). If further information is provided later on, then the evaluation can be updated accordingly. Alternatively, sensitivity analyses can be used to determine how much of an effect any one of the unknown factors of the activities has on the value of the findings (Biedermann and Taroni, 2006). If the strength of the observations is particularly sensitive to some aspects then efforts should be made to find additional information about those aspects rather than every aspect of the activity. If scientists do not have such specialized scientific knowledge, the court will be even less likely to have such knowledge.

## 2.2. Data

2.2.1. "Because Each Case Has Its Own Features, the Use of Numerical Values from Experimental Studies Performed under Controlled (Laboratory) Conditions Cannot Be Used for Evaluation in Real-Life Cases"

This is a general claim (e.g., Jamieson, 2011) that conflicts with scientific practice. Throughout science, experiments are conducted in trials that reflect not all, but the essential, features of the problem at hand. Clearly, medical treatments administered to patients have not previously been "tested" on those particular individuals, but on other patients with the same disease. Similarly, the safety of consumer products (e.g., cars) is carefully assessed not by end-users but prior to marketing in a range of situations reflecting end-user profiles. Turning to forensic science, such as glass analysis, the phenomenon of transfer has been studied for a variety of factors, such as the mode of breaking (e.g., the number of blows), window dimension, etc. to build a model usable for assigning transfer probabilities in cases with features covered by this model (Curran et al., 1998, 2000). In the context of DNA, studies have been conducted to examine the rates of transfer, for example between shooters and guns (Polley et al., 2006), but also in more general situations (e.g., Phipps and Petricevic, 2007; Daly et al., 2012; Jones et al., 2016; Samie et al., 2016). So, when a scientist is faced with assigning a probability for finding trace material given the proposition of handling an object by a person of interest (e.g., the activity of discharging a firearm), we do see no harm in referring to studies that have focused on rates of transfer not exactly the same in the alleged circumstances of the case. Although some features of the individual case at hand may differ, nothing will

6The titles throughout Section 2 reflect our perceptions and summaries of recurrently encountered concerns. Specific references are cited within each subsection.

7 In all criminal justice systems that we know of, it is the right of defendants to remain silent and not incriminate themselves. Also, it is often considered strategic for the defense to make a statement as to the evidence only after the results of the analyses are known.

8 It is generally understood that scientists should concentrate on information that is relevant for the task at hand (i.e., so-called task-relevant information), keeping in mind challenges posed by human factors.

prevent the scientist from also judging that some additional casetailored experiments should be conducted in order to extend their knowledge and understanding, but case backlogs and limited resources may render this difficult. Besides, if a scientist refuses to assign a probability of observing some finding under a given set of propositions, there is a risk that the fact-finder will nonetheless assign such probabilities according to their own unaided judgment which, as highly-publicized past cases suggest, will often play out to the detriment of the defendant.

Contrary to a widely held view (e.g., Budowle et al., 2012), the availability of "hard" (i.e., numerical) data is not a necessary requirement for probability assignment. That is, the absence of data does not mean that no probability can be assigned. How is this possible? To understand this, it is important to recall that scientists can also derive probability assignments from their understanding of the principles of the process at hand, formulated in terms of a model. For example, weather forecasters cannot "play" the next day over and over again to find the number of times there will be rain on exactly the next day. This makes no sense, essentially because there is only one tomorrow and its weather will be observable only once tomorrow arrives. And yet, scientists are able to formulate previsions about the state of the atmosphere based on related data (e.g., today's weather) and their understanding of the relevant science and technology. Similarly, forensic scientists can make probability statements for outcomes based on their general understanding of a phenomenon. In genetics, for instance, there is knowledge about population structure and the ways in which genetic traits are passed on between generations. Take, for example, a crime stain with a haplotype for which no other occurrence is found in a relevant database: clearly, the observed relative frequency (i.e., zero) in the database does not mean that we should retain a zero probability for observing the same haplotype in another person from the population of interest. Instead, a value for the haplotype population proportion can be obtained using reasonable assumptions (Brenner, 2010).

In reply to the above, it may be suggested that forensic genetics is not an insightful example because of the sophisticated mathematical models available in this area, so let us consider the case of DNA transfer phenomena. Here, many studies have found, for example, that the quantity of DNA recovered after touching (i.e., primary transfer) a surface with bare hands varies approximately between zero and more than 150 ng, depending on the experimental conditions (e.g., Daly et al., 2012). But can such knowledge help formulate probabilities of finding DNA under the assumption of secondary transfer? We contend that it can, by constructing an argument. It is known, for example, that secondary transfer is conditioned on the amount of DNA transferred initially. Hence, when a quantity above, say, 200 ng is found—something not typically expected when touching with bare hands (i.e., primary transfer)—it can be argued that this should be considered an even less probable event assuming secondary transfer. Thus, despite the fact that explicit data for a particular secondary transfer scenario may not be available, forensic scientists can still convey domain-relevant knowledge intelligibly in probabilistic terms. It is important, however, to ensure that the data are relevant to the analytical methods used in the case of interest, because detected quantities after secondary transfer will be sensitive to the method used to collect material and to detect DNA. It will also be necessary to ensure that probabilistic models, such as Bayesian networks (see also below), used to interpret findings, are informed by such relevant data and available for auditing.

There remain, however, justified questions as to where and how to obtain data. Currently, results from empirical research are mainly published in peer reviewed scientific journals, but there is no systematically organized body of research. The idea of developing a knowledge base (Evett, 2015), to be shared among scientists who all contribute to the system, would thus represent a major contribution to strengthen the data-supported evaluations.

In summary, probability assignment is feasible and justifiable even with limited data, but must be amenable to a critical analysis. Further, despite the fact that data are collected under conditions that do not exactly reflect all the features of the case at hand, this does not preclude, in principle, these data from being used at least to some extent9 . Of course, this does not mean that any data are acceptable to support any claim, but—as noted above—data that the scientist regards pertinent. Moreover, expert assessment is not exclusively given by data alone; in fact, it never is because, while reference data have been collected in controlled studies, the probabilities we assign relate to one-off individual incidents. Instead, scientists use data to inform their judgment, by constructing an argument, explaining what data they have used, to what extent and why.

A topic related to the above viewpoint is the question of how to conduct assessments, i.e., reasoning in the face of uncertainty, whatever the data may be and the extent to which they are available. Often, one can note that scientists shy away from seeking support from conceptual devices that could help them structure their reasoning and, thus, avoid the impression of being overwhelmed by the inferential complexity of the evaluative task. It is thus worthy to mention one common method, known as Bayesian networks (BNs) (Evett et al., 2002; Biedermann and Taroni, 2012; Fenton and Neil, 2013; Taroni et al., 2014), for pulling together many aspects of information that need to be considered when activity level propositions are of interest. BNs are a graphical tool in which the problem can be constructed in a framework of logical inference10. The formulation of such a framework does not rely on having any data, it will in fact inform the analyst of what data is required in the evaluation of the findings. The formulation of a framework of inference should be the first step in any evaluation given activity level propositions. Sometimes, however, analysts claim that an insufficient amount of data exists, and they do so even before they know what data is

9On this point, see also Casey et al. (2016), who argue "(...) not evaluating DNA evidence in case work is potentially more dangerous and reckless than carrying out an evaluation based on limited datasets." We concede that this topic is delicate and it is important not to suggest that scientists allow themselves to suggest an answer as they please and hide behind statements such as "I am an expert."

10While there is substantial literature available on BNs for evaluating forensic DNA results (e.g., Biedermann and Taroni, 2012), BNs remain inaccessible to many biologists, mainly because of lack of training. An easily accessible repository with freely available examples that can be utilized with open-source software would be an asset to complement the idea of a knowledge base, as mentioned above.

actually required to help address the issues that are of interest in the case. Also, once constructed, and lack of data is found to be an issue for some aspects of the evaluation, there are still several avenues open to the scientists. These include the weighing of the various states that potential factors may take in terms of probabilities, informed by the scientist's documented knowledge and experience, and then conducting sensitivity analyses to determine how the evaluation changes as those probabilities vary over plausible ranges.

Reluctance to such introspective thinking, and expert probability elicitation in general, is surprising and odd to see among those scientists who would find no objection to be asked and to give opinions in court about the probabilities of various competing activities, regardless of whether relevant data exists, and regardless of whether they have undertaken some activity level consideration. In this situation, analysts are likely to express themselves non-numerically in the form of an answer such as "that is improbable," "I don't believe that is likely to have happened," "I think that sort of transfer is barely feasible," etc. So, analysts who would be willing to express themselves in such probabilistic terms about propositions, but refuse to provide probabilities for findings given propositions, exhibit an inherently contradictory position. We thus maintain that analysts who have considered their findings in a framework of logical inference, using experimentally derived data to assign probabilities and varying assignments for influencing factors will be in a much better position to usefully inform the court. This will include any limitations that characterize the data actually used (as well as detailing the information available to the scientist at the time of writing the report, see also Section 2.3.3).

## 2.2.2. "Expert Professional Experience Is Not Enough (Data) To Safely Assign Probabilities"

A recent exchange (Casey et al., 2016; Meakin and Jamieson, 2016) raised the latent issue of whether, and to what extent, expert experience forms an acceptable basis for assigning probabilities. As asserted in Champod (2014), and reiterated recently in Meakin and Jamieson (2016), the critical issue is disclosure of data and making it available early enough in the process in order to allow for a proper consideration by the defense. The deeper issue, however, appears to lie in the notions of expert experience and so-called "personal" probability assignments. The ENFSI Guideline, for example, mentions expert experience as one possible source for informing the process of probability assignment: "Such data can take, for example, the structured form of scientific publications, databases or internal reports or, in addition to or in the absence of the above, be part of the expert knowledge built upon experiments conducted under controlled conditions (including case-specific experiments), training and experience" (Willis et al., 2015, p. 19). This should not be read as meaning that a vague reference to personal experience is on a par with other, more structured data. This would amount to misconceiving the fact that there is actually a hierarchy in the data, with a clear preference given to scientific publications and otherwise widely accessible scrutinized data.

## 2.2.3. "Evaluations Given Activity Level Propositions Are Massively Vague and Hence Cannot Be Trusted"

This objection may be the result of the discomfort that can be experienced when faced with incomplete knowledge about factors that influence the assessment of the probative value of scientific observations. However, incomplete knowledge, and hence uncertainty, do not per se prevent the conduct of science and its operational use in legal proceedings. What is more, in all parts of legal and everyday practice, one needs—inevitably to act despite knowledge being incomplete. It is the very task of science, thus, to quantify the extent of available knowledge so that it can be used in an informed way. The reply "it's possible"11 when confronted with the event of transfer or contamination, as scientists still often do in criminal proceedings, is a vague statement and is not a quantification of knowledge as we understand it in the discussion here.

The view that partial knowledge can be used is challenged, for example, when outcomes are subject to variation and scientists refrain from addressing them, equating variation with "no knowledge" about the topic. Forensic examination of glass provides a telling illustration for this. Research has shown that the quantities of glass fragments transferred to the surfaces of the clothing of the breaker vary considerably even for experiments with the "same" controlled conditions (e.g., regarding mode of breaking, distance between the breaker and the window, etc.). But does this mean that we "know nothing"? Clearly, scientists have knowledge about the phenomenon of glass transfer12 in the sense that they won't expect to encounter all possible numbers of fragments with the same probability. For example, depending on factors such as the distance between the breaker and the window, the mode of breaking, the time since window-breaking and seizing a suspect's clothing etc., scientists may consider it more probable to recover less than, say, five glass fragments, rather than more than five. It is the scientists' core task to elicit and convey such expressions of expert knowledge, because no one else in the proceedings is in a better position to do this. It may be a challenge for scientists to provide probabilities for recovering exactly 0, 1, 2, ... fragments (although simulation approaches exist e.g., Curran et al., 1998), but it is feasible also to choose a strategy going from the general to the particular, starting with probability assignments for apportionments of fragments such as "none," "few," "some," and "many." This helps break down the difficulty of probability assignment and make particular assignments more intersubjectively acceptable.

What is important is not the variation per se but how different the expected outcomes are given both propositions. Imagine that 2 min after a window is broken, a person is arrested and his sweater searched for glass. More than 80 fragments (sharing the same physical properties as the broken window) are recovered. Is this result more probable given that he broke the window or given that he had nothing to do with breaking incidents? In

11The same applies to "could have"; and there have been several notable judgments where courts have ruled against the unqualified use of such phrases.

12A further relevant factor is background presence. Regarding DNA, there is limited knowledge about naturally present DNA in the environment, which is especially important in cases where the defendant and victim cohabit (as in the Amanda Knox case, for example Gill, 2016).

some breaking experiments, it was observed that the number of fragments transferred was between 44 and 241 (with a mean of 127, see Hicks et al., 1996). So, there is variation, but when one looks at persons who have not broken windows and searches their garments, one finds that, in general, on sweaters, there are only between 0 and 2 fragments (sharing the same characteristics, see Coulson et al., 2001). Thus, clearly, finding 80 fragments is much more probable given one proposition than the other, despite the variation observed. Using the data from both surveys, one can assign a probability to the results given each proposition. In cases involving DNA, most studies focus on one activity only, but what is important is the comparison of the probability of the outcomes given both the alleged activity and at least one alternative. This comparison will then enable us to see if the variation that we have observed has an impact or not on our conclusions.

One further point that warrants a comment is "vagueness." We would strongly advise against the use of this term as a qualifier for a forensic evaluation if that evaluation has been conducted thoroughly. Let us recall again the undisputed starting point: there is variation in findings. What scientists do is to accommodate this through probability. What does this mean? It means that scientists will assign probabilities to the various outcomes depending on the extent to which they expect them to occur. For example, consider a case in which a victim has been punched several times to the head, resulting in profuse bleeding. Given the proposition that the suspect (arrested immediately after the assault) is the assailant, we can postulate one main, potential outcome: finding, on the suspect's fist, blood with a DNA profile corresponding to that of the victim. A fairly high probability (i.e., toward the upper end of the range between 0 and 1) may be assigned for this particular finding. By coherence, other findings such as no blood at all, or blood with a profile different from that of the victim will thus be assigned lower probabilities (i.e., toward the lower end of the range between 0 and 1). So, there is variation in the potential findings but there is a major finding that dominates our expectations. Consider now a different version of this case, in which the assault was less violent, did not result in the bleeding of any protagonist, but involved several victims. In such a case, it may be necessary to specify a broader range of potential findings, possibly including mixtures. In the event that the person of interest is the assailant, there may not be one outcome that stands out over all the others. Instead, one's probabilities for several of these potential outcomes may be quite similar. Thus, probability will be distributed over several outcomes, with no outcome receiving a probability assignment close 1. The assigned probabilities will express less strong beliefs (in the various outcomes) but—and this is the important point this does not mean that the assignment as such is vague. Less strong beliefs simply reflect the fact that one is less affirmative. Every statement of probability expresses a particular state of uncertainty, that is a well defined opinion, but none of these expressions are deficient if they are derived properly to reflect the expert's current state of knowledge.

So, even if the scientist may report a neutral finding13 , due to limited expert knowledge (to enable the assignment

13In more technical language, this would correspond to a likelihood ratio of 1.

of probabilities that are different in the numerator and the denominator), this is still an important evaluation to present to the fact-finder. If nothing else, it will inform the decision-maker that they need rely on non-DNA evidence to decide the case.

## 2.3. Reporting

## 2.3.1. "It Is Impossible to Know from the Quantity of DNA Obtained, and the Quality of the Profile, Whether the DNA Was Deposited by Direct Contact or Indirect Transfer."

The concern expressed in the section title is also sometimes seen as the problem of whether we can determine or, as noted by some discussants, "deduce" whether a given finding is the result of primary or secondary transfer. The misconception here is not to understand that the process is not deductive14, but remains inductive. Hence one cannot "know for sure"—but one can offer guidance, in the form of probabilities for the results, to help fact-finders decide on the truth of the propositions of interest.

More generally, the claim that particular observations do not allow one to draw categorical conclusions about a particular activity is uncontested and also holds for many, if not all, types of forensic traces. Taking glass as an example, no proficient forensic scientist would conclude that finding a number x fragments is the result (or the probable result) of smashing a given window. Similarly, finding a number y particles of gunshot residue does not allow one to say that the person of interest discharged a firearm, to the exclusion of other propositions. The impossibility of such direct "jumps" from observations to conclusions in these examples does not derive, however, from the fact that the trace material is present in small quantities. The shortcoming in the reasoning also holds for the so-called macro-traces. To illustrate this point, imagine that large quantities of fresh blood are observed on the hands of a person of interest. Such a result does not entitle one to argue that the exclusive or probable cause is stabbing the victim. Depending on the case circumstances, trying to help the victim may also be a viable proposition.

As discussed, the scientists' task, when operating in evaluative mode, is not to "infer activities" but to provide expressions of probative strength to help the court discriminate between competing propositions regarding activities. This requires the scientist to assign probabilities for the DNA results as obtained in the case at hand given each of the propositions of interest. The fundamental question associated with probative value then is: "Under which of the competing propositions regarding activities do we consider the findings more probable?" It may be that scientists think that they have no reason to consider the observations more probable in one version of the events than another. But this will not be a defect of reporting given activity level propositions, nor of the framework of evaluation. It only means that, in the current state of knowledge, the findings do not have any discriminative capacity (in a technical sense, such results would have a likelihood ratio of 1). As discussed, this is a well-defined result and should be reported, so that people are not prosecuted on the basis of forensic results that are not probative at this stage. As much as it is useful for a recipient of expert

14On the notion of deductive logic, see also Jackson et al. (2013), for example.

Frontiers in Genetics | www.frontiersin.org

information to hear when observations support one proposition rather than another, and (if possible) to what extent, it is useful for them to know when findings do not allow them to alter their beliefs in the propositions of interest.

In other terms, it is not a matter for the scientist to say whether a proposition, such as "Mr. A stabbed the victim (i.e., the DNA is from primary transfer)," is true, given the forensic observations, but the extent to which she expects to see these observations, given the proposition "Mr. A stabbed the victim." The scientist should be assessing the probability that DNA would be transferred, that it would persist and that a matching profile would be obtained, given the truth of this proposition. But there is one more dimension to the latter question. In order to be balanced, scientists must not only think about their results, given one activity, but also given at least one alternative activity (for example, that the suspect handled the knife innocently after the incident), and assess whether, and if so to what extent, the observations are more probable given one activity rather than another. It is therefore of paramount importance that scientists do not confuse the probability of primary transfer with the probability of observing the results if Mr. A stabbed the victim. We agree that the difference is very subtle and this is a reason why, in the propositions, one ought to describe activities and not use the terms "primary/secondary transfer15." This allows one to distinguish, on the one hand, what the court will assess (i.e., activities), and on the other hand, what the scientist will assess, that is the probability of observing the results given the activities. One of the terms used to assign the latter probability is commonly known in the literature as the "transfer probability" (Evett, 1984; Evett and Buckleton, 1989) 16 . We thus stress that a transfer probability focuses on the findings, given propositions, not the reverse.

## 2.3.2. "You Cannot Say That He Stabbed the Victim! The Only Thing That DNA Allows You to Say Is That He Had Recent Direct Contact"

This objection is often heard from recipients of expert information when the propositions of interest in the scientist's report are formulated closely to the specific actions that define the crime. For example, propositions such as "he handled the knife," "he punched the victim," "he fired the gun," may provoke such objections. It is often felt that less specific formulations such as "he is in contact with" are more appropriate. However, this objection stems from a misconception about the role of the scientist with respect to the propositions. As noted at the end of the previous Section, by writing down propositions in their report, scientists are only "setting the context" in which the findings will be assessed. That context is given to the scientists by the court and/or the parties and this context naturally relates to the alleged actions. Hence, scientists do not express any opinion directly on those propositions, notably regarding their truthfulness, adequacy or otherwise. The scientist's reporting only focuses on the weight to be assigned to the DNA findings in light of these propositions. Scientists should not suggest in any way that they are in a position to say, for example, that "he handled the knife," or that "he was in direct contact with the knife." If they do, they fall for the same fallacious thinking explained above. The only opinion they are allowed to express is in relation to the probability of the DNA findings if one or the other proposition is true. Specifically, when the scientist writes that "this amount of DNA is what we expect if Mr. A. stabbed the victim," the scientist is reporting only about the DNA results, and is not taking any stance on whether or not Mr. A. stabbed the victim. The latter is simply what is alleged by the parties in their own terms.

## 2.3.3. "Because Many Lawyers May Lack Awareness as to the Problem of Transfers, Analysts Should Flag the Issue in Their Reports Whenever the Analysis Process Suggests That Various Transfer Mechanisms May Explain the Findings"

Explaining the observations is a procedure that would be acceptable for the scientist to perform if they were at the investigative phase and not being asked to evaluate the forensic biological results in the context of the case, at court. To clarify this point, it is useful to recall the following two fundamentally different perspectives. In the investigative phase, observations are taken as a starting point. They are used to suggest what happened (i.e., activities). For example, one takes the finding of small quantities of DNA on the suspect's shoe as a starting point to suggest that the suspect was the person who kicked the victim. The other perspective takes propositions as a given (as it would be the case in court), to reason about the findings. One assumes that the suspect is the person who kicked the victim, and then one reasons about the kind of trace pattern one would expect to observe on the suspect's shoes. In evaluation, it is the latter perspective that is appropriate for forensic scientists. As noted by Margot, "[w]hether these results could be observed if one proposition for the event is true rather than another proposition is the central relevant matter on which the forensic scientist may comment" (Margot, 2011, p. 796). Note however that there may also be more than two propositions of interest.

At this juncture we would like to include a brief note on the distinction between explaining the observations17 and evaluating them, as well as the difference between explanations and propositions (Evett et al., 2000a). We often hear that, after scientists do all the complex evaluations that activity level propositions may require, and provide their results on the stand, the defense are just going to suggest an explanation. For example, the defendant may argue that he spat on his hand as he was walking down the street and touched a bench on which the victim later sat, or some other explanation. It is worth stating that this is explaining the results, and that the defense18 provides such explanations once the results are known. Therefore, such explanations are generally based on the results and may not be based on the relevant circumstantial information in the case. Such explanations do not count as acceptable,

15For more on this topic, we refer the reader to Hicks et al. (2015).

16Note that there are also phenomena of persistence and detection/recovery to be taken into account.

17While technically the word "observations" is to be preferred, we will be using the more colloquial word "results" to refer to the outcome of the scientist's analyses.

18The problem of post-hoc rationalizations is not restricted to the defense as explanations can also be brought up by the prosecution. See, for example, the bleach cleaning hypothesis in the Amanda Knox case (Gill, 2016).

formal propositions, because one cannot meaningfully assess the probability of the results given explanations that themselves are merged with the results (i.e., this would constitute circular thinking). Explanations are generated post-hoc in order to account for the results. They can be statements of the blindingly obvious, they can be speculative or they can be fanciful, having no logical connection with the circumstances of the case, even to the point of having no grounding in reality (see in particular Evett et al., 2000a and more recently Jackson et al., 2013 for examples and detailed discussion). In contrast, propositions are formal statements of competing allegations or suggestions that are dictated by the relevant circumstances of the case and not by the results themselves. So if changes are suddenly brought up at trial, scientists need to be careful not to give ad-hoc assessments where evaluation would require detailed checks with relevant literature and specialized knowledge. This is also why, for example, the ENFSI Guideline (Willis et al., 2015) emphasizes thatscientists should mention in their report that their evaluation is based on their understanding of the relevant circumstances at the time of writing the report and that if any assumptions or information is incomplete or incorrect, they will have to re-evaluate their findings19 .

The above distinction between explanations and propositions is crucial and it is worth to summarize and relate it to standard notions from other inferential disciplines. Characteristically, explanations account for—or are made to "fit around"—the findings that have been made in a case. Explanations entail a deductive mode of reasoning as they seek to explain existing results, typically in causal terms. As such, forensic explanations are generated and considered after relevant observations are known. Often, the generating process for explanations results from abductive reasoning not limited to the forensic scientist, but may also extend to case investigators. As such explanations are theoretically open-ended, with no limit on their number though some of the explanations may be more or less fanciful (e.g., not testable in a logical sense), implausible or even speculative than others, to the point that no meaningful probability may be assigned to them. Unlike explanations, propositions are formal statements that can be clearly related to the case context and subjected to a proper inductive mode of reasoning.

We would suggest that consideration and proposal by a scientist of the various possible modes of transfer to account for DNA findings may be of use in the investigative phase of a case. But scientists should not systematically explore and comment on all conceivable mechanisms of transfer (so-called caveats) in their statements (but may do so within the lab-file, or in a "Technical Issues" section of the report Evett et al., 2000b). Moreover, when a case enters the evaluative phase, and particularly when in court, a scientist should resist offering a view on explanations for transfer but concentrate on evaluating probabilities for the results given formal propositions based on the circumstances of the case. Advancing explanations at the evaluative stage amounts to treating transfer dismissively, rather than considering its impact on probative value in a formal and explicit way.

## 2.3.4. "The Safest Course Is for an Analyst Simply to Report the Results of the DNA Test, Alert Both Counsel and the Jury to the Possibility of Transfer, and Leave the Jury or Factfinder to Assess Their Implications"

This argument is a similar to the previous one, about caveats, but here the burden of how to assess the implications of transfer is left to the factfinder. We are of the opinion that leaving the factfinder to assess implications of transfer threatens the appropriate conduct of the forensic findings: if scientists do not—or cannot—evaluate their results, then how could the factfinder do so? Hence we find this position problematic. Clearly, proceeding in this way is an easy course for scientists, because it reduces their task to technical reporting, but it could be very misleading for innocent defendants because findings will be left uninterpreted at the propositional level that really matters (i.e., activity level). Forensic scientists have (or should have) specialized knowledge on transfer and persistence, as shown by publications in forensic journals, and they therefore have the duty to report the value of their results at that level. If the knowledge is not sufficient, then scientists must tell the instructing magistrate or the court (or preferably even before, earlier in the process) that, as a consequence, their results do not help discriminate between the propositions at hand. We do not believe that in the evaluative phase—scientists should provide a list of all theoretically possible modes of transfer of DNA (see also Section 2.3.3). If the scientist were to provide such a list, how does the court choose which is the most likely mode of transfer? This would leave the court in the difficult position of having to choose which of a potentially large number of possibilities (that are not necessarily exhaustive) is the most likely, without being able to rely on any specialized knowledge to do so. However, we do believe that it is the proper role of the scientist to talk generally about transfer and persistence of DNA (see also Section 2.3.3).

An intricacy related to the above is the use of the term "possible." As human beings, we refer to a lot of events as being "possible" (i.e., the probability of the event is not 0), but forensic scientists should be more informative than this: they should assess how probable their results are given the propositions at hand, just like they do when they assess the probability of observing a given DNA profile if it came from some unknown person. If a scientist were to be asked "what is the probability of obtaining a matching DNA profile if it came from Mr. A or if it came from someone else who happens to have, by coincidence, the same profile," which is an explanation, the scientist would have to answer that those two probabilities would be the same (i.e., approaching 1). Therein lies the problem for the scientist and the court, generating explanations leads to probabilities for the results of a value approaching 1. Provision of explanations is deeply rooted in general forensic science thinking and we regularly see reports in which the scientist writes "It is possible that this DNA comes from Mr. A. But, it is also possible that it comes from his brother or an unknown person." This sort of explanation-based answer is unsatisfactory because it leads the

19Interruptions in the proceedings can be granted by the court both in inquisitorial and adversarial proceedings, and will usually be granted if the question is of importance to the court.

scientist to opine directly on a proposition (see also Section 2.3.2 regarding the role of the forensic scientist).

## 2.3.5. "When Scientists Are Unable to Evaluate Their Observations Given Activity Level Propositions, Then They Should Retreat to Evaluations Given Source or Sub-source Propositions"

This claim rejoins Section 2.3.4, which refers to the claim that an evaluation given source level propositions is a "safe course" for forensic scientists, when they cannot help address activity level propositions. However, what a safe course of proceeding is for the scientist may again not be so for other participants in the process20. The problem is that the "safe course" for the scientist inevitably restricts the evaluation to the (sub-) source level. Consequently, the court is given no guidance about how to evaluate with respect to activity level propositions. If the court confuses the scientist's (sub-) source evaluation with the activity level evaluation, then there is a risk that this may lead to a miscarriage of justice21 .

For the above reason, recent recommendations by ENFSI specify that the choice of source level propositions is limited to well defined situations, that is "(...) cases where there is no risk that the court will misinterpret [the findings] (...) in the context of the alleged activities in the case" (Willis et al., 2015, p. 12)22 . But for small quantities of trace material, this is rarely if ever the case, because such traces require expert knowledge "(...) to consider factors such as transfer mechanisms, persistence and background levels of the material which could have an impact on the understanding of scientific findings relative to the alleged activities" (Willis et al., 2015, p. 11). For all of these reasons, the ENFSI guideline concludes that "(...) the choice between (sub-) source and activity should not be influenced by the availability of data or expert knowledge but solely from the consideration of factors such as transfer, persistence and background levels that could crucially affect the strength of the findings within the context of the case circumstances" (Willis et al., 2015, p. 13). This includes a statement of limitations as to the data and the individual expert knowledge (see also Section 2.2.1).

An objection that may be raised against the position outlined above is its feasibility. That is, although activity level propositions may be recognized as the relevant propositional level, specialized knowledge necessary for evaluation given this level of propositions may be unavailable. A natural consequence of this starting point would be not to introduce the results at trial, in order to protect defendants against unwarranted interpretations in such cases. However, there appears to be no consensus among scientists on how to proceed in such situations. Some scientists maintain their intention to report findings given source level propositions although they are clearly unable to help with the issue of activities. As a consequence, we do not subscribe to this view of retreating to evaluation given (sub)source level propositions, and neither does the ENFSI Guideline, which requires scientists to clearly acknowledge that their evaluation falls short of the real needs. In Guidance note 2, the Guideline states that "(...) if the examiner chooses (...) to report the findings at source level (...), the examiner shall explicitly state that the rarity of the profile does not address the question of the relevance of the findings in relation to the alleged activity" (Willis et al., 2015, p. 14).

Proponents of the view according to which uninterpretable results should be mentioned at trial appear to misconceive the fact that different stages in the forensic process have different requirements (Jackson et al., 2006, 2013). It is beyond dispute that, at an investigative stage, scientists can help the process move on when they factually report about the observation that a defendant's traits are also observed in trace material (e.g., in the case of mixtures). This is useful information for selecting possible candidates on whom to focus further investigations. At trial, however, the requirements are different. At trial, the defendant has already been selected, and if DNA is to play any further role, it must be given a weight (Evett, 2015) not against any propositions, but propositions at the relevant level.

## 2.3.6. "When Evaluating Forensic DNA Traces Given Activity Level Propositions, the Scientist Infringes on the Duties of the Court"

A perception encountered among legal practitioners, as noted earlier in Section 2.3.2, is that when evaluating DNA traces given postulated activities, scientists take on the role of the fact-finder. This observation is a cause of concern because it does not reflect the scientist's intention and laying bare this misconception is challenging. We think that there is merit in reiterating that it is not for the scientist to give an opinion on whether the transfer is primary or secondary (or the probability that the transfer is primary or secondary) because giving such an opinion would amount to giving an opinion on the propositions of interest, for example whether "Mr.  A" (transfer was primary) or "Mr. A and Ms. A attended the same party, but had no particular interaction" (transfer was secondary). Clearly, this is a question for the court.

The above distinction is very subtle, for all discussants, including scientists. It comes down, in one way or another, to the problem of the transposed conditional. Many authors have formally described the contribution of the scientist and the nature of expert opinion in the criminal justice system, with the one key aspect being that the scientists' role is to evaluate their results given the competing propositions regarding activities, and that it is for the court itself to assess the truth of the

20See for example cases such as Jama (Gill, 2014, p. 27) or the Ruelas Case (Murphy, 2015, p. 56) which illustrate that the sole consideration of sub-source issues does not accommodate and represent an evaluation of the joint probative value of the results from different swabs, the quantities of DNA found, nor the presence or absence of other trace materials. When considering activity level propositions, all observations should be assessed, which is what is needed.

21See http://www.scientificamerican.com/article/when-dna-implicates-the-inno cent/ for recent case involving a stupefying situation in which a forensic DNA report found a correspondence between the DNA profile of a hospitalized person and DNA found on a murder scene. It appeared that the same paramedics treated the hospitalized person but also worked on the crime scene a few hours later (see also https://californiainnocenceproject.org/2013/06/how-an-innocent-mansdna-was-found-at-a-crime-scene/).

22This may be the case, for example, when there is a large and fresh bloodstain at the point of entry (e.g., broken window) and it is not contested that the blood stain is relevant to the case (i.e., the trace is a direct consequence of committing the burglary).

propositions. Unless scientists are operating at the investigative stage, they should express probabilities only for their results given propositions, but not the reverse. An example of a relevant statement would be: "The probability of observing this quantity of DNA if Mr.  A as alleged by the prosecution is in the order of 0.6, whereas the probability of observing this quantity of DNA if Mr. A had social interactions as alleged by the defense is in the order of 0.01. This means that it is about 60 times more probable to observe this DNA result if the prosecution's case is true rather than if the case of the defense is true." However, there is the risk that the receiver of this information will interpret the low probability for the findings, given the alternative proposition, as meaning there is a low probability for the proposition, given the findings—a reasoning error that is known as "transposing the conditional," and which was not intended by the scientist. This is why some reporting agencies explicitly mention, in their written reports, examples of sentences of what their conclusions do not mean.

## 3. CONCLUSIONS

From the discussion presented throughout Section 2, three main points emerge:

First and foremost, forensic interpretation, as conducted by the scientist, focuses on the observations, not on the propositions. Stated otherwise, the question for the scientist is "What is the strength of these findings with regards to the propositions of interest?" The scientist should not attempt to answer the question "How probable are the propositions given the findings?" Scientists do not evaluate and provide an assessment of the probative strength of scientific findings when they express opinions on propositions. Hence—for the scientist—evaluation given activity level propositions does not mean to opine, probabilistically, on competing activities that may have "caused" the findings. Evaluating forensic results means to provide information that helps the recipient of expert information discriminate between propositions, whatever their belief in those propositions is prior to hearing the scientific findings.

Second, reporting on the probability of the observations given competing versions of the case, regarding activities, does not exclusively depend on numerical data, but is also informed by expert knowledge and experience, for which scientists can provide appropriate documentation and demonstrate how it shapes their opinion. What is more, the scientist invokes information that is available for disclosure and auditing. An important corollary of this is that even though task specific data may be unavailable or scarce, it does not mean that no probability can be assigned. In particular, this is not to suggest that any opinion, or mere guesswork, is a valid substitute for thorough scientific assessment. It highlights the need for the elicitation of expert probabilities and knowledge through formal methods and techniques, known also in other areas of specialization, such as risk and safety assessments (e.g., O'Hagan et al., 2006; Aven and Reniers, 2013). There is merit in further developing these approaches for forensic science applications, as well as strengthening the body of structured knowledge (i.e., relevant data on phenomena such as transfer and persistence) for various types of forensic traces (Evett, 2015). This rejoins the idea of developing a knowledge base system that would include experiments and exemplar probabilistic models for evaluation (e.g., BNs). This is widely considered a critical step that the field needs to take now.

Third, variability in the observations (e.g., with respect to quality and quantity of transferred material) observed in experiments under controlled conditions, is both natural and expected. It does not mean that such data cannot be used for evaluation in actual cases, nor does it mean that no conclusion may be drawn. This view is also supported by professionally organized forensic caseworkers (Casey et al., 2016). Variability is an inevitable feature of scientific experiments, observations and measurements, and produces uncertainty. The scientific approach to such uncertainty is to capture it by probability and to take it into account in the scientist's evaluation (e.g., it will be ensured that the data used come from experiments that relate directly to the analytical methods used in the case of interest). Therefore, variation per se is not a primary matter of concern; what does matter for the scientist is to see whether the probability of the outcomes given different propositions varies. That is, for the results to be useful, the outcomes need to be more probable given one version of the case (i.e., proposition) than given an alternative version of the case. It is on this latter issue that the scientists need to focus their attention.

The above observations diffuse the call for so-called "unpredictable" forensic DNA traces, in particular low quantities, to be withheld from being used in the process. This is so because the perceived drawbacks, although inspired by known difficulties, do not properly acknowledge additional levels of scientific observations (e.g., extrinsic features such as the quality and the quantity of recovered material, and the position in which it was found) that may be available and that characterize a comprehensive evaluation of forensic results. This perspective goes beyond the mere assessment of the rarity of the analytical features (i.e., the genetic profile). Indeed, for decades, forensic scientists and recipients of expert information have found comfort in seeing forensic DNA analyses provide "constant" and "stable" results in the sense that the DNA profile observed for a sample from a person of interest will, broadly speaking, be observed to be the same for a stain left by that person - as long as quality and quantity of the staining are appropriate, and the chain of custody is impeccable. To a large extent, this has led to technical efforts and investments being spent on ensuring that analyses will reveal the same profile for materials that come from the same source. This is, undoubtedly, an important preliminary requirement for use in forensic science. Unfortunately, however, this perspective was accompanied by the idea that all that is necessary to assess the strength of the findings is an assignment for the probability of observing the profile of interest for an unknown person. This focus on analytical accuracy and rarity of features conflicts with the intricacy of additional dimensions that DNA profiling entails, such as the very fact of finding DNA at a particular place on a receptor surface (i.e., extrinsic aspects). Stated otherwise, what we have come to see now are conventional interpretation schemes conditioned mainly on source (or sub-source, e.g., in the UK) level propositions being applied to questions, issues and challenges for which these schemes have not been designed, and this has the potential to create stupefying situations in which reports on forensic DNA results are at odds with the case as a whole23. Worse still, evaluation given sub-source and source propositions alone can lead to an over-valuing of the scientific evidence, risking miscarriages of justice (Gill, 2014; Jackson, 2014).

The above calls for a readjustment of perspective. To ensure that forensic DNA results are meaningfully used in the legal process, scientists must work on improving their knowledge and understanding about additional factors that characterize not only intrinsic features (e.g., DNA profile) but also extrinsic features (e.g., location where DNA was found). This call is not new (e.g., Evett and Weir, 1998; Taroni et al., 2013; Champod, 2014), but we see that the field is rather reluctant and awareness increases only slowly. At the same time, reports accumulate on real cases (e.g., Gill, 2016) where DNA turned out to be a source of conflict essentially because the key issues for the court related to activities whereas the scientist evaluated the findings in light of questions of source. Thus, evaluation given activity level propositions corresponds to a real need and we foresee that both prosecution and defense counsels will intensify their probing of forensic science regarding this propositional level, not least because recently issued guidelines (i.e., Willis et al., 2015) on evaluation and reporting explicitly set this forth as the standard of interpretation. Achieving this standard is a delicate and challenging endeavor because it operates at the frontiers of current knowledge. However, by gathering, sharing and organizing specialized knowledge in a structured and systematic

## REFERENCES

way (i.e., a shared knowledge base), the forensic community as a whole has the potential to work toward (i) increasing the number of cases in which findings can be assessed given activity level propositions, and (ii) rendering activity level evaluations more trustworthy in those cases where such evaluations are feasible.

In this paper, a discussion format has intentionally been chosen. The aim was to concentrate and restate replies to recurrent objections to emphasize on the need to pursue this topic from a broad perspective, associating both forensic scientists and lawyers. In view of all the arguments presented, our view is that evaluation given activity level propositions represents a main point of the agenda of future research. Besides the justified calls for more structured expert knowledge, we also recognize the need to report on more practical case examples that demonstrate the feasibility of this perspective in a way that practitioners can understand. Such reports on practical examples exceed the space available in this communication, but is the object of ongoing collaborative work between the authors.

## AUTHOR CONTRIBUTIONS

All authors listed, have made substantial, direct and intellectual contribution to the work, and approved it for publication.

## ACKNOWLEDGMENTS

Research reported in this paper was presented at the 2nd International Symposium on Sino Swiss Evidence Science (2nd ISSSES), University of Lausanne, School of Criminal Justice, September 5th to 9th 2016. The 2nd ISSSES was supported by a Grant from the Swiss National Science Foundation (SNSF Grant no. IZ32Z0\_168366; awarded to CC). The work of AB was supported by the SNSF through Grant No. BSSGI0\_155809 and the University of Lausanne. The work of JV was supported by the SNSF under the Ambizione Grant No. PZ00P1\_154955.

Inference, Vol. 4, eds A. Biedermann, J. Vuille, and F. Taroni (Lausanne: Frontiers Media S.A.), 22–24.

ENFSI Board (2005). Code of Conduct, BRD-GEN-003. ENFSI Board.

23See also the case mentioned in footnote 21.

**Disclaimer:** Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the National Institute of Standards and Technology. Certain commercial equipment, instruments, and materials are identified in order to specify experimental procedures as completely as possible. In no case does such identification imply a recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that any of the materials, instruments, or equipment identified are necessarily the best available for the purpose.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Copyright © 2016 Biedermann, Champod, Jackson, Gill, Taylor, Butler, Morling, Hicks, Vuille and Taroni. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# Commentary: A "Source" of Error: Computer Code, Criminal Defendants, and the Constitution

Duncan A. Taylor 1, 2 \*, Jo-Anne Bright 3 and John Buckleton3

*1 Forensic Science South Australia, Adelaide, SA, Australia, 2 School of Biological Sciences, Flinders University, Adelaide, SA, Australia, 3 Institute of Environmental Science and Research, Auckland, New Zealand*

Keywords: source code, STRmix, DNA profile interpretation, closed source, court challenge

### **A commentary on**

### **A "Source" of Error: Computer Code, Criminal Defendants, and the Constitution** by Chessman, C. (2017). Calif. Law Rev. 105, 101–193.

Chessman (2017) warns of the current trend to admit into court unchallenged the results of complex computerized calculations. He provides a number of examples and arguments claimed to demonstrate the need for open source software to remove the "black box" element. We agree with parts of this sentiment, and the topic of this special issue, that there is a danger with those using and receiving information from black box systems.

Some care however is needed with simple diagnoses and prescriptions such as these.

Modern probabilistic genotyping software are replacing methods previously applied manually. We have great confidence in the forensic community with regard to both integrity and dedication. The previously applied processes are usually a composite of standard operating procedure and human judgment. The difference between these and probabilistic software is largely that the processes in the software are encoded.

Many disciplines are sufficiently broad that practitioners need to rely, in part, on the work of others. This is not new (for a discussion on this point see Taylor, 2016). The risk to which Chessman refers arises when the individual using the system has so little understanding that they do not know how to use the system, or when it has not worked1 . Chessman provides some helpful suggestions for how breaking down black box barriers can be addressed on an individual and systemic scale. As developers of expert system STRmixTM2 (Taylor et al., 2013), we wish to address some of the alarmist points in Chessman (and echoed by others3 ) that gives the impression that producers of expert systems are all either incompetent or corrupt.

We first wish to correct a couple of points in (Chessman, 2017). Regarding the "erroneous assumption" referenced by footnotes 49–51: This miscode, and indeed any miscode found that has been identified in STRmixTM development or use, was identified by examination of the program's output and not the source code. It would be nearly impossible to identify subtle errors in code by viewing the code. The identification has always been a result of comparison of the results produced by a program to some known control4 . The results of these comparisons then trigger the examination of a specific section of the code in order to discover the source of the discrepancy.

3For example see EPIC (https://epic.org/state-policy/foia/dna-software/). 4Commonly a "by-hand" recreation of the expected value(s).

#### Edited by:

*Sue Pope, Principal Forensic Services, UK*

#### Reviewed by:

*Peter Ronald Gunn, University of Technology Sydney, Australia*

> \*Correspondence: *Duncan A. Taylor<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Statistical Genetics and Methodology, a section of the journal Frontiers in Genetics*

> Received: *03 November 2016* Accepted: *28 February 2017* Published: *15 March 2017*

#### Citation:

*Taylor DA, Bright J-A and Buckleton J (2017) Commentary: A "Source" of Error: Computer Code, Criminal Defendants, and the Constitution. Front. Genet. 8:33. doi: 10.3389/fgene.2017.00033*

1Note that this is not an issue with just computer programs, recent history has numerous examples within forensic biology showing that a misunderstanding of the way a system works at a fundamental level can cause issues even when the calculations themselves are relatively simple and able to be done by hand (Budowle and Bieber, 2015). 2An expert system that analyses STR DNA profile data.

Even as developers, during the developmental validation of new versions of STRmixTM, we utilize the extended outputs of the software to validate, and do not validate by examination of code. A further reference (footnote 98) makes the same incorrect assumption that it was code review that lead to the discovery of a programming error. Our experience has been that even more crucial than a review of source code, is the ability to have access to outputs that demonstrate each step of a calculation. We should also note that our ongoing evaluation and testing of the software is a marker of continuous validation and refinement, rather than just fixing "errors" and "blunders."

The second point we wish to make is that the type and magnitude of miscodes are important to consider. The majority of programming errors will lead to instances of a program "crashing" or failing to produce an answer. These types of errors are arguably inconsequential as they will not lead to any erroneous results being produced. More serious are miscodes where no errors are identified or displayed by the software. These can be split into those that will be clearly identifiable5 and those that are more subtle and may go initially unnoticed. Even in this latter category, the question should be asked "What effect does this error have?" If the magnitude of the difference in the result caused by the miscode is small compared with the natural variability in the results being produced6 then arguably the consequences are minimal. We are by no means suggesting that that these types of errors are acceptable, they should be rectified as soon as found. We simply suggest that they tend to be used for scaremongering in a manner disproportionate to their impact. Case in point is the oft quoted article (David Murray, 2015), which contains the never quoted sentence "The DNA likelihood ratios in both the new and original statements appear to be the same."

We agree with the suggestion of Chessman that source code should be available for scrutiny. STRmixTM abides by one of the mechanisms that Chessman suggests, namely the ability for code to be disclosed under confidentiality agreements7 . We note that running of STRmixTM is just the final step in a long journey of computerized activities that ultimately lead to an answer.

## REFERENCES

A true challenge of all steps in the process would require the examination of the source code underlying the Java programming language in which STRmixTM is written, the WindowsTM operating system on which it is run, the software used to process the raw electrophoretic data, the software used to collect the raw electrophoretic data from the electrophoresis instrument, the code used to run the electrophoresis instrument, the PCR thermocycler, the quantification instrument and a myriad of no doubt thousands of blocks of code that sit within the numerous Peripheral Interface Controllers that control hardware components.

With the advent of complex computerized evaluation of evidence, there is a shift from a time where an expert can testify to all aspects of the evaluation, to one where, at some level, the workings of an expert system are accepted without absolute understanding. This may initially seem frightening, but an examination of the bigger picture suggests otherwise. It would be difficult to argue that the use of computerized breathalyzers is a backwards step from the reliability of the Field Sobriety Test. Similarly, virtually all senior advisory bodies relating to DNA profile evaluation recognize the clear benefits of the probabilistic interpretation systems (which by nature of their complexity require computerized implementation) over the preceding manual or binary interpretation methods (Coble et al., 2015; SWGDAM, 2015). In our efforts to ensure that software is not the "source" of errors, it is important to recognize that even with the noted occurrences of these errors, the current computerized solutions, when used by trained experts, represent a vast improvement to the quality and reliability of evidence presented in court.

## AUTHOR CONTRIBUTIONS

All authors contributed to the discussions and writing of the manuscript. Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the author's organizations.

## FUNDING

Funding to write this manuscript was provided by the author's institutions only in the sense of allowing work time to be used to develop the document.

5 Such as value of a probability greater than one, or a negative amount of some substance.

6This may either be in the raw results due to inherent variability in the laboratory process or it may be variability in the statistical result due to an evaluation method that utilizes random number generation (Bright et al., 2015).

7The code of STRmixTM has been viewed under such conditions in the past.

Probabilistic Genotyping Systems [Online]. Available online at: http://media.wix.com/ugd/4344b0\_22776006b67c4a32a5ffc04fe3b56515.pdf [Accessed 3 October 2016].

**Conflict of Interest Statement:** Authors are technical developers of commercial software STRmixTM but do not benefit financially from STRmixTM.

Copyright © 2017 Taylor, Bright and Buckleton. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

# General Commentary: Federal labour Court [2009] – 8 aZr 1012/08

### *Kyriakos N. Kotsoglou\**

*Law Department, Liverpool Hope University, Liverpool, UK*

Keywords: evidence and proof, law of evidence, labour law, forensic sciences, reference class problem

### **A commentary on**

### **Federal labour court [2009] – 8 AZR 1012/08**

For Silke K., it must have been one of these days that define someone's story beyond her life. After years of suffering gender discrimination and mobbing at work, the Berlin-Brandenburg's Labour Court (LAG) had just issued a game-changing decision in her favor. The question before the court essentially boiled down to whether the claimant had discharged her burden of proof simply by employing formal statistical methods. LAG's answer was positive. It awarded damages that included the difference (€1.468) between Silke K.'s monthly salary and a director's one, damages for sex discrimination (€28.214,66), and damages for violating rights of personality (€20.000) (Second Instance Land Labour Court, 2008).

The decision did not only implement the new policy enshrined in the General Equal Treatment Act 2006 (Allgemeines Gleichbehandlungsgesetz)—whose purpose was to put an end to discrimination against vulnerable social groups of workers and vertical segregation in labor market creating steep asymmetries between the percentage of women in higher positions and the total labor force (ILO Director-General, 2011)—but also most importantly constituted a paradigm shift regarding the legal protection of female employees. More specifically, whereas women were making up at that time the majority (69%) of the defendant's workforce (Second Instance Land Labour Court, 2008, para 23), not a single one of them was in the board of directors (Second Instance Land Labour Court, 2008, para 14), exemplifying thus in a clear way that women are "widely underrepresented" in decision-making positions in the private sector of Germany (EU Document of the Directorate General for Internal Policies, 2015). But what exactly propelled this long expected success and engineered the closing of the persistent gender pay gap?1 Silke K.'s team of lawyers had hired a mathematician to calculate the probability that it is not random that the board of directors includes no women. The probability, the statistical analysis (Monte–Carlo simulation) showed, was between 98.7% and 100% (Second Instance Land Labour Court, 2008, para 34). Therefore, it was formal statistical calculations that gave thrust to the gender equality machinery, since the LAG explicitly equated this statistical result with the probability of discrimination against the claimant (Second Instance Land Labour Court, 2008). By employing a rigorous framework to draw inferences from data, courts broke the "glass ceiling", i.e., the "unseen, yet unbreakable barrier that keeps minorities and women from rising to the upper rungs of the corporate ladder, regardless of their qualifications or achievements" (Federal Glass Ceiling Commission, 1995).

Alas all good things come to an end. The Federal Labour Court (BAG) quashed the decision as it held that statistics are not conclusive for the individual case (Federal Labour Court [2009]). Although there was no explicit mention of the reference class problem in the decision, the Federal Court raised once again questions of sufficiency of proof by making clear that proof of unlawful behavior hinges

#### *Edited by:*

*Alex Biedermann, University of Lausanne, Switzerland*

#### *Reviewed by:*

*Marcello Di Bello, Lehman College, USA*

#### *\*Correspondence:*

*Kyriakos N. Kotsoglou<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Sociology*

*Received: 02 March 2017 Accepted: 11 April 2017 Published: 18 May 2017*

#### *Citation:*

*Kotsoglou KN (2017) General Commentary: Federal Labour Court [2009] – 8 AZR 1012/08. Front. Sociol. 2:6. doi: 10.3389/fsoc.2017.00006*

**38**

1According to the EU report (EU Document of the Directorate General for Internal Policies, 2015, p. 15): "In general, Germany is ranked by the European Gender Equality Index (GEI) lower than the EU average; its performance in achieving gender equality is 'mediocre'."

on "statistical data being conclusive for the employer in question", to wit, on *specific evidence* (Federal Labour Court [2009], para 68): proof of membership to a reference group is inconclusive.

This brings us to our main issue, the meaning of "specific evidence" and the renowned reference class problem (Colyvan et al., 2001). Can statistical information—accurate as they may be—motivate action? The question is at its kernel whether an epistemic inference from a relevant population—which serves as a basis for calculating and assigning probabilities—to an individual is valid—given that we only have information about the reference class. Since we deal with the problem of factual generalization and individualization, we rather unwillingly have to raise fundamental questions about the nature of our reasoning processes, in both law and elsewhere. Unsurprisingly, these issues have spawned an extensive debate Allen and Roberts (2007). For very good reasons, since legal adjudication aspires to be rational. However, there is no consensus on what lessons to draw. The debate between the opposing parties has stalled. It would not be exaggerating to say that we have reached the point "where one would like just to emit an inarticulate sound" (Wittgenstein, 1958, § 261).

This short commentary suggests that we should not be so pessimistic. From Aristotle who observes that "it is evidently equally foolish to […] demand from a rhetorician scientific proofs" (Aristotle, The Nicomachean Ethics, Book I, Ch. 3) to modern forensic scientists who are at pains to stress that the idea "of a frequency being attached to an outcome for a single event is ridiculous" (Lucy, 2006, 5), scholars have continuously rejected

## REFERENCES

Allen, R., and Roberts, P. (eds). (2007). The special issue on the reference class problem. *Int. J. Evidence Proof* 11, 243–317.

Aristotle. *The Nicomachean Ethics, Book I: The Good for Man*, Chap. 3.

(bogus) aspirations of generality when it comes to (judicial) decisions. True, statistics enable us to validate knowledge-claims about the world; but at the same time, we resort to quantitative evidence in order to gain an understanding of a population *in its entirety*. This simple expression—"in its entirety"—destroys the riddle. Courts and decision makers do not formulate general rules. They give answers to questions such as "Is the defendant guilty?" to which we do not have scientific answers, not because they are intractable, profound mysteries, but simply because decision-making is not a scientific process yielding a *generally* valid solution. Of course, statistics should inform the evidential basis of decisions and help settle arguments. However, judges do have discretion when they apply the law, so that we have to willynilly reject the idea(l) of a mechanical jurisprudence. Extending statistical ideas and methods to procedural and forensic contexts can broadly be classified as scientism.

The Federal Labour Court made a move in the right direction. It criticized the transgression of the bound of "specific evidence" and reaffirmed the individualistic character of legal adjudication by authoritatively cutting the Gordian knot (reference class problem). The academic community has to deliver *ex post* facto the theoretical framework that (dis-)solves this problem.

## AUTHOR CONTRIBUTIONS

The author confirms being the sole contributor of this work and approved it for publication.

Wittgenstein, L. (1958). *Philosophical Investigations*. Oxford: Basil Blackwell, § 261.

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2017 Kotsoglou. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

# Commentary: Van Beelen [2016] SASCFC 71

*David R. A. Caruso\* and Brigid Symes*

*Litigation Law Unit, The University of Adelaide, Adelaide, SA, Australia*

Keywords: second appeal, scientific advance, fresh evidence, compelling evidence, substantial

**A commentary on**

**Van Beelen [2016] SASCFC 71**

## INTRODUCTION

On 15 July 1971, Deborah Leach was found dead on the beach. Dr. Manock, the State forensic pathologist, used stomach content analysis to determine the time of Leach's death. Manock concluded that death was between 4:00 p.m. and 4:30 p.m. Only Mr. Van Beelen had been witnessed on the beach during this time. Van Beelen was convicted of Leach's murder in 1973.1 His conviction was upheld on appeal.2

In 2013, South Australian appellate law was amended to permit second and subsequent appeals if there was "fresh" and "compelling" evidence.3 This reversed the common law position that allowed for only a single, perfected appeal.4 Van Beelen's case was subsequently appealed in 2016.5 The defense argued that scientific development concerning stomach content analysis constituted fresh and compelling evidence. The Appeal Court rejected that argument by majority.

### *Edited by:*

*Alex Biedermann, University of Lausanne, Switzerland*

#### *Reviewed by:*

*Paul Roberts, University of Nottingham, United Kingdom*

#### *\*Correspondence:*

*David R. A. Caruso<EMAIL_ADDRESS>
#### *Specialty section:*

*This article was submitted to ELSI in Science and Genetics, a section of the journal Frontiers in Sociology*

*Received: 20 October 2017 Accepted: 28 November 2017 Published: 20 December 2017*

#### *Citation:*

*Caruso DRA and Symes B (2017) Commentary: Van Beelen [2016] SASCFC 71. Front. Sociol. 2:20. doi: 10.3389/fsoc.2017.00020*

## STOMACH CONTENT ANALYSIS AT THE 1973 TRIAL

Manock relied on Leach's schoolmates for the content and timing of her last meal: a pasty, a glass of milk, and a slice of pie between 12:15 p.m. and 12:30 p.m. Manock found that Leach's stomach was three-quarters empty at death. He concluded that death could not have occurred before 3:30 p.m. or after 4:30 p.m. Dr. Pocock testified, for the defense, that it would "indeed be a rash, irresponsible man, who would dare pronounce the exact time of death in the witness-box; or for that matter, be ready to estimate the time even to within an hour on either side of the actual time."6

## SCIENTIFIC DEVELOPMENTS IN THE 2016 APPEAL

In 2016, Professor Horowitz testified that stomach content analysis was now considered to be an unreliable way to accurately measure the time of death. This was due to the enormous variations in digestion between individuals. Horowitz testified that gastric emptying rates cannot (now) be estimated to within an hour.7 He said that even in the 1970s, it was completely unreliable science to provide an estimate of time of death once gastric emptying had commenced.8

1 See R v Van Beelen (1973) 4 SASR 353.

2R v Van Beelen (No 3) (1973) 7 SASR 125.

3 Statutes Amendment (Appeals) Act 2013 (SA) (No 9 of 2013) section 7.

4R v Edwards (No 2) (1931) SASR 376; Grierson v The King (1938) 60 CLR 431.

5R v Van Beelen (2016) 125 SASR 253.

6R v Van Beelen (2016) SASCFC 71 (129) Vanstone and Kelly JJ.

## COURT'S REASONING

Kourakis CJ, Vanstone and Kelly JJ comprised the secondary appeal court. They agreed that the evidence of Horowitz was "fresh" within the meaning of the legislation. The point of difference was that the Majority (Vanstone and Kelly JJ) did not consider the evidence "compelling"; the Chief Justice did.

"Fresh" and "compelling" are defined. Evidence relating to an offense is "fresh" if: (i) *it was not adduced at the trial of the offense*; and (ii) *it could not, even with the exercise of reasonable diligence, have been adduced at the trial*. That same evidence is "compelling" if: (i) *it is reliable*; *and* (ii) *it is substantial*; *and* (iii) *it is highly probative in the context of the issues in dispute at the trial of the offense*. 9

The Majority held that the evidence was not "compelling" because Horowitz's evidence was not "substantial."10 This was due to the doubt already placed on Manock's evidence by Pocock in the original trial.11 The Majority did not consider the prosecution case turned on the expert evidence regarding time of death. The Majority gave particular attention to the civilian evidence at the 1973 trial that set the parameters for Leach's death as occurring between 4:00 p.m. and 4:40 p.m., or at the latest 4:50 p.m. The fresh evidence was only capable of showing that time of death could have been a mere 10 or 20 min later. The Majority considered that, as a result, the evidence of Horowitz was not substantial and, therefore, not compelling.12

## OPINION

Common law courts vested with secondary appeal powers are obliged to review scientific advance within the confines of its possible meaning and interpretation in an historic trial. Appellate rights must be constrained by the trial issues to realize the utility in litigation ending. That is a narrow lens for what may be wideranging gains in scientific knowledge. The purpose of secondary appellate legislation should be to permit scientific advancement to expose errors of fact-finding at trial. The conclusion of the Majority requires the relevant evidence to be substantial in light of the trial as a whole. The provisions of section 353A, however, do not require the evidence to be considered in light of the trial holistically. The fresh evidence itself must be substantial. Kourakis CJ identified why the evidence was substantial: its reputable source and basis in current science.

Compelling evidence under the legislative regime is "highly probative in the context of the issues" at the trial. Scientific advance is almost ubiquitously probative with respect to evolution of human knowledge, but the courts are concerned with a more restricted notion of probity. The "probative value" of expert evidence to the law concerns the effect that evidence would have on the rational assessment of issues before the court. In the case of secondary appeals, the probative value lies in the extent to which the expert evidence would compel re-assessment of the original trial issues. This symbiotic relationship between the probative value of expert evidence and the disputed issues reveals why the reasoning of Kourakis CJ is to be preferred.

Probative value is to be assessed in the context of the issues, not the evidence, in dispute. The Majority rejected Horowitz's evidence as compelling because there was other lay evidence, which diminished the import of stomach contents emptying regarding time of death. The legislation does not invite the appellate court to examine the issues in dispute having regard to the evidence in the trial court. The requirement that evidence justifying a secondary appeal must be "fresh" guards against evidence being re-litigated or agitated on secondary appeal. If evidence adduced on secondary appeal relates to an issue in the trial below which the context of the trial reveals to be an important issue in the dispute at trial, then, the definition of "compelling" indicates that the subject evidence should be regarded as highly probative *in the context of the issues in dispute*. The time of death was a key element in the prosecution case and, without its certainty, a possibility arose that someone other than Van Beelen committed the offense.13 The Chief Justice found the relationship of Horowitz's evidence to the time of death was highly probative as the timing of death was an issue central to the context of the dispute at the 1973 trial.

In a trial where time of death is in issue and the context of the trial places emphasis on the timing of death, fresh evidence concerning errors in that timing should satisfy the criterion of "compelling." Secondary appellate legislation should be, unless a contrary parliamentary intention can be clearly shown, read with a view to assess whether the fresh evidence is compelling given the *issues* in the court below, not the *evidence* in the court below.

Van Beelen appealed the split decision to the High Court of Australia and the matter was heard in June 201714; final judgment is pending.

## AUTHOR CONTRIBUTIONS

This commentary was jointly written with a 80/20 division of work in reviewing the subject case.

*The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

9Criminal Law Consolidation Act 1935 (SA), section 353A (6).

10R v Van Beelen (2016) SASCFC 71 (159) Vanstone and Kelly JJ.

11R v Van Beelen (2016) SASCFC 71 (162) Vanstone and Kelly JJ.

12R v Van Beelen (2016) SASCFC 71 (164) Vanstone and Kelly JJ.

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Copyright © 2017 Caruso and Symes. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY).*

13R v Van Beelen (2016) SASCFC 71 (72) Kourakis CJ.

14R v Van Beelen (2017) HCATrans 19.

# Bayesian Hierarchical Random Effects Models in Forensic Science

Colin G. G. Aitken\*

*School of Mathematics and Maxwell Institute, The University of Edinburgh, Edinburgh, United Kingdom*

Statistical modeling of the evaluation of evidence with the use of the likelihood ratio has a long history. It dates from the Dreyfus case at the end of the nineteenth century through the work at Bletchley Park in the Second World War to the present day. The development received a significant boost in 1977 with a seminal work by Dennis Lindley which introduced a Bayesian hierarchical random effects model for the evaluation of evidence with an example of refractive index measurements on fragments of glass. Many models have been developed since then. The methods have now been sufficiently well-developed and have become so widespread that it is timely to try and provide a software package to assist in their implementation. With that in mind, a project (SAILR: Software for the Analysis and Implementation of Likelihood Ratios ) was funded by the European Network of Forensic Science Institutes through their Monopoly programme to develop a software package for use by forensic scientists world-wide that would assist in the statistical analysis and implementation of the approach based on likelihood ratios. It is the purpose of this document to provide a short review of a small part of this history. The review also provides a background, or landscape, for the development of some of the models within the SAILR package and references to SAILR as made as appropriate.

Keywords: Bayes' Theorem, evidence evaluation, forensic science, hierarchical models, likelihood ratios, random effects, SAILR, statistics

## 1. INTRODUCTION

Statistical analyses for the evaluation of evidence have a considerable history. It is the purpose of this document to provide a short review of a small part of this history. It brings together ideas from the last forty years for statistical models when the evidence is in the form of measurements and thus of continuous data. The data are also hierarchical with two levels. The first level is that of source, the origin of the data. The second level is of items within a source. The models used to represent the variability in the data are random effects models. The models are chosen from analyses of samples of sources from some relevant population. Finally, the analysis is Bayesian in nature with prior distributions for the parameters of the within-source distributions. The nature of the prior distributions is informed from training data based on the samples from the relevant population.

The remainder of the document is structured as follows. Section 2 provides a general introduction to the likelihood ratio as a measure of the value of evidence. Section 3 provides a framework for models for comparison and discrimination. Section 4 discusses the assessment of model performance. An Appendix gives formulae for some of the more commonly used models.

### Edited by:

*Sue Pope, Principal Forensic Services, United Kingdom*

### Reviewed by:

*Robert Brian O'Hara, Norwegian University of Science and Technology, Norway Ricardo De Matos Simoes, Dana-Farber Cancer Institute, United States*

\*Correspondence:

*Colin G. G. Aitken<EMAIL_ADDRESS>
#### Specialty section:

*This article was submitted to Statistical Genetics and Methodology, a section of the journal Frontiers in Genetics*

> Received: *27 December 2017* Accepted: *29 March 2018* Published: *16 April 2018*

#### Citation:

*Aitken CGG (2018) Bayesian Hierarchical Random Effects Models in Forensic Science. Front. Genet. 9:126. doi: 10.3389/fgene.2018.00126*

**42**

# 2. THE VALUE OF EVIDENCE

Part of the role of a forensic scientist is to interpret evidence found at a crime scene in order to aid fact-finders in a criminal case (e.g., the judge or jury) in their decision making. The forensic scientist may be asked to comment on the value of the evidence in the context of various competing statements about the evidence, each of which may be true or false.Generally, a forensic scientist must consider two competing statements relating to the evidence, one put forward by the prosecution in a criminal case, and one put forward by the defense (Cook et al., 1998b). These statements are known as propositions1 . They generally come in pairs that are mutually exclusive, though not necessarily exhaustive. For a debate about the requirement, or otherwise, for the propositions to be exhaustive (see Biedermann et al., 2014; Fenton et al., 2014a,b).

One member of the pair is associated with the prosecution and conventionally denoted Hp. The other member of the pair is associated with the defense and conventionally denoted Hd. The evidence to be evaluated is denoted E 2 . The value of evidence is taken to be the relative values of the probability of the evidence if a proposition put forward by the prosecution is true and the probability of the evidence if a proposition put forward by the defense is true. However, evidence is not evaluated in isolation. There is always other information to be taken into account, including, for example, personal knowledge of the fact-finder. Denote this information by I. The value of the evidence, denoted V say, can then be written formulaically as

$$\mathcal{V} = \frac{\Pr(E \mid H\_p, I)}{\Pr(E \mid H\_d, I)},$$

where Pr denotes Probability. This ratio is known as the likelihood ratio.

The likelihood ratio is the method used by SAILR to evaluate evidence. SAILR (Software for the Analysis and Implementation of Likelihood Ratios) is a user-friendly Graphical Interface (GUI) to calculate numerical likelihood ratios in forensic statistics and its development under the direction of the Netherlands Forensic Institute (NFI) was funded by the European Network of Forensic Science Institutes through their Monopoly programme. The likelihood ratio is a generally accepted measure for the value of evidence in much forensic case-work.

This representation of the value of evidence has a very good intuitive interpretation. Consider the odds form of Bayes' Theorem in the forensic context of the evaluation of evidence. The odds form of Bayes' Theorem then enables the prior odds(i.e., prior to the presentation of E) in favor of the prosecution proposition Hp relative to the defense proposition Hd to be updated to posterior odds given E, the evidence under consideration. This is done by multiplying the prior odds by the likelihood ratio. The odds form of Bayes' Theorem may then be written as

$$\Pr^{\Pr(H\_{\mathcal{P}} \mid E, I)} = \frac{\Pr(E \mid H\_{\mathcal{P}}, I)}{\Pr(E \mid H\_d, I)} \times \frac{\Pr(H\_{\mathcal{P}} \mid I)}{\Pr(H\_d \mid I)}.\tag{1}$$

The likelihood ratio (LR) is the ratio

$$\frac{\Pr(H\_{\mathcal{P}} \mid E, I) / \Pr(H\_d \mid E, I)}{\Pr(H\_{\mathcal{P}} \mid I) / \Pr(H\_d \mid I)}\tag{2}$$

of posterior odds to prior odds. It is the factor which converts the prior odds in favor of the prosecution proposition to the posterior odds in favor of the prosecution proposition. The representation in Equation (1) also emphasizes the dependence of the prior odds on other information I. Values of the LR > 1 are supportive of Hp, the proposition put forward by the prosecution. Values of the LR < 1 are supportive of Hd, the proposition put forward by the defense. The word "odds" should be used advisedly. If Hp and Hd are not exhaustive then the component probabilities Pr(Hp | E,I) and Pr(Hd | E,I) cannot be derived from this ratio. All that can be said is that the posterior ratio is different from the prior ratio by a factor V.

An advantage of this formulation of evidence evaluation is the ease with which the effect of the addition of new evidence can be determined. The posterior odds for one piece of evidence, E1 say, can be the prior odds for a second piece of evidence, E2 say. Then Equation (1) may be rewritten as

$$\frac{\Pr(H\_{\mathcal{P}} \mid E\_1, E\_2, I)}{\Pr(H\_d \mid E\_1, E\_2, I)} = \frac{\Pr(E\_2 \mid H\_{\mathcal{P}}, E\_1, I)}{\Pr(E\_2 \mid H\_d, E\_1, I)} \times \frac{\Pr(H\_{\mathcal{P}} \mid E\_1, I)}{\Pr(H\_d \mid E\_1, I)},\tag{3}$$

where the conditioning of the evaluation of E2 on E1 is made explicit.

An illustration of the effect of evidence with a value V of 1,000 on the odds in favor of Hp relative to Hd is given in **Table 1**.

The following quote is very pertinent.

'That approach does not ask the jurors to produce any number, let alone one that can qualify as a probability. It merely shows them

TABLE 1 | Effect on prior odds in favor of *Hp* relative to *Hd* of evidence *E* with value *V* of 1,000.

*Reference to background information I is omitted.*

1Other writers use the term hypothesis (see section 2.7). The term proposition will he used except when there is an explicit need for the term hypothesis; see, for example, section 3.1

2 In ENFSI guidelines ENFSI (2015) "findings" are distinguished from "evidence." "Findings are the result of observations, measurements and classification that are made on items of interest." "[E]vidence refers to outcomes of forensic examinations (findings) that, at a later point, may be used by legal decision-makers in a court of law to reach a reasoned belief about a proposition." However, the word "evidence" will be used in this document to refer to both situations for ease of nomenclature.

how a "true" prior probability would be altered, if one were in fact available. It thus supplies the jurors with as precise and accurate an illustration of the probative force of the quantitative data as the mathematical theory of probability can provide. Such a chart, it can be maintained, should have pedagogical value for the juror who evaluates the entire package of evidence solely by intuitive methods, and who does not himself attempt to assign a probability to the "soft" evidence.' Kaye (1979).

The "it" in this context is a chart depicting, in numerical terms, how much the prior odds in favor of a proposition is enhanced by the evidence being evaluated. This is a graphical equivalent of **Table 1**. The mathematical tool for devising such a chart is Bayes' Theorem. These remarks of Kaye's refer to characteristics of the general method for the evaluation of evidence that is the likelihood ratio. They do not refer to a particular case. For example, it is not possible to comment on the accuracy of a likelihood ratio estimation in a particular case because the true value of the likelihood ratio is not known nor can it be known. It is, however, possible to refer to the accuracy of a method and performance assessment in general is discussed in section 4.

The use of a likelihood ratio for the evaluation of evidence is not a new idea. In the Dreyfus case (Champod et al., 1999), it was argued that

. . . since it is absolutely impossible for us [the experts] to know the a priori probability, we cannot say: this coincidence proves that the ratio of the forgery's probability to the inverse probability is a real value. We can only say: following the observation of this coincidence, this ratio becomes X times greater than before the observation (Darboux et al., 1908).

The "ratio" in this quotation is the odds in favor of one proposition over another, The X refers to the likelihood ratio. The posterior odds in favor of the proposition is then X times the prior odds.

The ideas were also used in the work of I.J. Good and A.M. Turing as code-breakers at Bletchley Park during World War II (Good, 1979).

## 2.1. Background Information

The likelihood ratio updates the prior odds, those before consideration of evidence E, to posterior odds, which take E into account. The posterior odds are the odds with which, ultimately, the fact-finder is concerned. If the likelihood ratio multiplied by the prior odds is larger than one, then the probability of Hp given the evidence is larger than that of Hd given the evidence. As these propositions may not be exhaustive their explicit values, rather than their relative value, may not be known. It is the responsibility of the fact-finder to determine a value for the prior odds. The prior odds can then be combined with the likelihood ratio to obtain posterior odds. A forensic scientist is concerned only with the value of the evidence as expressed by the likelihood ratio so cannot usually comment on the value of the posterior odds. The likelihood ratio is considered as the strength of support of the evidence for one of the two propositions Hp or Hd.

The application of this form to a specific case is crucially dependent on the background information I. However, the background information available to each person is different. In part, this is because each person is different. In part it is because of professional differences. The information that a forensic scientist should use for their determination of the likelihood ratio is different from that which a fact-finder, such as judge or jury member, should use for their determination of the odds in favor of the prosecution proposition. There are differences in the background information available to these participants in the judicial process but these differences have no effect on the posterior odds in favor of the prosecution proposition.

Let I = Ia∪Ib where Ia is background information available to the forensic scientist and Ib is background information available to the fact-finder. There will be information available to both, the intersection Ia ∩ Ib is not empty. It can then be shown (Aitken and Nordgaard, 2017) that the posterior odds may be written in the form

$$\frac{\Pr(H\_{\mathcal{P}} \mid E, I)}{\Pr(H\_d \mid E, I)} = \frac{\Pr(E \mid H\_{\mathcal{P}}, I\_b)}{\Pr(E \mid H\_d, I\_b)} \times \frac{\Pr(H\_{\mathcal{P}} \mid I\_a)}{\Pr(H\_d \mid I\_a)}.$$

The fact-finder and the forensic scientist have to treat the common information (Ia ∩ Ib ) with appropriate discretion.

## 2.2. Uniqueness of the Likelihood Ratio

The role of the likelihood ratio as the factor that updates the prior odds to the posterior odds has a very intuitive interpretation. There is also a mathematical derivation that shows it, or a function of it such as the logarithm, is the only way to update evidence. It was shown many years ago by I.J.Good in two brief notes in the Journal of Statistical Computation and Simulation (Good, 1989a,b) repeated in Good (1991) and in Aitken and Taroni (2004) that, with some very reasonable assumptions, the assessment of uncertainty inherent in the evaluation of evidence leads inevitably to the likelihood ratio as the only way in which this can be done.

Consider evidence E which it is desired to evaluate in the context of two mutually exclusive propositions Hp and Hd. Denote the value of the evidence by V. As always, the value will depend on background information I but this will not be stated explicitly. There are other assumptions implicit in this approach, namely that there is a probability that can be associated with evidence and one that is dependent on propositions and only on propositions (and background information). Another assumption is that V is a function only of the probability of E, given Hp to be true, and of the probability of E, given Hd to be true.

Let x = Pr(E | Hp) and y = Pr(E | Hd) where I is omitted for ease of notation. The assumption that V is a function only of these probabilities can be represented mathematically as

$$V = f(x, y)$$

for some function f .

Now, consider another piece of evidence T which is irrelevant to E, to Hp and to Hd. Irrelevance is taken in the probabilistic context to be equivalent to independence so that T may be taken to be independent of E, of Hp and of Hd. It is then permissible for Pr(T) to be given notation which does not refer to any of E, Hp or Hd. Thus, let Pr(T) be denoted by θ. Then

Alternatively, the evidence may be a single measurement such as that of a DNA profile.

$$\begin{array}{rcl} \Pr(E, T \mid H\_{\mathcal{P}}) &=& \Pr(E \mid H\_{\mathcal{P}}) \Pr(T \mid H\_{\mathcal{P}}) \text{ by the independence of } E \text{ and } T \\ &=& \Pr(E \mid H\_{\mathcal{P}}) \Pr(T) \\ &=& \varkappa \theta. \end{array}$$

Similarly,

$$\Pr(E, T \mid H\_d) = \mathcal{Y}\,\theta.$$

The value of (E, T) is f(θx, θy) by the definition of f . However, evidence T is irrelevant and has no effect on the value of evidence E. Thus, the value of the combined evidence (E, T), f(θx, θy), is equal to the value V of E, f(x, y), and

$$V = f(\mathbf{x}, \boldsymbol{\jmath}) = f(\boldsymbol{\theta} \mathbf{x}, \boldsymbol{\theta} \boldsymbol{\jmath})$$

for all θ in the interval [0,1] of possible values of Pr(T).

The only class of functions of (x, y) for which this can be said to be the case is the class which are functions of x/y or

$$\Pr(E \mid H\_{\mathcal{P}}) / \Pr(E \mid H\_d)$$

which is the likelihood ratio. Hence the value V of evidence has to be a function of the likelihood ratio. It has been argued (Lund and Iyer, 2017) that the forensic community view the likelihood ratio as only one possible tool for communication with decision makers. The argument of Good shows that it is the only logically admissible form of evaluation.

## 2.3. Weight of Evidence

An interesting note of terminology can be mentioned here. It is common in some legal circles to talk of the weight of evidence. The concept of weight of evidence is an old idea. The term weight of evidence should be used for the logarithm of the likelihood ratio. The terminology was first used by Peirce (1878). The likelihood ratio is the value of the evidence and its logarithm is the weight of the evidence. The logarithm of the likelihood ratio has the pleasingly intuitive operation of additivity when converting the logarithm of the prior odds in favor of a proposition to the logarithm of the posterior odds in favor of the proposition.

$$\log\left\{\frac{\Pr(H\_{\mathcal{P}} \mid E)}{\Pr(H\_{d} \mid E)}\right\} = \log\left\{\frac{\Pr(E \mid H\_{\mathcal{P}})}{\Pr(E \mid H\_{d})}\right\} + \log\left\{\frac{\Pr(H\_{\mathcal{P}})}{\Pr(H\_{d})}\right\},\tag{4}$$

with I omitted. When considering the scales of justice it is the logarithm of the probabilities of the evidence given each of the two competing propositions that should be put in the scales, not the probabilities.

## 2.4. Terminology for Evidence

The evidence under consideration in this document and within the SAILR project is evidence that could have been transferred either from the crime scene to the criminal or from the criminal to the crime scene. Evidence that could have been so transferred is in the form of traces. Thus it has two names transfer or trace evidence. The evidential material discussed here is in the form of individual items. Thus, there may be a finite number of items, such as tablets or sachets of drugs or fragments of glass.

Consider the situation in which a crime has been committed, there is a crime scene and the investigation has reached the stage where a suspect has been identified. Trace evidence, denoted E, of a particular type has been found at the crime scene and on the suspect and its value is of interest. The evidence E may be partitioned into two parts, that found at the crime scene and that found in association with the suspect. In practice, the terminology takes a different form which depends on whether the source of the evidence is known or not known. A distinction is also drawn between evidential material and the evidence for evaluation. Evidence for evaluation is the observations made on the material. Only evidence which is in the form of measurements and thus represented by continuous data is considered here. Other factors such as the locations in which the material was found and the quantity of the material are not considered. Evidence of a discrete nature such as binary data as in the presence or absence of striation marks is also not considered.

Evidence whose source is known is called control evidence Ec . Evidence whose source is not known is called recovered evidence Er . Measurements on Ec are conventionally denoted **x** where **x** = (x1, . . . , xm) are m sets of measurements and where xi , i = 1, . . . , m may be univariate or multivariate. Measurements on Er are conventionally denoted **y** where **y** = (y1, . . . , yn) are n sets of measurements and where yj , j = 1, . . . , n may be univariate or multivariate 3 .

For an evaluative comparison of **x** and **y**, background data **z** are needed. These background data should be a representative sample of all possible sources from the population of interest, known as the relevant population. Ideally, the sample should be a random sample but this is rarely possible for practical reasons. The sample is often what might be called a convenience sample. If the convenience sample can be demonstrated to be composed of sources chosen in a manner independent of the case under investigation then the inference based on the comparison of **x** and **y** informed on **z** should be valid. Computation of the likelihood ratio requires data files from **x**, **y** and **z**.

One example of evidence in the form of multivariate data relates to glass elemental content. Such data are often subjected to a logarithmic transformation after taking the ratios of a particular elemental content to the oxygen content, for example, log10(NaO) = log10(Na/O). These measurements can be for each of m fragments of control evidence and for each of n fragments of recovered evidence (Zadora et al., 2014). This evidence can be multivariate as there can be several ratios measured for each fragment, e.g., log10(NaO), log10(MgO) and log10(AlO). The control evidence is the measurements from a number m of fragments of glass from a broken window at a crime scene; the source of the fragments is known to be the window, items within source are the fragments. The recovered

3The use of **x** and **y** here is not to be confused with the use of x = Pr(E | Hp) and y = Pr(E | Hd) in section 2.2.

evidence is the measurements from a number n of fragments of glass found in association with a suspect, for example on clothing identified as theirs. The source of the fragments of glass from the suspect is unknown. It may or may not have come from the window at the crime scene. A second example could be the measurements of color chromaticity coordinates on fibers and the evidence is bivariate (Martyna et al., 2013). There are three color chromaticity coordinates. The sum of their values is fixed so given the values of any two, the third is known. Control evidence is the measurements of color chromaticity coordinates from a number m of fibers from an article of clothing belonging to a suspect; the source is the article, the items are the fibers. Recovered evidence is the measurements of color chromaticity coordinates from a number n of fibers found at a crime scene. Thus control evidence may be found at a crime scene or in association with a suspect. Similarly, recovered evidence may be found at a crime scene or in association with a suspect.

Often the number m of control items can be chosen by the investigator. The number n of recovered items may be determined by what is available and the investigator has little choice in the selection of this number. If the number of recovered items is large, in some sense, and perhaps so large as for it to be impractical to count or analyse them, then the investigator may decide to select n items where n is less than the number available. Procedures for the choice of n and the manner of selection of the items are not discussed in this document or SAILR other than to note that the evidence selected should be representative of the total evidence available as far as is possible. Further information is available in Aitken and Taroni (2004) and references therein.

The likelihood ratio V for the comparison of {**x**, **y**} where E is replaced by {**x**, **y**} is then

$$\mathcal{V} = \frac{\Pr(\mathbf{x}, \mathbf{y} \mid H\_{\mathcal{P}})}{\Pr(\mathbf{x}, \mathbf{y} \mid H\_d)},\tag{5}$$

where again the conditioning on I, the background information, has been omitted for clarity of notation.

Often, the propositions being considered are Hp that the control and recovered evidence are from the same source and Hd that the control and recovered evidence are from different sources. In such a circumstance, **x** and **y** may be assumed independent if Hd is true as they come from different sources. Then Equation (5) may be written as

$$\mathbf{V} = \frac{\Pr(\mathbf{x}, \mathbf{y} \mid H\_{\mathcal{P}})}{\Pr(\mathbf{x} \mid H\_d)\Pr(\mathbf{y} \mid H\_d)}. \tag{6}$$

If **x** and **y** are continuous data, as is the case when the evidence is in the form of measurements rather than counts, the probabilities in the numerator and denominator are replaced by probability density functions, denoted say f(**x**, **y**) for the joint density and f(**x**) and f(**y**) for the marginal distributions. The continuous analog of Equation (6) can then be written as

$$\mathbf{V} = \frac{f(\mathbf{x}, \mathbf{y} \mid H\_p)}{f(\mathbf{x} \mid H\_d) f(\mathbf{y} \mid H\_d)}. \tag{7}$$

In most cases, the full specification of the probability density function is unknown. The form of the distribution may be known or a reasonable assumption of its form may be made. For example, it may be known or can be assumed that the appropriate distribution is a Normal distribution. This assumption may be based on the unimodal, symmetric nature of the distribution. If the distribution has a positive skew then a transformation to normality with a logarithmic transformation of the data may be possible before consideration of the likelihood ratio. However, the parameters may neither be known nor able to be assumed known.

The numerator of Equation (7) may be written as f(**x**, **y** | Hp) = f(**y** | **x**) | Hp)f(**x** | Hp). Since the distribution of **x** is independent of whether Hp or Hd is true, f(**x** | Hp) = f(**x** | Hd) and Equation (7) may be written as

$$f(\mathbf{y} \mid \mathbf{x}, H\_{\mathcal{P}}) / f(\mathbf{y} \mid H\_d).$$

See Equation (18) in Appendix for an example.

## 2.5. Training Data

When parameters are not known, information about their possible values may be obtained from data independent of the crime but thought to be relevant for consideration of the variability in the measurements of the data comprising the evidence. These data are the training data or background data and are conventionally denoted **z**. These data are considered to be a sample from a population, known as a relevant population. There is considerable continuing debate as to how to choose a population that is relevant for a particular crime and, once chosen, how a sample may be chosen from it to be a representative sample of the population. See, for example, R. v. T [2010] EWCA 2439, where the debate related to the choice of populations of shoes relevant for the consideration of evidence of shoeprints. Often the sample is a convenience sample; see section 2.4.

An alternative procedure would be to sample anew each time from a population deemed relevant to the case under investigation. A relatively early example of this is the investigation of a murder in Biggar, a town near Edinburgh, in 1967. A bite mark found on the breast of a young girl who had been murdered had certain characteristic marks, indicative of the conformation of the teeth of the person who had bitten her. A 17-year-old boy was found with this conformation and he became a suspect. Examination of 90 other boys of the suspect's age showed that the particular conformation was not at all common. The 90 other boys could be considered as a sample from a relevant population. Further details are available in Harvey et al. (1968). However, in most individual investigations it is not practical to obtain such a bespoke relevant population.

## 2.6. Hierarchy of Evidence

Often, with measurements, the training data can be thought of as a set of sources of items. Measurements are made of one or more characteristics of the items. For example, consider again the composition of the elemental ratio of various elements of glass to oxygen for glass fragments from a set of windows. The items are glass fragments. A source would be a window. The training set is a set of windows. The set of windows is a sample from some population of windows, deemed relevant for crimes involving windows. The measurements are said to be hierarchical with two levels. One level is the fragment of glass within a window. Variation amongst measurements of fragments within a window is known as within-group or within-source variation. The second level is the window. Variation amongst measurements between windows is known as between-group or between-source variation. Measurements are taken from an item (fragments of glass) within a source (window). Notationally, the training data **z** has two indices, one for each level and may be represented as **z** = {zkℓ ; k = 1, . . . , g, ℓ = 1, . . . , h} where g is the number of sources in the training set and h is the number of measurements within sources. The number of measurements within sources need not necessarily be constant though it is computationally convenient if this can be arranged during the compilation of the training set. Occasionally there may be further levels, for example measurement error.

## 2.7. Propositions

As well as evidence (E) and background information I, evidence evaluation depends on propositions Hp and Hd. There are different types of propositions, also known as levels. Both propositions (Hp and Hd) in any particular situation for the evaluation of evidence are at the same level. There are four different levels of propositions, known respectively as offense level, activity level, source level and sub-source level (Cook et al., 1998a; Evett et al., 2000).

The levels, with examples, are described as follows.

the defendant. The quantity of material found is insufficient to identify its source, e.g., whether it came from blood or semen.

## 3. FRAMEWORK FOR MODELS

The likelihood ratio may be used in the context of forensic science in two different ways, that of comparison and that of discrimination. For comparison, two pieces of evidence found in different places are compared to see if they had a common source. For discrimination, one piece of evidence is compared with several sets of training or background data from different sources to see from which source the evidence may have come.

Most of the models described here are so-called feature-based models. These are models developed from the measurements (features) on the evidential material. Other models described are so-called score-based models. There may be occasions with multivariate data when a feature-based model is not tractable, e.g., multidimensional binary data where the number of possible models is unmanageable. On such occasions, the distance, denoted d(**x**, **y**), between control (**x**) and recovered (**y**) data can be used instead.

## 3.1. Comparison for Feature-Based Models 3.1.1. The Likelihood Ratio Approach for Continuous Univariate Evidential Data With Normal Distributions for the Means and Known Variances

A common problem occurs in forensic science when the prosecution and defense propositions concern whether two objects are from the same source or from different sources. For example, if a glass fragment is found on a suspect and there is a broken window at the crime scene, one proposition might be that the glass fragment found on the suspect came from the window at the crime scene, and the other proposition might be that the glass fragment came from some other window. The evidence is given by a set of measurements from the glass fragment found on the suspect (the recovered sample) and a set of measurements from one or more glass fragments from the crime scene (the control sample). The problem is one of comparison.

The structure of these models reflects the hierarchical nature of the underlying data (measurements and variation within a source and then variation between sources). Using a distribution for the means θ1 and θ2 in this way accounts for variance within source (σ 2 ) and variance between sources (τ 2 ).

The problem for the fact-finder is to determine which of the two propositions (Hp or Hd) is more likely, given all of the evidence in the case. Denote the other evidence and background information by I as before. The fact-finder can consider which proposition is more likely by considering the relative size of the two probabilities Pr(Hp | ¯x, y¯,I) and Pr(Hd | ¯x, y¯,I) (technically, in cases where the statistical assumptions include knowledge of the variances σ 2 and τ 2 and of a Normal distribution for the measurements, the means of the control and recovered samples are sufficient statistics so can be used in place of the measurements **x** and **y**). Let f(x¯, y¯ | Hp,I) be the joint probability density function of x¯ and y¯, given proposition Hp and I and let f(x¯, y¯ | Hd,I) be the joint probability density function of x¯ and y¯ given proposition Hd and I. In this context Equation (1) may be represented as

$$\frac{P(H\_{\mathcal{P}} \mid \bar{\mathbf{x}}, \bar{\mathbf{y}}, I)}{P(H\_{d} \mid \bar{\mathbf{x}}, \bar{\mathbf{y}}, I)} = \frac{f(\bar{\mathbf{x}}, \bar{\mathbf{y}} \mid H\_{\mathcal{P}}, I)}{f(\bar{\mathbf{x}}, \bar{\mathbf{y}} \mid H\_{d}, I)} \times \frac{P(H\_{\mathcal{P}} \mid I)}{P(H\_{d} \mid I)},\tag{8}$$

where E is replaced by (x¯, y¯). For examples where the withinsource variance is not known, the sample variances of **x** and **y** will also be included in the representation.

Denote the common mean of the measurements under the prosecution proposition by θ1 = θ2 = θ. The likelihood ratio V is given by Equation (7). This may be rewritten as

$$\mathbf{V} = \frac{\int f(\bar{\mathbf{x}} \mid \theta) f(\bar{\mathbf{y}} \mid \theta) f(\theta) \mathbf{d}\theta}{\int f(\bar{\mathbf{x}} \mid \theta\_1) f(\theta\_1) \mathbf{d}\theta\_1 \int f(\bar{\mathbf{y}} \mid \theta\_2) f(\theta\_2) \mathbf{d}\theta\_2},\tag{9}$$

where the dependence on I has been suppressed for ease of notation. The analytical form of this likelihood ratio, given the independence and Normality assumptions detailed above, is given by Lindley (1977). The density functions f(x¯ | θ) and f(y¯ | θ) are taken to be density functions of a Normal distribution. Note that when the prosecution proposition is chosen the random variables X¯ and Y¯ , of which x¯ and y¯ are realizations, are conditionally independent, conditional on θ. They are independent if it is known they are from the same source. The distributions associated with these density functions are termed the within-source distributions, because they account for the within-source variability. The distribution associated with the density function f(θ) is termed the between-source distribution because it accounts for between-source variability, and it is a prior distribution for θ. The use of a between-source distribution allows the rarity of the data **x** and **y** to be taken into account when assessing the strength of the evidence; see Equation (13) in the Appendix for an example. Information to assist with the estimation of the prior distribution is contained in the training set. If the control and recovered samples have similar means, and the mean is unusual, then the strength of evidence supporting the proposition that the samples are from the same source should be stronger than if the mean is relatively common.

A solution to this problem of the comparison of sources in the case where the measurements are univariate and are assumed to be independent and Normally distributed was developed by Lindley (1977). Some details are given in the Appendix; see Equations (12) and (13) in the Appendix. Denote the m measurements on the control sample by **x** = (x1, . . . , xm) and the n measurements on the recovered sample by **y** = (y1, . . . yn). The corresponding means of each of these samples are denoted x¯ and y¯. The two propositions to be considered are at the source level and are:

Lindley's solution assumes that the means x¯ and y¯ of the control and recovered samples are sample means of data, whose corresponding random variables have Normal distributions with means θ1 (control) and θ2 (recovered), respectively, and variances σ 2 /m (control) and σ 2 /n (recovered). The variance σ 2 is a within-group (e.g., within window) variance. The means θ1 and θ2 are the means of the groups associated with **x** and **y** in the terminology of hierarchical data. Variability between groups has also to be considered. This is done with consideration of the variation in the group means. The two means θ1 and θ2 are also assumed to be realizations of a random variable which is Normally distributed, this time with mean µ and variance τ 2 . At present the variances σ 2 and τ 2 are assumed known. Also, the within-group variance σ 2 is assumed constant within groups. An expression for the likelihood ratio if the betweengroup distribution is not Normal but is represented with a general distribution p(·), with second derivative p ′′(·) is given by Equation (14) in Appendix.

An extension using kernel density estimation has been derived to allow for a general non-Normal between-group distribution Equation (15) in Appendix. Checks of the distributional assumptions and estimation of hyperparameters are made using a training set of groups which are assumed to be a random sample of groups (sources) from some relevant population. Later work (e.g., Bozza et al., 2008 with an extension to multivariate data, Equation 24 in Appendix) relaxes the assumption that σ 2 and τ 2 are known.

The likelihood ratio can be used to assess evidence in a criminal trial and hence is a solution to the comparison of sources problem; Lindley (1977).