(1) Assumptions and inferences underlying fMRI processes and technologies need to be confirmed (or dispelled) so as to give credence to the scientific claims being made. Cognitive neuroscience, for example, assumes that complex thoughts have a physical counterpart that is both accessible and interpretable with technologies such as fMRI (Erickson, 2010). Many fMRI researchers operate on the basic assumption that lying involves additional efforts than telling the truth, which in turn can be signaled by heightened blood flow in specific brain regions (Gerard, 2008). However, several fMRI studies have been employing "reverse inference" as a central feature, whereby the activation of certain brain regions (X) is taken as evidence of a particular cognitive function (Y). As thoroughly discussed by Poldrack (2006), such inferences are only deductively valid if brain state X only occurs when cognitive function Y is engaged (i.e., if a selective association between X and Y is established), yet this one-to-one matching is not the case. Rather many-tomany matching of brain states to mental states are observed, and thus valid reverse inferences cannot be made here. What is required first of all is the creation of a robust "cognitive ontology" specifying the component brain operations that comprise specific mental functions, even before trying to establish univocal associations with functional anatomy (Poldrack, 2010). Furthermore, data-driven pattern analyses approaches (e.g., Haynes, 2008), although more current in terms of the methodology and less constrained by theoretical assumptions, still rely on the objective identification of what a lie is (e.g., Sip et al., 2008). However, this is not always possible and is especially unlikely in forensic contexts where lie detectors would be employed when neither facts nor subjective intentions can be directly verified. The validity of underlying assumptions must be addressed and a wide consensus reached within the scientific community before possible applications of the technology can achieve broad credibility.

not operate on shared but unproven assumptions that all or most brain's process lies similarly (Ellenberg, 2009; Holley, 2009). This includes correcting for variations in brain processes based on age; particularly juveniles. They need to be able to cope with the types of individuals usually encountered by law enforcement officers, including substance addicts, those with high incentives to lie, and those with mental disorders. Doubts already exist as to whether fMRI would be usable for those presenting with conditions such as delusions and amnestic disorders with confabulation (Langleben et al., 2006). Finally there is the issue of possible differences in outputs resulting from the social diversity of those tested, given that what is considered a lie is a matter of social convention which may vary on a cultural basis (Holley, 2009).

approaches increases the variability of results and the general reliability of the technique (see Bennett and Miller, 2010, for a thorough discussion of the issue); the consequences of which should not be underestimated and will be amplified in an applied setting. For if competing (but accepted) algorithms produce conflicting results when interpreting fMRI questioning data then opposing prosecution and defense experts will each exploit the one which best serves their purposes in court leading to a stalemate of probative value and a negation of fMRI evidence.

law setting then this technology will remain unacceptable. Randomized controlled trials with currently available testing protocols may not be a straightforward solution to this. In addition to result interpretation problems and the current lack of a comprehensive model of deception, "translational validation" requires access to real-world situations with minimal interference and the possibility to derive an objective index of performance for deception detection. The outcome of court proceedings, for example, could not be taken as an objective parameter for the discrimination between lie and truth (whatever lie detection task is being translationally validated). It is instead already possible to predict that the introduction of fMRI evidence will significantly influence juror decision-making, if unchallenged (see e.g., McCabe et al., 2011).

According to skeptics the enthusiasm for brain imaging and related "mind reading" applications largely overestimate its current ability to identify unique neural correlates of complex mental functions such as lying (but see Haynes and Rees, 2006; Haynes, 2008). Brain activations look extremely persuasive but they result from a long series of manipulations, assumptions, and interpretations. A precise and robust model of the mental processes involved in lying should guide hypotheses about brain activations, however such a generally accepted model remains absent. In addition, lies can be of different types (i.e., denying an event that has occurred vs. making up a slightly different story vs. telling a truth which will be interpreted as a lie; for example, think of a betrayed partner asking "with whom have you had dinner last night?," and the cheater sarcastically replying: "with my lover, obviously!" thereby telling the truth with the intent to make it sound like a lie). The context of basic fMRI experiments is artificial and one often has to sacrifice external for internal validity, and any attempts to make them more similar to real world scenarios will almost inevitably undermine internal validity.

In summary, while fMRI may be a useful research tool in combination with other techniques to clarify the mechanisms involved in lying, and its degree of sensitivity and specificity in lie detection may be higher than that of the polygraph, most scientists currently agree that fMRI research evidence is still weak and lacks both external and construct validity (Spence, 2008). We also must conclude that the current state of the science does not at this time meet the legal standards for admissibility in court proceedings (see Simpson, 2008 and Merikangas, 2008, for exhaustive discussions).

#### **LEGAL AND ETHICAL HURDLES**

by the judiciary as to the reliability of assisted lie detection techniques. This skepticism is partly borne out of the failure of the polygraph and now threatens to taint this new generation of neuroimaging technologies. The perverse irony for the cognitive neuroscientists who have been developing these new technologies in a conscious effort to address the legal short-comings of polygraphs, is that while techniques like fMRI might well-tick the boxes of reliability and objectivity when perfected, the solution of bypassing physiological responses in favor of the direct recording of neural activity may itself constitute grounds for the judiciary to reject neuroimaging technologies. Not because such solutions will necessarily lack reliability or objectivity, but because they potentially infringe other human/constitutional rights and legal principles. The developers of neuroimaging technologies need to acknowledge and engage with these legal issues *before* they seek to impose their new techniques into criminal courts if they are to maximize their chances of winning over the already skeptical judicial gate-keepers. For should they fail to find a way to square their new technologies with the existing legal principles set out below, then without legislative intervention their technologies will remain excluded from criminal courts.

(1) Possible constitutional and human rights violations (illegal search, right to silence, freedom of thought, right to privacy, human dignity, right to integrity of the person, and protection of personal data):

Looking across various common law legal systems, a number of constitutional principles, and human rights conventions10 will be engaged to differing degrees within different jurisdictions by the neuroimaging processes of fMRI. Ultimately without legislative intervention it will be the respective national courts who will be forced to rule on each of these issues, either when parties first seek to introduce fMRI evidence of statement veracity into criminal trials, or upon appeals to the first convictions/acquittals where this technology played a material part in arriving at a verdict. It is not our intention to examine each of these in depth here, rather to discuss broadly the various legal hurdles which must be addressed if fMRI is to find its place within criminal trials for determining the veracity of statements made.

The first set of issues is whether fMRI questioning constitutes a *search* of the subject, and when such a search will be considered lawful or unlawful. Discussions in this area tend to center on the US Constitutional Fourth Amendment protecting against unreasonable or unlawful searches (see Pardo, 2006, and Holley, 2009, amongst others for in-depth discussions on this point). A view exists that neuroimaging techniques will constitute a legitimate search under established legal doctrine should neural activity be

Since the 1920s proponents of assisted lie detection technologies have been predicting their inevitable acceptance by courts; first for polygraphs and now for neuroimaging technologies (Gerard, 2008), however to date fMRI evidence has never successfully been admitted in court for determining the veracity of statements by witnesses or defendants. This reflects the deep skepticism held

10The sources examined here are: the US Constitution (limited in application to US citizens within US territories), the European Convention on Human Rights (produced by, and applying to, the 47 member states of the Council of Europe, and overseen by the European Court of Human Rights), and the Charter of Fundamental Rights of the European Union (applicable to all citizens and residents of the 28 member states of the EU, this Charter enshrines a range of personal, civil, and social rights and existing conventions and treaties (including the European Convention on Human Rights) into EU law thus ensuring their legal certainty).

equated to other forms of physical evidence gathered from the human body, such as blood or DNA sampling, fingerprints, voice sampling, etc., providing probable cause exists justifying such sampling (Pardo, 2006). However, it is easy to conceptualize neural activity as distinct from other forms of physiological evidence. For example, while we can manipulate neural activity by conducting mathematical problems in our head, we cannot change our DNA profile through thought processes. What legal weight such a distinction would carry is moot until tested in court. The more challenging question is whether or not authorities should be allowed to record our neural activity without our consent, or even our knowledge. When confronted with this problem courts will be forced to either shoehorn this new technology into existing legal frameworks governing conceptually similar subject matter (i.e., DNA, blood, fingerprints, etc.) or produce new bespoke legal frameworks for their governance. In the latter case, the form of any new framework cannot be predicted. A final difficult question is whether police could require a person to undergo an fMRI test without a warrant, with no clear consensus existing between commentators on this point (see Pardo, 2006 and Holley, 2009).

Another set of issues is whether fMRI questioning undermines the right to silence and the right not to self-incriminate. Neuroimaging technology has the potential to undermine these rights if it can operate without the individual needing to speak. Within the United States, the Supreme Court has previously speculated that "the involuntary transmission of incriminating lie-detection evidence would violate a suspect's right to silence" (Simpson, 2008: 767). Under the European Convention on Human Rights (ECHR) whilst there is no explicit protection against self-incrimination, in the case of *Funke v France* (A/256-A, 1993; 1 C.M.L.R. 897, ECHR) the European Court of Human Rights (ECtHR) was explicit that the right not to selfincriminate is an implicit component of one's *Right to a fair trial* under Article 6 ECHR (Jackson, 2009) though it is not an absolute right (Berger, 2006). The ECtHR in *Saunders v United Kingdom* (1997, 23 EHRR 313) drew a distinction between material which respects the will of the suspect to remain silent and materials which exists independent of the suspect's will such as DNA, blood, urine, and breath. Unfortunately what they left for a future court to decide is whether or not an individual's brain activity exists independent of their will to remain silent?

It must also be asked whether questioning in fMRI without consent engages Article 8 Right to respect for private and family life and Article 9 Freedom of thought, conscience and religion of the ECHR. Article 8(1) has been broadly interpreted in the past, and it is readily conceivable that processes which seek to determine the veracity of our statements by measuring neural activity will engage this right. The question will turn on whether or not police will be able to conduct questioning in fMRI under the Article 8(2) qualifications of national security, public safety, and crime prevention, and what protections will needed to ensure proportionality. The answer to this might well be tied with Article 9, for to allow the state to access thoughts without consent and knowledge may have a chilling effect on both individuals and society as they seek to exercise their freedom of thought. Courts may well-seek to impose stringent safeguards on neuroimaging technology to prevent both the overuse and misuse of them if they feel these rights are threatened.

Additionally a number of rights within the Charter of Fundamental Rights of the European Union (CFREU) may also prove challenging for fMRI use in court proceedings. Article 1 *Human dignity* states that human dignity is inviolable and must be respected and protected. It is possible to argue that fMRI questioning without consent undermines an individual's dignity. Article 3 *Right to the integrity of the person* potentially poses the greatest challenge; especially Article 3(1) that everyone has the right to respect for their *physical* and *mental integrity*. This right may also be engaged by enforced or non-consensual fMRI questioning, especially given the express recognition of both physical and mental components. It is worth noting here that within France, Art.45 of LOI n◦2011-814 of 7 July 2011 (i.e., post ratification of the CFREU) which created Art.16-14 of the French Civil Code11 specifically limits the use of brain imaging techniques to medical purposes, scientific research, and in the context of judicial enquiries carried out by experts. But most importantly that the express and informed consent of the individual *must be obtained in writing* prior to *any imaging*, and that this consent is revocable at any time. Given how few national legislators have specifically acknowledged the use of neuroimaging in judicial proceedings, let alone the issue of consent, this early approach by the French government takes on considerable significance.

Finally Article 8 CFREU *Protection of personal data* poses a number of interesting questions. Firstly would fMRI data constitute personal data and thus fall under the protection of this article? Given the uniquely nature of such imaging in relation to an individual this must surely be the case. Assuming this is correct, under Article 8(2) everyone has the right to access their personal data and to have it rectified. As a result given the current fallibility of fMRI evidence, one could always argue that an interpretation of such data is incorrect and must be rectified. It would be interesting to see the effects on the admissibility of fMRI evidence in courts where one party seeks to challenge the ostensibly incorrect interpretation of their fMRI results by the other.

#### (2) Compelled questioning and covert surveillance:

The issues of compelled questioning and (as the technology develops) covert neuroimaging surveillance are ones which courts will be forced to face given the potentially profound impact covert surveillance of this nature will have on society as a whole. One of the concerns raised is the potential for authorities to use these technologies for *fishing trips* whereby police would question an individual to determine whether they have committed criminal acts in the past without any pre-existing evidence or reasonable suspicion. For police to search before they suspect is to undermine the presumption of innocence upon which our common law legal systems are built. Though as Pardo (2006) notes, it is not certain that such actions will be prevented under current regimes.

11Taken from the French Civil Code, Book I: People, Title I: The Civil Rights, Chapter IV: the use of brain imaging techniques.

(3) Probative value, unfair prejudice, and undermining the Province of the Jury:

For evidence (including scientific evidence) to be admissible in criminal courts it must be *relevant*, thus possessing *probative value*. The probative value of evidence can be defined as "the extent to which [this evidence] increases or decreases the probability of a fact in issue" (Dennis, 2007, p.108). Thus, for fMRI evidence the probative value is the extent to which it increases or decreases the *subjective veracity probability* of a declarant's statement; i.e., how it affects the factual probability that a person does or does not believe what they are saying.

However, probative value also refers to the *degree of relevance* evidence possesses, which is the extent to which evidence influences the probability of a fact in issue in the mind of a rational juror. Within England and Wales if the judge considers the probative value of evidence will have a prejudicial effect on this juror "disproportionate to the rational strength of the evidence as a means of proof, [then] the exclusionary discretion is available [to the judge] to prevent an accused suffering prejudice" (Dennis, 2007, p.108); thus the judge can exclude such disproportionate evidence. An example of evidence excluded may include full-color graphic photographs of injuries when a party seeks to admit these in addition to clear and factual medical reports. Similarly, within the US Federal Rules of Evidence (Rule 403, Federal Rules of Evidence) where the probative value of otherwise admissible evidence is substantially outweighed by the danger of unfair prejudice then this evidence too can be excluded. The question therefore becomes, will the courts reject *prima facie* admissible fMRI evidence on the basis that, because of its nature (or its presentation) it risks unfairly prejudicing the accused? Fears have been raised that: the graphic nature of fMRI evidence will result in unfair prejudice; that scientific lie detection evidence will unduly influence and taint jury deliberations; that jurors will not use their intuition and independent reasoning to critically challenge neuroimaging evidence; and that through function-creep such evidence will trespass into the *Province of the Jury* by effectively usurping their role as arbiter of fact (Gerard, 2008; see Weisberg et al., 2008 for evidence that fMRI images are seen as more compelling that other types and formats of data).

Supporters of neuroscience technologies consider concerns over the undermining of judges and juries as unfounded; rather neuroimaging evidence will simply make the predictions of veracity by jurors and judges more reliable (Pardo, 2006). Pardo makes the argument that:

Because even a highly reliable neuroscience test would not establish knowledge or lies directly, jurors would still need to play their traditional role in assessing it. In making these assessments, the jury would, for example, consider whether other evidence regarding credibility should override the test results, rendering the test conclusion unlikely. (2006: 318)

While this statement seeks to defend and support fMRI in criminal courts, it unwittingly demonstrates the danger that this technology will import unfair prejudice into criminal trials. To explain; by rightly accepting that even a highly reliable neuroimaging test does not directly establish knowledge or lies one must ask "what is the point of introducing evidence to a jury from a technology that cannot provide direct evidence as to the veracity of statements made but is still marketed and promoted as a scientifically accurate lie detector12?" The obvious danger here is that the nuance between *lie detection* and *statement veracity* will not be clearly explained at the start of a case and/or not maintained and reinforced as the case progresses, leading juries to overestimate the capabilities of this technology. This is highlighted by the remainder of the above quote where the neuroimaging test results are already being presented by the author as the *de facto* position of truth, one which can only by overridden should *other evidence regarding credibility override the test results*; i.e., the tests shall be the truth unless you can prove otherwise. This statement, while seeking to defend neuroimaging technologies, actually serves to highlight the potential disproportional probative effect of neuroscience lie detectors. Cognitive neuroscientists must be careful not to overplay what these technologies can offer criminal courts nor their vision of the potential future role of neuroscience within criminal courts, lest they overplay themselves out of the courtroom altogether.

#### (4) Right to a fair trial:

Depending on how questioning in fMRI is conducted for criminal trials it can be argued that the fairness of trials will be placed at risk unless *all* parties to the trial are subjected to pretrial questioning in fMRI. Presenting fMRI evidence from only one party to the case may result in an artificial disparity of evidence; i.e., the neuroimaging evidence plus testimony vs. testimony without neuroimaging evidence. Justice may now depend on whether or not a jury will question a technology promoted as a highly accurate lie detector, so to ensure parity of arms and a fair trial all parties should be subjected to pre-trial questioning in fMRI if they are ever introduced. Of course such a scenario depends on all the parties being capable of undergoing fMRI testing which is not the case when the victim is dead or comatose. In these circumstances fMRI evidence may need to be prohibited.

Nevertheless, it is conceivable that fMRI evidence may become admissible solely as a defense instrument given that structural brain scans have already found acceptance and are widely admissible as mitigating evidence during sentencing proceedings. Indeed such use may help ensure a trial is ultimately fair. However, any attempt to extrapolate from this niche application such that fMRI evidence can be used throughout the entirety of

12Both of the two commercial companies offering fMRI detection service specifically and deliberately promote their technologies as scientific lie detection tools and not as veracity probability enhancement tools; No Lie MRI claims their technology 'represents the first and only direct measure of truth verification and lie detection in human history' (see http://noliemri*.*com/), while CEPHOS Corp claims to have developed 'the latest, most scientifically advanced, brain imaging techniques for scientifically accurate lie detection' (see http://www*.*cephoscorp*.*com/).

the trial *but only by the defense* represents an arguably unacceptable asymmetry of measures; one which potentially undermines the overall fairness of the trial (both actual and perceived) and the rights of victims.

#### **CONCLUDING POINTS**

Our discussion throughout has focused on the scientific, legal, and ethical hurdles facing those seeking to introduce fMRI evidence into trials as a means of assisting judges and juries in determining the veracity of statements made. Schauer (2010) suggests that as the goals of the law differ from those of science, what is not good enough for science may yet be good enough for the law and *vice versa*. However, following our assessments of the science underpinning fMRI as a lie detector and how this relates to the law, we must conclude that the current state of this technology, and potentially the technology *per se*, fails to meet either acceptable scientific or legal standards.

The evaluation of fMRI accuracy in lie detection—in some cases claimed to be as high as 0.90—is indeed based on laboratory experiments conducted with compliant participants, which is unlikely to be true of most legal settings where non-compliance and the use of countermeasures would make its accuracy figure drop dramatically (e.g., Ganis et al., 2011). In the cognitive neurosciences fMRI is not sufficient by itself to unveil which brain areas are epiphenomenal, which are strictly necessary to lying. It may thus pick up some noise together with the real signal. Even if it was possible to produce a correlational map where a constant pattern could be detected indicating a lie, issues of replicability and generalizability across conditions and participants could be raised. And even more so in those cases where facts are unknown to the tester and there is no objective reality against which to establish whether a person is lying or not. From the legal perspective, until the science behind fMRI testing improves it will not meet the relevance and reliability thresholds required for any scientific evidence to be admissible in criminal trials. The assumptions, inferences, and questions of internal validity which so pervade current fMRI testing and analysis need to be addressed. As does the challenge of successfully applying this technology to criminal justice scenarios characterized by their confrontational emotional nature and the personal high-stakes involved for the participants.

Neuroimaging in courts also raises the specter of potential constitutional and human rights violations. Questions arise as to whether or not such testing constitutes an illegal search as well as how it respects rights to privacy, silence, thought, and a fair trial are all engaged by this technology yet left unanswered. Unless the admissibility decision is taken out of the hands of the judiciary by politicians, which is itself a likely scenario, ultimately it will be for the courts to decide the fate of fMRI evidence in criminal trials. Given the range and depth of the legal and ethical issues identified in the earlier sections, the likely outcome will probably fall on a spectrum somewhere between outright rejection through to some form of restricted and regulated usage, as opposed to the highly unlikely scenario of *carte blanche* acceptance.

What we have not discussed within this paper are both the *operational* and *social* barriers to the widespread use of fMRI testing in criminal trials. These barriers are potentially just as daunting as their scientific and legal counterparts.

From the *practical operational perspective*, issues which require future examination include; the cost of purchasing, staffing, and maintaining sufficient fMRI machines to cater for a national justice system; the additional time and monetary costs fMRI testing will add to criminal cases; how fMRI testing can be made to work within adversarial systems of questioning and cross-examination based on earlier responses; and the lack of a courtroom-friendly portable fMRI system. Additionally there are questions specific to the assessment algorithms used when interpreting fMRI response data: will only a single universal *official* algorithm be allowed?; will commercial patented algorithms be admissible if they are not completely open for inspection and independent verification?; and finally what happens when new algorithms and new fMRI scanners are inevitably developed as the science is continually refined which prove to be more reliable and sensitive than previous algorithms/machines? It is conceivable that those who maintain their innocence and are appealing their conviction under the previous technology will seek to be re-tested with the new machines and the new algorithm in an effort to prove their innocence placing a further burden on the criminal justice system. All of these points possess the potential to impact upon the fairness of future trials.

A final hurdle to the widespread introduction of fMRI testing is *societal acceptability*, without which technologies such as neuroimaging techniques for determining the veracity of statements within criminal trials will lack both public confidence and legitimacy. Future research needs to gauge the levels of public support for such technologies, for even if neuroimaging proves superior to humans as arbiters of statement veracity in criminal courts, this fact in of itself may not be enough for the public to accept their introduction if they are apprehensive or hostile to what such technologies represent for their future. We cannot escape from asking the question, *will people accept mind reading machines?* This is obviously not what the current generation of neuroimaging technologies is, but they are a small step down this long path.

Our societies have developed to both accept and respect an individual's right to keep secrets, and in so doing they do not seek to override human beings' evolved capacity to keep secrets, for a society where individuals are denied secrets is not a human society as we know it. The developers and proponents of fMRI testing must respect this fact and engage society in their research as it progresses. Otherwise they may find they successfully negotiate the frying-pan of scientific and technical challenges in perfecting fMRI testing only to be consumed by a fire of legal, ethical, social, and political opposition.

#### **ACKNOWLEDGMENTS**

The authors would like to thank Roberto Cubelli for his comments on a previous version of the manuscript, the Editor and three Reviewers for their very constructive comments. This study was funded by EPSRC (grant number EP/G037264/1).

#### **REFERENCES**

*Biobehav. Rev.* 35, 516–536. doi: 10.1016/j.neubiorev.2010.06.005

Detecting deception: the scope and limits. *Trends Cogn. Sci.* 12, 48–53. doi: 10.1016/j.tics. 2007.11.008

trends and directions for future scholarship. *Am. J. Bioeth.* 7, 44–56. doi: 10.1080/152651607 01518714

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 July 2013; paper pending published: 27 August 2013; accepted: 03 September 2013; published online: 24 September 2013.*

*Citation: Rusconi E and Mitchener-Nissen T (2013) Prospects of functional magnetic resonance imaging as lie detector. Front. Hum. Neurosci. 7:594. doi: 10.3389/fnhum.2013.00594*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Rusconi and Mitchener-Nissen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Addressing social resistance in emerging security technologies

#### *Timothy Mitchener-Nissen\**

*Department of Security and Crime Science, University College London, London, UK*

#### *Edited by:*

*Elena Rusconi, University College London, UK*

*Reviewed by: Aaron Winter, University of Abertay, UK*

*Nicola Lettieri, ISFOL, Italy*

*\*Correspondence: Timothy Mitchener-Nissen, Department of Security and Crime Science, University College London, 35 Tavistock Square, London WC1H 9EZ, UK e-mail<EMAIL_ADDRESS>
In their efforts to enhance the safety and security of citizens, governments and law enforcement agencies look to scientists and engineers to produce modern methods for preventing, detecting, and prosecuting criminal activities. Whole body scanners, lie detection technologies, biometrics, etc., are all being developed for incorporation into the criminal justice apparatus.1 Yet despite their purported security benefits these technologies often evoke social resistance. Concerns over privacy, ethics, and functioncreep appear repeatedly in analyses of these technologies. It is argued here that scientists and engineers continue to pay insufficient attention to this resistance; acknowledging the presence of these social concerns yet failing to meaningfully address them. In so doing they place at risk the very technologies and techniques they are seeking to develop, for socially controversial security technologies face restrictions and in some cases outright banning. By identifying sources of potential social resistance early in the research and design process, scientists can both engage with the public in meaningful debate and modify their security technologies before deployment so as to minimize social resistance and enhance uptake.

**Keywords: technology, social, resistance, ethics, security**

#### **INTRODUCTION**

Social constructionism is a sociological theory of knowledge which holds that our knowledge of the world is not derived from observing nature, rather that it is constructed through the social interactions and processes of people (Burr, 2003). By adopting a social constructionist perspective one can comprehend the phenomena of criminality and criminal behavior as existing from the moment individuals and societies began socially constructing and adopting laws which proscribed certain acts or omissions as constituting criminal activities (Newburn, 2007). Under this formulation that which is considered *criminal* can differ spatially (different countries, states, districts, towns have different laws), by ascribed categories (i.e., different laws for different religious groups, genders, sexual orientations, professions and/or social classes) and temporally (laws are not "set-in-stone," rather are subject to change). Yet while laws can change to reflect both prevailing social views and the organization of activities within a society, the slow pace of this change often results in the law struggling to catch up. The advent of the digital age, the pace of technological development, and the widespread adoption of technologies in many societies all pose challenges for the application of existing laws and the timely creation of new ones.

This paper begins by examining the phenomenon whereby states embrace technologies as solutions or *fixes* for the problem of crime. The negative consequences of this policy in the form of social resistance are then discussed. Finally the question is asked as to why the design and implementation of emerging security technologies continues to repeat mistakes observed in previous technologies? Four answers are provided here, including; (i) the paucity of social education within science, technology, engineering, and mathematics (STEM) courses, (ii) the lack of priority afforded social and ethical issues within the research and design of security technologies, (iii) a general failure by STEM practitioners in comprehending the importance of social acceptability to the technologies they create, and (iv) restricted public engagement.

#### **SECURITY TECHNOLOGIES AS TECHNOLOGICAL FIXES**

The development and interpretation of new technological advancements have been adopted with considerable enthusiasm by governments, law enforcements agencies, universities and private companies as potential methods for preventing, detecting, and prosecuting criminal activities. In this regard they represent *technological fixes* for the social problem of crime; a technological fix is broadly defined as a technological solution for *solving* social problems (Weinberg, 1967) reflecting the views of technological optimists. Technology

1I am using this umbrella term to cover all the organisations involved in every stage of the prevention, detection, and prosecution of criminal activities. This includes: (i) the work of the security services with their roles of collecting intelligence (both domestic and international) to protect the national security and economic well-being of a nation as well as supporting the prevention and detection of serious crimes; (ii) domestic law enforcements organisations such as police and border agencies with their various roles in preventing, detecting and deterring criminal activities, as well as gathering evidence to assist in the prosecution of those accused of committing crimes; and (iii) the criminal court system, including the prosecution and defence who make use of scientific evidence and experts when furthering the case of their clients.

is presented as a panacea for social problems by being cheaper and more effective than alternative human-centric approaches for dealing with issues which negatively impact society.

The current range of technological fixes designed specifically for addressing crime (hereafter referred to as *security technologies*2) continues to increase as scientists and engineers seek to apply the knowledge and approaches of their specific fields to this particular goal. Whole body scanners at airports utilize X-ray backscattering or millimeter wave technology so as to identify metallic and non-metallic objects, plastic and liquid explosives, flora, fauna, drugs, and cash, concealed within or beneath the clothing of passengers (European Commission, 2010; Mitchener-Nissen et al., 2012). Data mining, being the application of database technology and techniques (such as modeling and statistical analysis) to data to identify valid, novel, implicit and potentially useful information and patterns within that data, is employed with the aim of analysing intelligence and detecting terrorist activities, fraud, and other criminal patterns (Tien, 2004; Steinbock, 2005; Schermer, 2011). The use of biometrics enables crime-scene technologies that can assist in the identification and prosecution of offenders (such as DNA databases and fingerprinting technologies), tackling identity fraud, and counteracting illegal immigration (Grijpink, 2006; Goldstein et al., 2008). And to assist in the investigation and prosecution of criminal acts, lie detection technologies designed to directly access brain function (including fMRI and EEG) are trying to be developed by researchers and private companies (Wolpe et al., 2010). This selection represents a tiny snapshot of the cornucopia of security technologies both under development and already implemented.

#### **RESULTING SOCIETAL RESISTANCE**

Without further examination it would be tempting to conclude that security technologies do indeed constitute justifications for Weinberg's vision of technological fixes as the solution to social problems. However, the notion of the technological fix has been subject to robust criticism. It has been described as "a quick cheap fix using inappropriate technology that creates more problems than it solves" (Rosner, 2004). The truth of this statement is evident within the social controversies (or in the case of the lie detection technologies, the possible future social controversies) produced by each of the security technology examples provided above. Whole body scanners have been accused of conducting digital strip-searches (Klitou, 2008), and the backscatter variation is to be removed from US airports because of the images produced. Data mining has been associated with both a fear of totalitarian-style state observation, as well as the targeting of individuals by governments (Steinbock, 2005). Different biometric technologies can discriminate against various groups within society and are plagued by the problem of false positives (Hunter, 2005; Whitley and Hosein, 2010). Additionally the UKs DNA database (the largest in the world) has created controversy by holding the details of innocent people and a disproportionate number of samples from ethnic minorities. And the new generation of potential lie-detection technologies have faced criticism over the potential ethical, social, and legal implications of their operation to existing social and legal institutions should they ever be made to definitively and consistently "work." This social resistance to a security technology begins individually, as solitary citizens question the rationale and/or operation of a particular measure. These may be individuals who actively critique government security policy, those who prioritize privacy and liberty, or as is often the case these are individuals who find themselves adversely impacted upon by a security technology without just cause. For example; individuals who are incorrectly prevented from flying because either they have the same name as another person on a no-fly list, or their details have been added in error to such a list without them being previously notified or provided a way to rectify this error. Recognition of an individual's issues with a security technology can now begin to coalesce into social resistance once knowledge of their plight becomes known to others. The media, lawyers, NGO's, social activists, political figures, and independent commissioners amongst others can all assist is raising awareness here, which in turn can influence other citizens thereby snowballing the effect and reducing support for the security technology in question.

The manifestation of social resistance present in the technologies discussed above represents only a snap-shot of the controversies produced by security technologies which have in the past undermined their social acceptability and widespread uptake. In an on-going examination of security technologies which have evoked social resistance, I have identified numerous recurring controversies which continue to arise within new security technologies with depressing regularity. These can be organized into eight high-level categories; the causing of physical and mental harm, questions of legality, financial costs, liberties and human rights issues, broader public responses, issues of functionality, security and safety issues, and abuse/misuse issues. A selection of commonly recurring controversies includes; privacy concerns, function creep, false positive/negative rates, lack of public trust, the failure of a technology to achieve what its designers claim it can do, and the potential for the technology to be abused by the state.

#### **WHY NEW SECURITY TECHNOLOGIES REPEAT THE MISTAKES OF THE PAST**

The question which needs addressing here is why have lessons not been learnt such that new security technologies consistently evoke such ethical and social controversy? I suggest there are four complementary elements underpinning the answer to this question. The first is the paucity of social and ethical education within university STEM courses. Within university engineering courses in the UK it is highly likely that a student can (and will) complete their education without ever undertaking a single lecture on the importance of identifying and incorporating social and ethics factors into

2By *security technologies* I am referring to the product of an engineering endeavour which seeks to deter, prevent, detect or prosecute crimes, and/or enhance the security of individuals, their property, or the state (including its infrastructure).

their work. This is despite the creation of the field of engineering ethics which arose in the early 1980s following a number of technological developments, designs and failures which negatively impacted human wellbeing (Johnson and Wetmore, 2008). The situation is repeated within the hard sciences with the possible exception of medical ethics. For those who counter with the claim that ethics and ethical research is ensured by the presence of university ethics boards; while a particular research or design project may meet all official conduct requirements such that it is considered ethical, this does not mean that what is being undertaken or created will be accepted by the public. The diverse groups which comprise a society ultimately determine what is considered socially or ethically acceptable, and yet university engineering and hard science courses regularly fail new researchers and designers by not equipping them with an understanding of this fact nor the tools to adequately interact with the public.

The second element in the lack of priority afforded social and ethical issues within research and design projects. Interviews with engineers and scientists engaged in the process of designing and developing new security technologies have highlighted a clear hierarchical structure to the design process. For commercial projects it begins with cost; if it is determined that there is not a viable market for a product then it will not be produced. If this test is passed and the project is considered feasible than design specifications are produced in accordance with the client's requirements and the product is created. Similarly with university research projects, the presence of funding and/or the potential for future commercial exploitation dictates the research undertaken. When this is directed toward addressing perceived security deficiencies the focus is on attaining a specific security goal. These processes leave little space for the consideration and incorporation of social and ethical issues the focus is on "can we achieve what we have set out to achieve," and not "is this a socially acceptable way of achieving the desired goals" or "are these goals socially acceptable *per se.*"

The third element is a general failure by scientist and engineers to comprehend just how important social acceptability is to the life cycles of their technologies. In the majority, scientists and engineers do not develop an appreciation of the importance of identifying and addressing social concerns until they are confronted by social resistance; a point often reached *after* a product has been released to market.

The fourth element is the challenge of, and the resistance to, achieving effective public engagement in relation to the design of security technologies. The arguments in favor public engagement hold that just as democracy derives its legitimacy through participation, so too will increasing participation within the development of new or controversial technologies help to infuse the finished products with similar legitimacy and reduce societal resistance. The primary argument against is that lay people are handicapped by a lack the technological literacy, or access to and understanding of, security-sensitive intelligence, which together constrain their ability to provide relevant input or make informed decisions. But as Kleinman (2005) highlights, the flawed nature of such views is driven home by the fact that experts3 are never value-neutral, unbiased, all-seeing individuals; rather are bounded by the nature of their expert knowledge and will necessarily view a phenomenon from a partial perspective. In other words, experts are handicapped to view the world through blinkers and in this respect have similarities with the very lay public whose input they would seek to exclude.

By introducing socially unacceptable technologies in the first place, trust in both the developers and the end-users (i.e., governments and agencies of the state) is threatened, research and design capacity is diverted from acceptable technologies, and money is wasted that could otherwise have been used for legitimate programmes. The challenge becomes identifying what is acceptable and unacceptable *before* a technology is developed and deployed. By accepting that judgments over acceptability of a technology differ between social groups and that rejection of a technology can lead to its permanent inferiority through neglect (MacKenzie and Wajcman, 1999), the consideration of wider social and ethical issues upstream in the design process to anticipate and mitigate negative social reactions becomes both a valid and logical response.

#### **CONCLUSION**

The list of technologies developed which have been banned or their use restricted in various societies, (not necessarily because of deficiencies in the underlying science) but because the developers did not seek to anticipate and mitigate social resistance through upstream design modifications is long and growing. It includes backscatter body scanners, instances of data mining, less lethal weapons, polygraph lie detectors, CCTV, national ID card, etc.

To avoid the ignominy of this situation for emerging security technologies developers must take meaningful steps to identify sources of potential social resistance early in the research and design process. This requires truly reflexive engagement with the public to identify concerns which then can be translated into upstream design requirements; thereby heading off social resistance before it coalesces and becomes synonymous with the technology being developed. The enormity of this challenge cannot be overestimated for if a proposed technology cannot by created in such a fashion which respects and reflects the values held within a society, then those developing the technology are wasting valuable time, money and resources on research which will ultimately be rejected.

#### **ACKNOWLEDGMENTS**

This research was funded by the Engineering and Physical Sciences Research Council of the United Kingdom through their Centers for Doctoral Training programme, specifically the Security Science Doctoral Research Training Centre (UCL SECReT) based at University College London.

3In this case STEM practitioners, state officials, and lawenforcement/intelligence officers.

#### **REFERENCES**

Burr, V. (2003). *Social Constructionism,* 2nd Edn. Hove: Routledge.

**Conflict of Interest Statement:** The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 26 April 2013; accepted: 31 July 2013; published online: 20 August 2013. Citation: Mitchener-Nissen T (2013) Addressing social resistance in emerging security technologies. Front. Hum. Neurosci. 7:483. doi: 10.3389/fnhum. 2013.00483*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Mitchener-Nissen. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

## Can laptops be left inside passenger bags if motion imaging is used in X-ray security screening?

#### *Marcia Mendes 1,2, Adrian Schwaninger 1,2\* and Stefan Michel 1,2*

*1 School of Applied Psychology, University of Applied Sciences and Arts Northwestern Switzerland (FHNW), Olten, Switzerland 2 Center for Adaptive Security Research and Applications (CASRA), Zürich, Switzerland*

#### *Edited by:*

*Andrea Szymkowiak, University of Abertay Dundee, UK*

#### *Reviewed by:*

*Elena Rusconi, University College London, UK Lynne M. Coventry, Northumbria University, UK*

#### *\*Correspondence:*

*Adrian Schwaninger, School of Applied Psychology, Institute Humans in Complex Systems, University of Applied Sciences and Arts Northwestern Switzerland, Riggenbachstrasse 16, 4600 Olten, Switzerland*

*e-mail<EMAIL_ADDRESS>
This paper describes a study where a new X-ray machine for security screening featuring motion imaging (i.e., 5 views of a bag are shown as an image sequence) was evaluated and compared to single view imaging available on conventional X-ray screening systems. More specifically, it was investigated whether with this new technology X-ray screening of passenger bags could be enhanced to such an extent that laptops could be left inside passenger bags, without causing a significant impairment in threat detection performance. An X-ray image interpretation test was created in four different versions, manipulating the factors packing condition (laptop and bag separate vs. laptop in bag) and display condition (single vs. motion imaging). There was a highly significant and large main effect of packing condition. When laptops and bags were screened separately, threat item detection was substantially higher. For display condition, a medium effect was observed. Detection could be slightly enhanced through the application of motion imaging. There was no interaction between display and packing condition, implying that the high negative effect of leaving laptops in passenger bags could not be fully compensated by motion imaging. Additional analyses were carried out to examine effects depending on different threat categories (guns, improvised explosive devices, knives, others), the placement of the threat items (in bag vs. in laptop) and viewpoint (easy vs. difficult view). In summary, although motion imaging provides an enhancement, it is not strong enough to allow leaving laptops in bags for security screening.

**Keywords: aviation security, X-ray screening, threat detection, human factors, motion imaging, multiple views, laptop screening**

#### **INTRODUCTION**

A secure air transportation system is vital for society and economy. Aviation security measures have been increased substantially in response to several successful and attempted terrorist attacks since September 11, 2001. One major aspect in this field is the mandatory process of baggage screening using X-ray machines. Before entering the secure area of an airport, all passengers, as well as members of airline and airport staff have to pass the security checkpoints to have themselves and all their belongings screened. The security checkpoint is a socio-technical system consisting of human and technical elements, working together. The goal is that no threat items are brought past security checkpoints and onto an airplane. Strong efforts are being made in order to improve and further develop X-ray screening equipment. Yet, the final decision whether threat items are contained in the baggage still relies on human operators (screening officers) who visually inspect the X-ray images provided by the machine. As a consequence, manmachine system performance depends on human factors and display technology (e.g., Bolfing et al., 2008; Koller et al., 2008; von Bastian et al., 2008, 2010; Michel and Schwaninger, 2009; Graves et al., 2011). When evaluating new technological developments with regard to their added value for security screening purposes, this should be taken into account appropriately (see also Yoo and Choi, 2006; Yoo, 2009).

In X-ray screening, three image-based factors have been identified as relevant for human operators to detect threat items in X-ray images (Schwaninger, 2003b; Hardmeier et al., 2005, 2006; Schwaninger et al., 2005a). The first one is the view difficulty of an object, resulting from the position of a threat item in a bag (effect of viewpoint). The second factor is the superposition of an item by other objects contained in the bag (effect of superposition). The third factor refers to the complexity of a bag, which depends on the number and type of objects in the bag (effect of bag complexity). The intensity with which X-rays can penetrate through materials in a bag depends on the specific material density of a substance (e.g., Brown et al., 1995). Therefore, the material density of the items contained in a bag will also affect the factors superposition and bag complexity and thus will influence the difficulty to detect threat items. Schwaninger et al. (2005b) have developed algorithms to automatically estimate X-ray image difficulty based on viewpoint, superposition, and bag complexity. Their algorithms were highly correlated with human perception of the above mentioned image-based factors and could well predict human threat detection performance (see also Schwaninger et al., 2007; Bolfing et al., 2008).

State-of-the-art X-ray screening equipment is able to provide high quality images with good image resolution. Yet, the detection of threat items in X-ray images remains a challenging task for screening officers and becomes even more difficult when dense objects, such as large electronic devices, are contained in the baggage. Due to their compact construction, electronic devices (e.g., laptops) are hard to penetrate. Hence, they can conceal other parts of luggage or could be used to intentionally hide threat items (e.g., an improvised explosive device, IED). Especially when single view X-ray systems are used or even multi-view systems, if the additional views do not provide enough meaningful information, the inspection becomes difficult. Threat items which are behind, in front of, or hidden inside a laptop case become very challenging or even impossible for human operators to recognize (see also von Bastian et al., 2008). In a previous paper, Mendes et al. (2012) documented how threat detection can be substantially impaired when laptops are not taken out of passenger bags and a threat item (e.g., an IED) is placed either behind, in front of, or within a laptop. The present paper extends these results by investigating how a new technology which allows presenting bags in multiple views as an image sequence (i.e., motion imaging) could possibly reduce such an impairment.

Considering the large number of views which can be produced by a single object, the question arises how objects can be recognized when presented in unusual views. In the object recognition literature, two types of theories can be distinguished (see Peissig and Tarr, 2007; Kravitz et al., 2008): viewpoint-invariant theories (e.g., Marr, 1982; Biederman, 1987) and viewpointdependent theories (e.g., Poggio and Edelman, 1990; Bülthoff and Edelman, 1992; Tarr, 1995). Most viewpoint-invariant theories assume that objects are stored in visual memory by their component parts and their spatial relationship (see Marr and Nishihara, 1978; Biederman, 1987). Once a particular object has been stored, recognition of that object should be unaffected by the viewpoint (including novel viewpoints), given that the necessary features can be recovered from this view (Burgund and Marsolek, 2000). The viewpoint-dependent theories propose that objects are not stored in memory as rotation invariant structural descriptions, but in a viewer centered format. Thus, if an object has never been seen from a certain viewpoint and is therefore not stored in visual memory, recognition is impaired if view-invariant features are not available (Kosslyn, 1994; Bülthoff and Bülthoff, 2006; Schwaninger, 2005). Several studies on viewpoint-dependent theories could show that viewpoint can strongly affect recognition performance (e.g., Bülthoff and Edelman, 1992; Edelman and Bülthoff, 1992; Humphrey and Khan, 1992; Graf et al., 2002). Even though our visual perception can be considered highly robust with respect to changes of viewpoint, we are more facile with certain views relative to others, such as often encountered views and views that make larger numbers of surfaces available (Palmer et al., 1981; Blanz et al., 1999). Such views have also been referred to as "canonical" views. Research in aviation security X-ray screening has shown that threat items are easier to identify when depicted in frontal (canonical) views than when horizontally or vertically rotated (e.g., Michel et al., 2007; Bolfing et al., 2008; Koller et al., 2008). Consequently, having machines featuring multiple X-ray images of the same bag from different viewpoints could ease recognition of threat items in passenger bags for screening officers.

At present, most of the machines deployed at airports provide single view images, which do not allow screening officers to analyze an image from different viewpoints. A human operator will only be able to identify a threat item and make a correct decision if the threat can be recognized in the provided single view image (Schwaninger, 2003b; Schwaninger et al., 2005a; Graves et al., 2011). Considering the above mentioned image based factors (viewpoint, superposition and bag complexity) and the density of electronic devices, it becomes evident why most international and national regulations specify that portable computers and other large electronic devices shall be removed from passenger bags and screened separately at security checkpoints (e.g., the current regulation of the European Comission, 2010). Based on the model by Schwaninger et al. (2005b) one would predict that leaving laptops in passenger bags results in decreases of threat detection performance due to increases of superposition and bag complexity. Threat items placed behind, in front of, or inside a laptop could become very challenging for human operators to detect. Moreover, recognition would become additionally challenging if in the provided X-ray image the threat item would be depicted from a difficult viewpoint (e.g., vertically or horizontally rotated).

This study was conducted to examine the above mentioned effects by comparing conventional single view display technology to a new technology.  "Motion imaging" means that five images are available, which are rotated around the vertical axis. These can be either displayed in a short video sequence or can each be statically viewed. In relation to the initial image (0◦), the angles of the five images are −25◦, −12*.*5◦, 0◦, 12*.*5◦, 25◦ (see **Figure 1**).

One could hypothesize that through the application of motion imaging and the availability of multiple views, recognition of certain objects could become easier. There are several possible advantages dynamic displays may confer over static ones (Vuong and Tarr, 2004). For example, object motion may enhance the recovery of information about shape (e.g., Ullmann, 1979). Furthermore, it may provide observers with additional views of objects (Pike et al., 1997), or it may allow observers to anticipate views of objects (Mitsumatsu and Yokosawa, 2003). Moreover, when objects rotate in depth, certain features can become visible while others become obscured (Vuong and Tarr, 2004). Thus,

**FIGURE 1 | Example of motion imaging X-ray images provided by the machine evaluated in this study.** The image in the middle shows the initial image (0◦).

objects could become less superimposed and could possibly be displayed from an easier viewpoint (i.e., from a more canonical perspective).

The first goal of our study was to determine whether motion imaging improves detection of threat items in passenger bags. The second goal was to investigate whether leaving laptops in passenger bags results in a decrease of detection performance (effect of superposition and bag complexity), while the third goal was to evaluate whether such an effect can be compensated when motion imaging is available. Additional analyses were carried out to examine effects depending on different threat categories (guns, IEDs, knives, others), the placement of the threat items (in bag vs. in laptop) and the viewpoint effect (easy vs. difficult view).

#### **METHODS AND MATERIALS**

An image interpretation test containing bags and laptops was created in four versions to examine the factors display condition (single vs. motion imaging) and packing condition (laptops inside vs. laptops outside). Each test version differed with regard to these two factors (see section Experimental Design). Four experimental groups with certified screening officers were formed. Each group conducted one of the test versions. Detection performance scores and reaction times (RTs) of all groups were compared to evaluate the effects of the above mentioned factors.

#### **PARTICIPANTS**

The study was conducted with 80 airport security screening officers employed at an international European airport. All participants were certified screeners, meaning they were all qualified, trained and certified according to the standards set by the national appropriate authority (civil aviation administration) and consistent with the European Regulation (European Comission, 2010). The screening officers were randomly distributed into four different experimental groups (A, B, C, and D, 20 per group). **Figure 2** illustrates the experimental design. In order to verify that all experimental groups were comparable with regard to the screeners' X-ray image interpretation competency, all participants conducted the X-Ray Competency Assessment Test (X-Ray CAT) before the main experiment was carried out. The X-Ray CAT for cabin baggage screening is a standardized instrument to measure X-ray image interpretation competency of airport security screening officers and has been applied in several previous scientific studies (Koller and Schwaninger, 2006; Michel et al., 2007; Koller et al., 2008). It is currently used for screener certification at several European airports. The test consists of 256 trials and is based on 128 different color X-ray images of passenger bags, which are each used twice: once without (non-threat image) and once containing a threat object (threat image). For more information on the X-Ray CAT see Koller and Schwaninger (2006). Average detection performance scores (A )1 of all four groups were compared using *post-hoc* pairwise comparisons with Bonferroni correction. No significant differences between the groups could be found (all *p* values *>*0.05), implying that they were comparable regarding their image interpretation competency. The average age of the participants was *M* = 40*.*69 years (*SD* = 10*.*78), with a range between 22 and 58 years. 53% of the participants were female. The average amount of job experience was *M* = 4*.*95 years (*SD* = 4*.*49, range: 0.5–23 years). Betweenparticipants analyses of variance showed no differences between the experimental groups with regard to age [*F(*3*,* 76*)* = 1*.*57, *p* = 0*.*204, η2 = 0*.*058] or job experience [*F(*3*,* 74*)* = 0*.*66, *p* = 0*.*579, η2 = 0*.*026].

#### **EXPERIMENTAL DESIGN**

All experimental groups conducted a computer-based X-ray image interpretation test. During the test, color X-ray images of passenger bags and laptops were displayed, sometimes containing threats (threat images) and sometimes without any threat items (non-threat images). Images were displayed in random order.  Each test condition differed with regard to the factors display condition (single view vs. motion imaging) and packing condition (laptops inside vs. outside of passenger bags). **Figure 2** displays the experimental design of the study. The following four

1For details on the calculation of A see section Results and Discussion.

different experimental conditions were conducted and compared to examine the effects and interactions of the above mentioned two factors using a between-participants design:

In all test conditions the same bags were presented to the screening officers. Originally, every bag contained a laptop. In conditions A and C the laptops were taken out of the bag and screened separately, whereas in conditions B and D the laptops were left inside the passenger bags. This allowed examining the effects of superposition and bag complexity caused by laptops. **Figure 2** illustrates the two different packing conditions (laptop inside vs. laptop outside).

In conditions A and B, images of the baggage and laptop could only be seen from one single viewpoint. Conditions C and D allowed examining the images from different viewpoints through motion imaging. As explained in the introduction, one important objective of this study was to test whether motion imaging could enhance the inspection of passenger bags to such an extent that laptops could be left inside passenger bags without affecting detection performance negatively. Would detection performance scores still be significantly higher in condition A compared to D, one could conclude that the detection of threat items is significantly impaired when laptops are left inside passenger bags, even when motion imaging is available.

#### **IMAGE INTERPRETATION TEST**

The image interpretation test was based on a representative set2 of 96 passenger bags (defined by screening experts from a specialized police organization), all of which originally contained laptops. All test images were recorded with the machine evaluated in this study. The test images were created and recorded in collaboration with aviation security experts from a specialized police organization and former airport security screening officers now employed by CASRA. As explained above, in conditions A and C the laptops were taken out of the bags and recorded separately, whereas in conditions B and D the laptops were left inside the bags. Each bag/laptop-combination was used twice, once containing a threat item in either the bag or the laptop, and once without any threat item. The test contained a representative sample of threat items selected and developed (the IEDs in laptops) by experts from an airport police department. These could be divided into four different threat categories: guns, IEDs, knives and other threat items (e.g., electric shock devices, etc.). For all categories except guns, in half of the cases the threat items were placed in the bag, while in the other half of the cases the threat items were placed within the laptop (see **Figure 3**). Due to their size, it would not have been

realistic to place guns inside a laptop. Moreover, the factor viewpoint was included in the test design. For those threat items placed inside the bags, half were positioned in easy views and half in difficult views. Easy view means that threat items were depicted from a frontal/canonical view in the X-ray image, while for difficult view the threat items were horizontally or vertically rotated. All the threat items placed inside the laptop cases were positioned in easy views. As laptops are comparably flat, it would have been difficult to place threat items in vertically or horizontally rotated positions. The IEDs which were placed inside the laptops were specifically built into the cases. It must be considered that since an IED consists of several component parts, it becomes more difficult to determine what the canonical view and thus an easy view would be. Each threat category contained 24 items. Therefore, the number of test images for the conditions were the following:

	- 4 × 24 threat images (60 bags and 36 laptops) +96 non threat *BAG* images +96 non threat *LAPTOP* images = *288 test images*

4 × 24 combined threat images (60 threats in bags and 36 threats in laptops)

#### **PROCEDURE**

All participants were invited to the experimenters' facilities to conduct the test. Four computer workstations with the corresponding consoles of the tested machine and 19 TFT monitors were set up in a normally lit room. X-ray images covered about 2/3 of the computer screen. The distance to the monitor was ∼60 cm. Four participants at a time were tested. Before the test started, all participants received a short introduction by the test supervisor, explaining the test procedure and introducing the new technology of motion imaging. All participants were able to try out the console and view test images for ∼20 min, in order to become familiar with the images, the technology and the handling of the console. Pre-testing had shown that this amount of time was enough to get well acquainted with the console and it was also recommended by the manufacturer. After a break of 10 min the actual test started. Tests were conducted quietly and individually, and under supervision. The test images remained on the screen

2The set was based on a two-month data collection, assessing the contents and types of bags that were passing through the security checkpoints at an international European airport.

until the participant either pressed the "OK" or "NOT OK" and the "move belt forward" button. RTs were measured in milliseconds and correspond to the amount of time it took for a screening officer to come to a decision and press the "OK" or "NOT OK" button after the first image pixel of the bag/laptop appeared on the screen. There was no time limit set for viewing an image. However, participants were instructed to inspect the images as quickly and accurately as possible. Breaks of 10 min were taken in 30 minute-cycles, to avoid eyestrain and fatigue, and to make sure that especially those participants conducting tests A and C (288 images instead of 192 images, see section Image Interpretation Test) would not become too tired toward the end. All participants completed the test in less than 2 h, including breaks.

#### **RESULTS AND DISCUSSION**

According to signal detection theory (Green and Swets, 1966), there are four possible outcomes to a screener's response when judging an X-ray image as either OK or NOT OK: hit, falsealarm, correct rejection and miss (Schwaninger, 2003a; Hofer and Schwaninger, 2004). In this study, A was applied as a measure for detection performance (Pollack and Norman, 1964). A is a measure of sensitivity which is commonly used for a variety of tasks including screener certification and competency assessments (Hofer and Schwaninger, 2004; Koller and Schwaninger, 2006; Michel et al., 2010). It considers the hit rate as well as the false-alarm rate and can be calculated using the following formula (Grier, 1971):

$$0.5 + [(H - F)(1 + H - F)]/[4H(1 - F)]\tag{1}$$

$$0.5 + [(F - H)(1 + F - H)]/[4F(1 - H)]\tag{2}$$

*H* is the hit rate and *F* the false alarm rate. If performance is below chance, i.e., when *H < F*, equation (2) must be used (Aaronson and Watts, 1987).

Due to the security confidential nature of performance values, these are not displayed in this paper. In order to provide meaningful results, relative differences and effect sizes are reported. All reported effect sizes are interpreted based on Cohen (1988). For *t*-tests, *d* between 0.20 and 0.49 represents a small effect size; *d* between 0.50 and 0.79 represents a medium effect size; *d* ≥ 0*.*80 represent a large effect size. For analysis of variance (ANOVA) statistics, η2 between 0.01 and 0.05 represents a small effect size; η2 between 0.06 and 0.13 represents a medium effect size; η2 ≥ 0*.*14 represents a large effect size.

#### **COMPARISON OF DETECTION PERFORMANCE BY CONDITION**

**Figure 4** shows a comparison of detection performance scores by condition (A, B, C, and D)3 . Most remarkable seems to be

the effect of packing condition. Performance was much better in conditions A and C, where laptops and bags were screened separately, compared to conditions B and D, where laptops were left inside the passenger bags. The graph also suggests that performance was slightly better when motion imaging was available (condition C compared to A and condition D compared to C, respectively). The ANOVA with the between-participants factors display condition (no motion vs. motion) and packing condition (laptop separate vs. laptop in bag) revealed a large main effect for packing condition, *F(*1*,* 76*)* = 105*.*22, *p <* 0*.*001, η2 = 0*.*581, and a medium main effect for display condition, *F(*1*,* 76*)* = 5*.*05, *p <* 0*.*05, η2 = 0*.*062. There was no interaction between display and packing condition, *F(*1*,* 76*)* = 0*.*361, *p* = 0*.*55, η2 = 0*.*005. Thus, although motion imaging enhanced detection performance slightly, it could not compensate the negative effects on detection performance resulting from leaving laptops inside bags. Further, the direct comparison of condition D (motion imaging available, laptops in bags) and condition A (no motion imaging available, laptops and bags screened separately) revealed a highly significant effect, *t(*26*)* = 5*.*89, *p <* 0*.*001, *d* = 1*.*86. This also shows that although motion imaging did improve detection performance (as shown by the main effect in the ANOVA), the large negative effect of packing condition could not be compensated.

In sum, the results imply that the packing condition had a high impact on detection performance. Motion imaging resulted in better detection but could not fully compensate the effect of packing condition (i.e., impaired detection when leaving laptops in passenger bags). The large main effect for packing condition is consistent with the assumption that the well-documented effects of superposition and bag complexity (Schwaninger et al., 2005a,b, 2007; Hardmeier et al., 2005, 2006; Bolfing et al., 2008; von Bastian et al., 2008) increase when laptops are left in passenger bags, resulting in impairments of threat detection performance.

A more detailed analysis was conducted by looking at each threat category separately. As can be seen in **Figure 5**, large differences between conditions, but also between threat categories were

3Due to the packing condition, the proportion of target present and target absent trials differed for conditions A/C and B/D (in conditions A and C the ratio is 1:2; in conditions B and D the ratio is 1:1). According to signal detection theory (Green and Swets, 1966), different ratios of target present and target absent trials can result in a criterion shift (i.e., changes in hit and false alarm rates). Measures of detection performance in terms of sensitivity such as d' and A' are thought to be relatively independent of criterion shifts, which could also be shown in studies on target prevalence (e.g., Gur et al., 2003;

Wolfe et al., 2007; Wolfe and Van Wert, 2010). Therefore, it can be assumed that the different proportions on target present trials in conditions A/C and B/D did not affect detection performance (A ) results in this study.

**IEDs, knives, others).**

**Table 1 | Results of the ANOVAs conducted with detection performance (A- ) as dependent variable4.**

found. A mixed-design ANOVA with the within-participants factor threat category (guns, IEDs, knives, others) and the betweenparticipants factors display condition (no motion vs. motion) and packing condition (laptop separate vs. laptop in bag) revealed large significant main effects for the factors threat category and packing condition and a medium effect for display condition (for details, see **Table 1A**). The interaction between threat category and packing condition was also highly significant, implying that leaving laptops in passenger bags affected performance differently, depending on threat category. None of the other interactions reached statistical significance. As **Figure 5** indicates, IEDs and other threats were most difficult to detect, especially in conditions B and D. In general, a slight advantage of motion imaging could be observed (compare condition C to A, and D to B), which according to **Figure 5** was most evident for guns.

**Table 2 | Results of the two-tailed independent samples** *t***-tests comparing detection performance A between conditions A and D for each threat category (guns, knives, IEDs, others)5.**

Additionally, we conducted direct comparisons between conditions D (motion imaging available, laptops in bags) and A (no motion imaging available, laptops and bags screened separately) for each threat category, to further examine whether for certain threat types the negative effect on detection performance of leaving laptops in bags could be fully compensated by motion imaging. For all threat categories except guns, large significant differences were revealed (see **Table 2**). This further indicates that even though motion imaging did improve detection performance (as shown by the main effect in the ANOVA, see above), it could not compensate the large negative effect of packing condition. Only for the detection of guns, motion imaging seemed to have helped to compensate the negative effect of leaving laptops in bags (which could explain the marginally significant interaction (*p* = 0*.*09) between threat category and display condition in **Table 1A**).

As described earlier, half of the threat items were placed inside laptops and half were placed inside the bags (except for guns, which could not be place inside laptops). **Figure 6** displays how detection performance differed for each condition with regard to threat category and the placement of threat items. Again, threat items were detected better when the bags and laptops were screened separately (conditions A and C). Planned comparisons were conducted for each condition and threat category (except for guns, as all guns were placed inside the bags), to compare the differences between detection performance with regard to the placement of threats for each condition (see **Table 3**). Biggest differences were found for IEDs. For each condition, detection performance was worse when the IEDs were built into the laptops, compared to when they were placed inside the bags. However, while for conditions A and C detection performance was still relatively high, the scores achieved in conditions B and D were much lower for the IEDs within the laptops. For the threat categories knives and others, in most conditions detection performance was higher when these were placed inside the laptops. This could be explained by the fact that all threat items placed within the laptops were positioned in easy views (see Method section), and thus were easier to recognize. For IEDs, this effect was not observed. As the IEDs were specifically built into the laptops and since an IED consists of several component parts, it becomes more difficult to determine what actually the canonical/frontal view and thus an easy view would be (see Method section).

4In all analyses of variance in this study were Mauchly's test indicated that the assumption of sphericity had been violated, the degrees of freedom were corrected using Greenhouse-Geisser estimates of sphericity.

5In all *t*-tests of this study where Levene's test indicated unequal variances, degrees of freedom were adjusted using the default procedure in SPSS.

**Table 3 | Results of two-tailed paired samples** *t***-tests comparing the detection performance A' with regard to the placement of each threat item (in laptop vs. in bag) for each threat category (guns, IEDs, knives, others) and each condition (A–D).**

In order to examine the viewpoint effect and whether this effect was influenced by condition, detection performance scores of all conditions were compared, broken up by easy vs. difficult view (see **Figure 7**). Since all threat items placed inside the laptop cases were positioned in easy views, this analysis was only conducted for the threat items placed inside the bags. A mixed-design ANOVA with the within-participants factors view difficulty (easy vs. difficult view) and threat category (guns, IEDs, knives, others) and the between-participants factor condition (A, B, C, D) revealed large significant main effects for all three factors (see **Table 1B**). There was no significant interaction between view difficulty and condition, while all other interactions were significant. Therefore, a viewpoint effect could clearly be observed, which differed with regard to threat category. However, view difficulty was not significantly affected by condition. Interestingly, as **Figure 7** indicates, throughout all conditions guns, IEDs and knives were detected better when depicted in easy views, while for the category others this was the other way around. The category others

**(easy vs. difficult view) for each threat category (guns, IEDs, knives, others).** Only threat items which were placed inside the bags are included.

contained a very heterogeneous group of threat items (e.g., pepper spray, taser, throwing star, etc.). Hence, it could have been that the screening officers were more familiar with those threat items positioned in difficult views, and therefore recognized these more easily. As **Figure 7** further indicates, for the category guns, motion imaging seemed to have been of help to reduce the viewpoint effect (see conditions C and D). This is consistent with the results reported above (see **Table 2**) and makes sense if one takes into account that guns change their shape more drastically than other objects when rotated. Thus, motion imaging can be more effective for supporting the recognition of guns.

#### **COMPARISON OF REACTION TIMES BY CONDITION**

**Figure 8** shows the average reaction times (RTs, converted into seconds) for all conditions and threat categories. For all categories, a similar pattern can be observed: More time was needed in conditions B and D where laptops were left inside the bags. Most time was needed in condition D, where motion imaging was available. As **Figure 8** implies, remarkable differences can be observed between the threat categories and conditions. A repeated-measures ANOVA (see **Table 4**) revealed large significant main effects for the factors threat category (guns, IEDs, knives, others) and condition (A, B, C, D). The interaction between both factors was also significant, implying that the size of the differences in RTs between the conditions varied with regard to threat category. As displayed in **Figure 8**, all conditions achieved fastest RTs for the category guns, while longest RTs are clearly observed for the category IEDs.

To determine which condition actually took the longest time to complete the test all RTs for each security screener in each condition were summed and averaged across screening officers. **Figure 9** displays these results. As described in the Method section, for test conditions A and C where laptops and bags were displayed separately, 288 images were displayed. In test conditions B and D, 192 images were shown. Even though fewer images were viewed in condition D, compared to conditions A and C, altogether, more time was needed to inspect these test images. While conditions A, B, and C did not differ from each other significantly, large differences were observed between each of these three conditions with condition D (see **Table 5**). These results indicate that **knives, IEDs, others).**

**Table 4 | Results of the repeated-measures ANOVA conducted with reaction time (RT).**

even though fewer images had to be viewed when laptops were kept in passenger bags, altogether more time was needed to apply motion imaging and investigate these images thoroughly. Thus, while motion imaging provides a security advantage, it comes with a certain cost of efficiency.

#### **SUMMARY AND CONCLUSIONS**

The benefits of an X-ray machine featuring a new technology offering multiple views of X-ray images and motion imaging were evaluated and compared to single view imaging. In specific, it was investigated whether leaving laptops inside passenger bags resulted in a decrease of detection performance and whether such an effect could be compensated by motion imaging. The results revealed that threat detection performance was much better when laptops and bags were screened separately (see also Mendes et al., 2012). Leaving laptops inside passenger bags resulted in a clear decrease of threat detection performance, supporting the view that increases in superposition and bag complexity affect detection performance negatively (Schwaninger et al., 2005b, 2007; Bolfing et al., 2008). Motion imaging technology could slightly improve threat detection performance. Yet, it could not compensate the negative effect of leaving laptops inside bags. Highest detection performance was achieved when motion imaging was available and laptops and bags were screened separately.

More detailed analyses indicate that performance differed remarkably with regard to the different threat categories [guns, improvised explosive devices (IEDs), knives, others]. IEDs and the others threat category were most difficult to detect, especially when laptops were not removed from passenger bags.  Merely for the detection of guns, motion imaging seemed to be of substantial benefit. Further analyses regarding the placement of threat items (in bag vs. in laptop) indicated that IEDs were particularly

**FIGURE 9 | Sum of reaction times (s) averaged across participants with standard errors of the mean for all four conditions (A–D).**

**Table 5 | Results of pairwise comparisons with Bonferroni correction for the sums of reaction times (in seconds) of all four conditions (SPSS Bonferroni adjusted** *p***-values are quoted).**

difficult to detect when these were built into the laptop cases. Specifically when laptops were left inside the bags, threat detection performance was quite low compared to when the laptops were displayed separately. Thus, when no automatic explosives detection is available and laptops are not removed from passenger bags, the detection of explosives and bombs, in particular, is impaired. For the categories knives and others, detection performance was higher when these were placed inside the laptops. This could be due to the fact that—for practical reasons—all threat items placed inside the laptops had to be positioned in easy views (canonical views). In general, threat items depicted in more difficult views were harder to detect. These findings are consistent with previous research on viewpoint effects, which showed that recognition of items depicted in frontal/canonical view is easier (e.g., Michel et al., 2007; Bolfing et al., 2008; Koller et al., 2008). Only for the category others, this effect was the other way round. As the category others contained a very heterogeneous group of threat items, possibly screening officers were more familiar with the items positioned in difficult views and thus detected these better. Results also showed that in general more time was needed to inspect the images when laptops were left inside the bags. Longest RTs were found when laptops were not removed from bags and motion imaging was applied. Thus, providing additional views is paid for by increasing RT (see also von Bastian et al., 2008). Even though fewer images were viewed when laptops were left inside the passenger bags, altogether more time was needed to apply motion imaging and inspect these images properly. Keeping factors such as throughput and efficiency at security checkpoints in mind, screening time is an important point to consider.

Technology for security screening will constantly be developed further. Yet, the final decision on whether threat items are contained in luggage still rely on human operators, who inspect the luggage based on an image provided by a machine. The presented study underlines the importance of thoroughly evaluating any new technological features with regard to their added value provided to the screening officers, prior to implementing these in the airport environment. In this study, only a slight benefit of motion imaging technology was revealed. No real advantage could be observed for the detection of IEDs, while the results do suggest that for certain objects such as guns, the rotation and availability of different viewpoints through motion imaging could improve identification. As previous research has shown (e.g., Michel et al., 2007) guns change their shape more drastically than other objects when rotated. Thus, one could assume that motion imaging would possibly be more helpful also for the detection of other threat types if larger rotations and more views are available (or even fully rotatable 3D images, see below).

All in all, the detection of threat items in cabin baggage screening currently still seems more reliable when laptops are taken out of passenger bags. Therefore, the outcomes of this study underline the appropriateness and importance of current regulations

#### **REFERENCES**

specifying that portable computers should be removed from passenger bags for X-ray screening. However, this might be reconsidered if effective and efficient automatic threat detection is available, which is particularly important for IEDs (see e.g., Singh and Singh, 2003; Eilbert, 2009; Mery et al., 2013). Furthermore, if more rotation in depth would be available, higher benefits could possibly be expected, which is of particular importance regarding new technological developments such as computer tomography offering 3D views. Effects of superposition and viewpoint could be reduced further and RTs could be decreased if screening officers can directly navigate to their preferred view of a bag image. In combination with automated threat detection this could possibly result in substantially higher human-machine system performance (see e.g., Flitton et al., 2010; Megherbi et al., 2010). However, this would have to be examined in further studies.

#### **ACKNOWLEDGMENTS**

We are thankful to aviation security experts from the German Federal Police Technology Center for providing their valuable expertise and support for the creation and recording of x-ray images. We thank Zurich State Police, Airport Division, for providing screeners and supporting this study.

a laboratory environment1. *Radiology* 228, 10–14. doi: 10.1148/radiol.2281020709

viewpoint effects resulting from recurrent CBT of X-ray image interpretation. *J. Transp. Secur.* 1, 81–106. doi: 10.1007/s12198-007- 0006-4

computer-based training predict results in x-ray image interpretation tests?," in *Proceedings of the 44th Carnahan Conference on Security Technology* (San Jose, CA).

*Psychol. Gen*. 136, 623–638. doi: 10.1037/0096-34<IP_ADDRESS>3

**Conflict of Interest Statement:** The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

*Received: 31 May 2013; paper pending published: 22 July 2013; accepted: 19 September 2013; published online: 18 October 2013.*

*Citation: Mendes M, Schwaninger A and Michel S (2013) Can laptops be left inside passenger bags if motion imaging is used in X-ray security screening? Front. Hum. Neurosci. 7:654. doi: 10.3389/fnhum.2013.00654*

*This article was submitted to the journal Frontiers in Human Neuroscience.*

*Copyright © 2013 Mendes, Schwaninger and Michel. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.*

### Suspiciousness perception in dynamic scenes: a comparison of CCTV operators and novices

#### *Christina J. Howard1\*, Tom Troscianko2 , Iain D. Gilchrist 2, Ardhendu Behera3 and David C. Hogg3*

*1 Division of Psychology, Nottingham Trent University, Nottingham, UK*

*2 School of Experimental Psychology, University of Bristol, Bristol, UK*

*3 School of Computing, University of Leeds, Leeds, UK*

#### *Edited by:*

*Kenneth C. Scott-Brown, University of Abertay Dundee, UK*

#### *Reviewed by:*

*Anne Hillstrom, University of Portsmouth, UK Damien Litchfield, Edge Hill University, UK*

#### *\*Correspondence:*

*Christina J. Howard, Nottingham Trent University, Burton Street, Nottingham, NG1 4BU, UK e-mail<EMAIL_ADDRESS>
Perception of scenes has typically been investigated by using static or simplified visual displays. How attention is used to perceive and evaluate dynamic, realistic scenes is more poorly understood, in part due to the problem of comparing eye fixations to moving stimuli across observers. When the task and stimulus is common across observers, consistent fixation location can indicate that that region has high goal-based relevance. Here we investigated these issues when an observer has a specific, and naturalistic, task: closed-circuit television (CCTV) monitoring. We concurrently recorded eye movements and ratings of perceived suspiciousness as different observers watched the same set of clips from real CCTV footage. Trained CCTV operators showed greater consistency in fixation location and greater consistency in suspiciousness judgements than untrained observers. Training appears to increase between-operators consistency by learning "knowing what to look for" in these scenes. We used a novel "Dynamic Area of Focus (DAF)" analysis to show that in CCTV monitoring there is a temporal relationship between eye movements and subsequent manual responses, as we have previously found for a sports video watching task. For trained CCTV operators and for untrained observers, manual responses were most highly related to between-observer eye position spread when a temporal lag was introduced between the fixation and response data. Several hundred milliseconds after between-observer eye positions became most similar, observers tended to push the joystick to indicate perceived suspiciousness. Conversely, several hundred milliseconds after between-observer eye positions became dissimilar, observers tended to rate suspiciousness as low. These data provide further support for this DAF method as an important tool for examining goal-directed fixation behavior when the stimulus is a real moving image.

**Keywords: eye movements, scene perception, expertise, security and human factors, visual search**

#### **INTRODUCTION**

Studies of naturalistic task performance have used eye movements as a measure of attentional deployment (e.g., Land, 1999; Findlay and Gilchrist, 2003; Underwood et al., 2003). Here we measure eye movements to investigate such attentional deployment in the context of closed-circuit television (CCTV) monitoring. CCTV monitoring is both a good model task in which to study the deployment of goal directed attention more generally and an important task more specifically because of its increased deployment in security and policing.

Recent research has examined human performance in some aspects of CCTV monitoring. For example, Troscianko et al. (2004) showed that people were able to anticipate antisocial behavior in the near future from CCTV footage. Others have examined the limitations of the use of CCTV footage in identifying unfamiliar individuals, although face recognition appears to be surprisingly resistant to viewpoint changes or poor image quality when individuals are familiar to observers (Bruce et al., 2001). However, much less is known about the dynamic allocation of attention during CCTV monitoring. Stainer et al. (2011) showed that eye movements in multiscreen displays tend to fall near the centres of individual video screens in a multiscreen display, and suggested that the lack of scene continuity and spatial contiguity between individual screens causes each one to be treated as an independent stimulus. However, there appears to be some direct competition between screens: Howard et al. (2011) showed that eye movements are driven to a great extent by the relative suspiciousness of different concurrent video screens in the display. CCTV is clearly a very rich visual stimulus and results are now beginning to emerge on several of the many aspects of human interaction with these stimuli. However, we are not aware of any work that seeks to examine exactly how attention is used within a single screen during on-line monitoring and decision making about video events, and this is addressed by the current study.

Examining how people perceive CCTV footage is one example of the more general task of perception of moving scenes. Much research has been conducted into the question of how we perceive static scenes and in particular, how long it takes to extract different types of visual information from scenes. Strikingly, the general "gist" of a scene can be processed from extremely brief (less than a tenth of a second) displays (e.g., Rousselet et al., 2005) or from a single glance (Biederman et al., 1974; Fei-Fei et al., 2007). When making global property classifications of a scene (e.g., naturalness, openness) and basic level categorisations (e.g., ocean, mountain), observers can reach asymptote levels of performance in 100 ms (Greene and Oliva, 2009). Little is known, however, about the time course of the perception of dynamic scenes. Of course outside of the laboratory, visual stimuli are rarely static and so it is important to investigate the extent to which the work with static images generalises to moving scenes. We recently examined this issue (Howard et al., 2010) by asking observers to make a continuous semantic judgement about a video of a semiconstrained real-world scenario: a football match. We found that responses continuously lagged behind eye movement behavior by over a second, suggesting that evaluation of moving scenes proceeds relatively slowly.

As well as being a task involving perception of real moving scenes, the task of monitoring CCTV images typically requires observers to search for and assess locations in the scene of maximum perceived suspiciousness. In this sense, whilst this task is very different from traditional visual search, some comparisons can be made. The task here could be considered as a visual search task for an extremely high level semantic target which is visually unspecified and could therefore take many different visual forms. From the traditional visual search literature, although target templates with high specificity are optimal for guiding attention, search can be driven by imprecise target information such as target categories (Malcolm and Henderson, 2009; Schmidt and Zelinsky, 2009; Yang and Zelinsky, 2009). Consistent with this, observers can use flexible target templates for search that are tolerant to some changes in target appearance e.g., changes in scale and orientation (Bravo and Farid, 2009, 2012). However, much less is known about the extent to which attention can be guided by very high-level semantic interpretation of scenes.

Another aspect of performance we addressed in the experiments presented here is the effect of expertise since our observers were both trained CCTV operators and untrained undergraduate observers. Howard et al. (2010) found that expertise affected the pattern of eye movements and the relationship between eye movements and responses. Specifically, individuals with more experience watching football matches made eye movements to goal relevant areas of the scene earlier than non-experts, and were thus able to spend longer evaluating the scene before making their responses. This would suggest that expertise may affect eye movement behavior in this CCTV monitoring task in a similar way i.e., that CCTV operators will be more able to direct their eye movements towards goal relevant areas of the scene than untrained observers. Indeed in a meta-analysis of several hundred effect sizes of expertise on eye movement behavior, Gegenfurtner et al. (2011) recently reported robust effects of expertise acting to increase frequency of fixations on goal relevant information and to reduce latencies for first fixations on the these areas. Some have claimed that this attention to goal relevant information (and consequently, reduced attention to irrelevant information) underlies the effect of expertise in visual tasks (Haider and Frensch, 1996). This "knowing what to look for" is likely in the CCTV operators since they are familiar with environments shown in the CCTV footage and the likely types of suspicious behaviors that may occur. For this reason, we hypothesised that CCTV operators would be able to process the scenes more efficiently than untrained observers. Scene "gist" or general layout can be extracted very rapidly but more detailed processing of scene content can take many seconds (e.g., Tatler et al., 2003). Given that CCTV monitoring requires complex semantic evaluation of scenes, we reasoned that this task would be likely to incur slower processing times and therefore may be sensitive to the effects of expertise.

There is evidence from a range of visual tasks that expertise affects processing efficiency. For example, in visual search tasks, domain-relevant expertise appears to enable observers to process a wider portion of their visual field at a time during visual search tasks (Hershler and Hochstein, 2009) and expert chess players can process visual information from across the visual field rapidly and with few fixations (Reingold et al., 2001). Similar processing advantages are seen in more applied tasks: expertise in driving facilitates wider visual scanning of road scenes (Underwood et al., 2002) and expertise in musical sight reading fosters greater storage of visual information from fixations on musical text (Furneaux and Land, 1999). We therefore hypothesised that CCTV operators would be able to process the unfolding CCTV scenes more efficiently than untrained observers. We expected CCTV operators to display greater between-observer consistency of gaze locations since their attention should be more consistently drawn to "suspiciousness" rather than other aspects of the scene or events within them.

We present here a task in which the attentional deployment of both trained operators and untrained, naïve observers is measured through eye tracking, whilst monitoring a single scene for potentially suspicious events. In this task, manual responses take the form of pushing a joystick to reflect the current degree to which events in the scene are perceived to be suspicious. The CCTV monitoring task requires continuous appraisal of the semantic content of scenes, and the evaluation of the current intentions and behaviors of people displayed in the scene. We will show a surprising degree of between-observer consistency in eye gaze locations in the scene, particularly between the gaze locations of trained CCTV operators. We will also show that periods of particularly high between-observer consistency in gaze positions are correlated with ratings of perceived suspiciousness in the scene. We will show that durations of between-observer eye position convergence are related to judgments of higher suspiciousness in the scenes, and that CCTV operators show longer periods of eye position convergence than untrained observers.

The current experiment demonstrates that this Dynamic Area of Focus (DAF) method works for a CCTV task as it did for the football watching task (Howard et al., 2010) with a different video stimulus and a very different semantic evaluation task. The DAF method is again shown to be a powerful tool for examining continuous perceptions of dynamic scenes without the need to analyse the content of the videos nor to measure low level physical characteristics or salience of the stimuli.

#### **EXPERIMENT 1: UNTRAINED OBSERVERS**

A computer programme was written in C++ to display a series of one–minute video clips of real CCTV footage obtained from Manchester City Council. Observers viewed a total of 40 oneminute clips comprising four clips from each of 10 different CCTV cameras. Observers viewed the videos in four blocks of ten minutes with breaks in-between, and the order of the 40 clips was quasi-randomised. The 10 cameras were chosen to represent as wide a range as possible in terms of the visual characteristics of the scenes. The ten camera views were as follows: night-time view of a carpark, pedestrian crossing, shopping street underpass, cash point at junction, busy retail street, landscaped open area, pedestrianised street, entrance to nightclub at night, bus stops and city centre street at night.

Observers made a constant judgement about the current perceived level of suspicious events in the scene by moving a joystick. Joystick ratings were sampled at 100 Hz resulting in a series of data points for each one-minute video stimulus. Note that this is a continuous response to a continuous stimulus, and the response takes the form of a rating about the video stimulus. Observers' eye positions were recorded at 25 Hz throughout the task using the ASL Mobile Eye head-mounted eye-tracker and Eye Vision software.

Video clips measuring 27 degrees by 22 degrees of visual angle were projected in a dimly lit room against a white background using a Canon SX6 projector onto a screen at a distance of 1.6 m. Black chequerboard markers subtending 4 × 4 degrees were placed at each corner of the video display such that a computer algorithm could be used after data collection to stabilise eye position recording for changes in head position.

#### **OBSERVERS**

Observers were thirty three undergraduate and postgraduate students at Bristol University, all naïve as to the purpose of the experiment, four of whom were male and twenty-nine female. The mean age of observers was 20 years, ranging from 18 to 34 years. All had normal or corrected-to-normal vision.

#### **PROCEDURE**

Observers were given written instructions as follows. They were asked to watch several videos of "urban scenes" and to monitor them for any suspicious events which, if seen in real life, might cause them to alert relevant authorities. They were asked to move a joystick according to what they perceived as being the current level of suspicious behavior in the video. They were told that at all times, the joystick should reflect what they perceived as being the current level of suspicious behavior. For instance, if they thought that the video was showing something very suspicious, they were told to move the joystick fully forwards for the duration of the suspicious events. If they perceived that there was currently absolutely no suspicious behavior, they were told not to move the joystick at all. They were informed that they could push the joystick to any level in-between these two extremes and that we would record the position of the joystick throughout the experiment. The joystick was in part chosen as a method of collecting responses as in the real CCTV control room where our operators were employed, operators may use a joystick to control the level of zoom on a particular camera, pushing the joystick further to zoom further in and vice versa. Hence this manual response was compatible with behaviors that occur in real CCTV monitoring contexts. Observers were asked to keep their hand on the joystick at all times to minimise the impact of manual reaction times.

#### **EXPERIMENT 2: TRAINED CCTV OPERATORS**

The method for Experiment 2 was similar to that used in Experiment 1, apart from the following differences. Video clips measuring 22 degrees by 18 degrees were projected against a white background. Observers viewed a total of 80 one-minute clips comprising eight clips from each of 10 different CCTV cameras. The 10 cameras were the same as those used in Experiment 1, but using twice as many clips from each: an extra four clips were used from each camera in addition to those used in Experiment 1. Observers viewed the videos in eight blocks of ten minutes with breaks in-between, and the order of the 80 clips was quasirandomised. Observers completed the experiment in two sessions over the week long testing period.

#### **OBSERVERS**

Observers were eleven trained CCTV operators working in the Manchester City Council CCTV control room of whom two were female and nine were male. All were naïve as to the purpose of the experiment but were aware that we were investigating the way that operators carry out their job, and the things that they look for whilst monitoring CCTV. All had normal or corrected-to-normal vision. The mean age of observers was 37 years, ranging from 23 to 60 years.

The trained CCTV operators differed from the untrained observers in terms of expertise since they had received training in the task of CCTV monitoring and all were currently employed as CCTV operators in the Manchester control room at the time of testing. Operators ranged in their level of experience from around six months to many years' experience in the job. Different individuals undoubtedly had achieved differing levels of expertise in the task but on average these observers would certainly be more familiar with the task and the types of CCTV images used than the untrained observers.

#### **RESULTS**

#### **RATINGS OF PERCEIVED SUSPICIOUS EVENTS**

The mean suspiciousness rating across videos for the untrained observers was 0.397 and for the operators was 0.387 (0.417 for operators watching the 40 videos seen by both groups). There was no difference in the overall suspiciousness ratings given by the two groups (t(39) = 0.574, p = 0.57). The range of suspiciousness ratings across videos for the untrained observers was 2.769 and for the operators was 2.258 (2.313 for the operators when viewing those 40 videos also seen by the untrained observers). The untrained observers showed a greater range of ratings at each given time (between-observer variability in ratings) than the trained observers (t(39) = 3.16, p < 0.01). In other words, trained observers' ratings were more consistent with one another at any given time.

#### **CONSISTENCY OF EYE GAZE POSITION**

For each frame in each video stimulus, we calculated a measure of spread of eye positions. In an example frame, there will be one recorded eye position for each observer, each with a horizontal and vertical position. As a measure of spread in eye positions across observers at a particular time, we took the mean of the interquartile ranges of the horizontal and vertical eye positions. We used the interquartile range as a measure of variability to minimise the influence of position outliers. The subsequent "spread value" is a measure of the extent to which all observers were looking at the same part of the screen at the same time.

The mean spread measure, expressed as a fraction of the size of the display was 19.0% (SD = 3.0%) for the operators (18.8%, SD = 3.0%, for operators watching those 40 videos seen by both groups) and 22.0% (SD = 3.2%) for the untrained observers. Trained observers showed less eye position spread than untrained observers (t*(*39*)* = 6.07, p *<* 0.01) indicating that they were more likely to be looking at a similar point in the videos as one another at any particular time.

#### **DYNAMIC AREA OF FOCUS ANALYSIS: RELATIONSHIP BETWEEN RATINGS AND EYE GAZE POSITION**

The DAF analysis captures the relationship between momentby-moment eye movement behavior and judgements of a group of observers viewing the same dynamic stimulus. To perform this analysis, we calculated estimates of the temporal relationship between eye movement behavior and responses. For these and all subsequent analyses, we calculated normalised suspiciousness ratings as follows: we first calculated the total overall mean and standard deviation of suspiciousness ratings for each observer and used these to normalise each observer's data set. For each video stimulus, we then calculated for each frame, the median of the normalised ratings. We chose the median to minimise the effects of outliers in the data.

To calculate an estimate of the time lag between eye movements and responses, we performed correlations between eye position spread and these normalised manual responses. At each point in time for a particular video, there will be one value of between-observer eye position spread and one value of suspiciousness ratings across observers as defined above. Note that across each whole video, the manual responses and the eye movements are both time series data and hence do not represent a single point in time but rather a continuous stream of events that relate to the continuous video stimulus.

To test for a non-zero lag, we performed each correlation after artificially shifting the eye spread data forwards and backwards in time. For instance, to test for a 100 ms lag, we shifted the eye spread data 100 ms backwards in time relative to the response data and recalculated the correlation value. At the best estimate of the lag, this correlation should be maximally negative. The lag estimate is the estimated time delay between changes in eye movements spread and the manual responses associated with them. For example, a reduction in eye position spread might be associated with an increase in suspiciousness ratings a short while later, whilst an increase in eye position spread is likely to be associated with a decrease in suspiciousness ratings soon afterwards.

Missing data created by these artificial time shifts were replaced with the mean value of spread for that stimulus. We tested each lag moving in steps of 10 ms through the range of up to 10 seconds both forwards and backwards in time. The results of these lag analyses are shown in **Figures 1**,**2** below. Error bars were obtained by bootstrapping: we sampled observers (with replacement) to create bootstrapped "new" data sets and obtained the lag for each of these data sets. This bootstrapping cycle was repeated 50,000 times and the standard error of this set of lags was then calculated.

We reasoned that the spread values and ratings would be related to one another but not necessarily in a straightforward way. Two factors are likely to drive eye movements, namely goal relevance (in this case, suspiciousness) and also low-level image salience differences such as differences in brightness, colour and motion in the scene. In addition, there may be multiple areas of a scene for which either or both of these drivers attracts attention

at any time. The extent to which observers will tend to fixate the same areas of the screen as one another (producing low spread values) was considered an empirical question. However, changes in eye position spread that occur close in time to changes in suspiciousness ratings may reflect scene events that are goal relevant. Of course, this is not to say that goal relevant events may not also be accompanied by changes in low-level scene salience, but by looking for the antecedents of high and low suspiciousness ratings, we will identify the extent to which eye position spread changes are related to goal relevance. We reasoned that if goal relevance is reliably related to eye position spread, then there will be a negative (and lagged) relationship between eye position spread and suspiciousness ratings. Overall, those events judged to be suspicious will tend to be preceded by different observers looking in similar places, and conversely, that events judged not to be suspicious will tend to be preceded by different observers looking in dissimilar places to one another. Since we are using a suspiciousness judgement along a continuum of joystick positions and not a discrete

suspicious/not suspicious judgement, we must also consider that intermediate suspiciousness ratings will tend to be preceded by intermediate spread values, to an extent determined by how suspicious the scenes are judged to be. Changes in eye position spread driven only by salience (for example, everyone's eyes being drawn to a street light being

turned on) will not be accompanied by a change in suspiciousness rating, and hence can only serve to decrease the strength of the correlation. Similarly, if there are two or multiple events in different parts of a scene that appear suspicious at any given time, this would produce high eye position spread measures and high suspiciousness ratings, thus decreasing the strength of the negative correlation between eye spread and ratings.

For both groups of observers at the obtained lags, eye position spread was negatively correlated with response (E1: r = −0.10, p *<* 0.05, E2: r = −0.07, p *<* 0.05) and these correlations coefficients between videos were significantly more negative than zero (E1: t*(*39*)* = −2.84, p *<* 0.01, E2: t*(*79*)* = −3.19, p *<* 0.01). The magnitude of the lag was much greater for the trained than the untrained observers. For those 40 videos seen by both sets of observers, the trained observers also showed a negative correlation between eye position spread and responses (r = −0.07, p *<* 0.01). This correlation was maximally negative at a lag of 1130 ms and was significantly more negative than zero (t*(*39*)* = −2.56, p = 0.01). Although this lag value is shorter than that seen when the data is analysed for the whole set of 80 videos seen by trained participants, it is still substantially longer than the lag found for untrained participants.

#### **RELATIONSHIP BETWEEN RATINGS AND EYE GAZE CONVERGENCE**

For each frame in each video, we classed the spread measure as either "low" or "not low" using a threshold of one standard deviation below the overall mean spread value. We then calculated the durations of periods of time in which this thresholded spread value remained consistently "low" on subsequent frames. Overall for the untrained observers, the mean duration of these low spread (or equivalently, "convergence") periods was 128 ms. For the untrained observers, there was a significant correlation (see **Figure 3**) between the mean convergence duration for each one minute video stimulus and the mean suspiciousness rating given

**FIGURE 3 | The relationship between convergence period duration and ratings of perceived suspiciousness for untrained observers.**

to that video (r*(*39*)* = 0.509, p *<* 0.01). The data for one of the video stimuli was more than two standard deviations above the mean on both variables of mean convergence duration and mean suspiciousness rating, but the correlation remained significant even after excluding this data point (r*(*38*)* = 0.324, p = 0.044).

Overall for the trained CCTV operators, the mean duration of these low spread "convergence" periods was 151 ms. As shown in **Figure 4**, there was a significant correlation between the mean convergence duration and the mean suspiciousness rating for each one minute video stimulus (r*(*79*)* = 0.296, p *<* 0.01). For those 40 videos also seen by the untrained observers, there was also a significant correlation (r*(*39*)* = 0.356, p *<* 0.024).

There was no significant difference in the strength of correlations between trained and untrained observers (p *>* 0.05) for any of the correlations reported above (correlations between convergence duration and ratings). However, for the 40 videos seen by both sets of observers, mean convergence period durations were longer for the CCTV operators than the untrained observers (t*(*39*)* = 3.540, p *<* 0.01) indicating that CCTV operators were more likely to spend longer periods of time consistently looking at the same part of the screen as one another than was the case for the untrained observers.

#### **DISCUSSION**

For this complex task, the DAF analysis reveals a temporal relationship between eye movements and subsequent manual responses. These results indicate that as found previously for a sports monitoring task (Howard et al., 2010), this DAF method is a powerful one for examining eye movement behavior towards moving stimuli. The method circumvents the need to analyse events within the video, nor low-level physical properties of the stimulus in order to examine goal-directed attention.

We found a significant negative correlation between eye position spread data and manual responses. In other words, observers tended to push the joystick to indicate perceived suspiciousness at times shortly after between-observer eye position differences decreased, and tended to give low suspiciousness ratings shortly after eye position spread increased. The time lag between eye position spread changes and corresponding suspiciousness judgement responses is relatively long and in the order of hundreds of milliseconds to seconds. This lag between eye movements and corresponding suspiciousness responses was longer in CCTV operators than in untrained observers. CCTV operators show reduced between-observer eye position spread and longer periods of eye position convergence than untrained observers. They also showed a greater degree of between-observer consistency in terms of suspiciousness ratings. We also show a relationship between the mean durations of eye position convergence events and the suspiciousness assigned to different videos. As discussed above regarding the lag analysis between eye movement spread and responses, both salience and goal relevance are likely to drive eye movements to different extents depending on the nature of the events depicted at any given time. However, we find a relationship between eye position convergence duration and ratings. Hence, like eye position spread, eye position convergence duration appears to be a strong enough indicator of goal relevance to show up over and above any effects of salient but not task-irrelevant events or the effects of multiple simultaneous relevant events.

This task was simpler than that of monitoring many screens at once as is the case in real CCTV control rooms. However, for multiple screens, untrained observers are able to perform this task since their eye movements are driven by goal relevance (i.e., suspiciousness) to a much greater extent than they are influenced by low-level image properties (Howard et al., 2011). The current study shows that both trained and untrained observers are able to respond to a single screen in such a way that their eye movements are related to goal relevance. Therefore, it seems likely that trained CCTV operators would be able to perform multiple screen monitoring to the same level or to a superior extent than untrained individuals and this deserves future investigation. In real CCTV control rooms, operators will need to monitor very many screens at once for suspicious activity and this method provides a starting point for understanding such a complex task. One way in which this method captures some of the processes involved in real CCTV monitoring is the use of joystick pushing/pulling as the manual response since in the real control room a similar joystick is used for operators to zoom into areas of interest or suspicious activity. The use of real CCTV footage from the urban areas familiar to operators and a realistic suspiciousness judgement are also very close to the demands of real CCTV monitoring in the control room. For these reasons, there should be a good degree of generalisability from our findings here to real CCTV tasks in terms of the relationship between eye movements and manual responses.