Understanding 'error' in the forensic sciences: A primer.

This paper distils seven key lessons about 'error' from a collaborative webinar series between practitioners at Victoria Police Forensic Services Department and academics. It aims to provide the common understanding of error necessary to foster interdisciplinary dialogue, collaboration and research. The lessons underscore the inevitability, complexity and subjectivity of error, as well as opportunities for learning and growth. Ultimately, we argue that error can be a potent tool for continuous improvement and accountability, enhancing the reliability of forensic sciences and public trust. It is hoped the shared understanding provided by this paper will support future initiatives and funding for collaborative developments in this vital domain.

Speaker identification in courtroom contexts - Part III: Groups of collaborating listeners compared to forensic voice comparison based on automatic-speaker-recognition technology.

Expert testimony is only admissible in common-law systems if it will potentially assist the trier of fact. In order for a forensic-voice-comparison expert's testimony to assist a trier of fact, the expert's forensic voice comparison should be more accurate than the trier of fact's speaker identification. "Speaker identification in courtroom contexts - Part I" addressed the question of whether speaker identification by an individual lay listener (such as a judge) would be more or less accurate than the output of a forensic-voice-comparison system that is based on state-of-the-art automatic-speaker-recognition technology. The present paper addresses the question of whether speaker identification by a group of collaborating lay listeners (such as a jury) would be more or less accurate than the output of such a forensic-voice-comparison system. As members of collaborating groups, participants listen to pairs of recordings reflecting the conditions of the questioned- and known-speaker recordings in an actual case, confer, and make a probabilistic consensus judgement on each pair of recordings. The present paper also compares group-consensus responses with "wisdom of the crowd" which uses the average of the responses from multiple independent individual listeners.

Speaker identification in courtroom contexts - Part II: Investigation of bias in individual listeners' responses.

In "Speaker identification in courtroom contexts - Part I" individual listeners made speaker-identification judgements on pairs of recordings which reflected the conditions of the questioned-speaker and known-speaker recordings in a real case. The recording conditions were poor, and there was a mismatch between the questioned-speaker condition and the known-speaker condition. No contextual information that could potentially bias listeners' responses was included in the experiment condition - it was decontextualized with respect to case circumstances and with respect to other evidence that could be presented in the context of a case. Listeners' responses exhibited a bias in favour of the different-speaker hypothesis. It was hypothesized that the bias was due to the poor and mismatched recording conditions. The present research compares speaker-identification performance between: (1) listeners under the original Part I experiment condition, (2) listeners who were informed ahead of time that the recording conditions would make the recordings sound more different from one another than had they both been high-quality recordings, and (3) listeners who were presented with high-quality versions of the recordings. Under all experiment conditions, there was a substantial bias in favour of the different-speaker hypothesis. The bias in favour of the different-speaker hypothesis therefore appears not to be due to the poor and mismatched recording conditions.

Speaker identification in courtroom contexts - Part I: Individual listeners compared to forensic voice comparison based on automatic-speaker-recognition technology.

Expert testimony is only admissible in common law if it will potentially assist the trier of fact to make a decision that they would not be able to make unaided. The present paper addresses the question of whether speaker identification by an individual lay listener (such as a judge) would be more or less accurate than the output of a forensic-voice-comparison system that is based on state-of-the-art automatic-speaker-recognition technology. Listeners listen to and make probabilistic judgements on pairs of recordings reflecting the conditions of the questioned- and known-speaker recordings in an actual case. Reflecting different courtroom contexts, listeners with different language backgrounds are tested: Some are familiar with the language and accent spoken, some are familiar with the language but less familiar with the accent, and others are less familiar with the language. Also reflecting different courtroom contexts: In one condition listeners make judgements based only on listening, and in another condition listeners make judgements based on both listening to the recordings and considering the likelihood-ratio values output by the forensic-voice-comparison system.

Reliability and validity of a quality tool for assessing clinical forensic medicine legal reports.

OBJECTIVE: It is essential that reports written by forensic medicine practitioners undergo appropriate quality control. The aim of this study is to develop and validate a tool for assessing the quality of medico-legal reports in adult and adolescent sexual assault cases. METHODS: The authors developed an audit tool and accompanying guideline aimed at detecting errors, omissions, and inadequacies in medico-legal reports following reported adult or adolescent sexual assault. The authors conducted a benchmarking exercise to reach an agreed audit standard. Subsequently two forensic examiners audited 5 legal reports, first without the tool, and then with the tool following standardised instruction. A further ten forensic examiners audited 20 reports after receiving instruction in use of the tool. Their results were compared to the agreed audit standard. Participants were interviewed about their experiences. RESULTS: Use of the tool to audit reports significantly increased the sensitivity of error detection compared to usual practice (sensitivity with usual practice 55% vs sensitivity with tool 80%). Study participants using the tool to audit 20 reports detected 73% of items designated as errors by the study authors. The overall accuracy in coding items as errors/not errors was 74%. Interrater reliability was good (Cronbach's alpha = 0.87). Sensitivity, overall accuracy, and interrater reliability results varied by category in the audit: reviewers had lower levels of error detection and lower levels of agreement when auditing opinions in the report as compared to items describing the clinical forensic evaluation. Participants had fundamental disagreements about what constitutes good quality in some aspects of a report, including: the acceptability of including 'non-relevant' history; whether to include references and, if so, what constitutes appropriate citations; and how to determine whether the opinions could be understood by a layperson. Study participants reported that using the audit tool and guideline to review medico-legal reports has merit and suggested improvements to increase usability. CONCLUSION: Use of an audit tool supported by a guideline and training is useful for improving error detection and standardising the review process for clinical forensic medicine legal reports. Further research aimed at improving the consensus about opinion standards in adult and adolescent sexual assault cases would be valuable.

Latent justice? A review of adversarial challenges to fingerprint evidence.

This article provides an overview of recent research on latent fingerprint evidence featured in reported legal decisions from England and Wales, Australia and New Zealand. The research casts doubts on the effectiveness of adversarial criminal procedure. Rather, than engage with the methodological foundations - e.g. validity and reliability - and the actual abilities of fingerprint examiners, for more than a century, challenges were based on legal considerations and the meaning of categorical identification for the specific proceedings. Lawyers challenged fingerprint evidence based on the circumstances in which reference prints were collected, whether fingerprint records were hearsay, whether relying on a fingerprint record is unfair because it suggests prior criminality, whether the jurors could make their own comparison and so forth. There is no reported consideration of the validity and reliability of fingerprint comparison, and no requirement for fingerprint examiners to qualify the significance of a match decision, even after the abandonment of point standards and the appearance of critical reports from the United States and Scotland, and advice from the Forensic Science Regulator. To the extent that they considered the admissibility and probative value of this prominent forensic science evidence, lawyers and judges relied heavily on proxies such as training, experience and long use. In consequence, the article considers how we should understand adversarial legal practice, the performance of lawyers and judges, as well as the implications for forensic scientists and their evidence.

Public attitudes towards the use of automatic facial recognition technology in criminal justice systems around the world.

Automatic facial recognition technology (AFR) is increasingly used in criminal justice systems around the world, yet to date there has not been an international survey of public attitudes toward its use. In Study 1, we ran focus groups in the UK, Australia and China (countries at different stages of adopting AFR) and in Study 2 we collected data from over 3,000 participants in the UK, Australia and the USA using a questionnaire investigating attitudes towards AFR use in criminal justice systems. Our results showed that although overall participants were aligned in their attitudes and reasoning behind them, there were some key differences across countries. People in the USA were more accepting of tracking citizens, more accepting of private companies' use of AFR, and less trusting of the police using AFR than people in the UK and Australia. Our results showed that support for the use of AFR depends greatly on what the technology is used for and who it is used by. We recommend vendors and users do more to explain AFR use, including details around accuracy and data protection. We also recommend that governments should set legal boundaries around the use of AFR in investigative and criminal justice settings.

Reply to Response to Vacuous standards - Subversion of the OSAC standards-development process.

This Letter to the Editor is a reply to Mohammed et al. (2021) https://doi.org/10.1016/j.fsisyn.2021.100145, which in turn is a response to Morrison et al. (2020) "Vacuous standards - subversion of the OSAC standards-development process" https://doi.org/10.1016/j.fsisyn.2020.06.005.

Communicating forensic science opinion: An examination of expert reporting practices.

Forensic scientists endeavour to explain complex scientific principles to legal decision-makers with limited scientific training (e.g., police, lawyers, judges, and jurors). Much of the time this communication is limited to written opinions in expert reports. Notwithstanding considerable scientific research and debate about the best way to communicate forensic science opinions, it is unclear how much of the advice has translated into forensic science practice. In conducting this descriptive study, we examined the reporting practices adopted by forensic scientists across a range of forensic science disciplines. Specifically, we used a quantitative content analysis approach to identify the conclusion types and additional information submitted by forensic scientists in proficiency tests during 2016 ("What would be the wording of the Conclusions in your report?"). Our analysis of 500 randomly selected responses in eight disciplines indicated that the conclusion type which has received the most criticism in recent years (categorical statements) remains the preferred means of expression in a clear majority of responses. We also found that the provision of additional information often considered necessary for rational evaluation of the evidence (e.g., information about reliability and validity) was rarely reported. These results suggest limited engagement with recent recommendations and are concerning given the gravity of the legal decisions that hinge on accurate and transparent forensic science communication.

Forensic science needs registered reports.

The registered report (RR) format is rapidly being adopted by scientific researchers and journals. RRs flip the peer review process, with reviewers evaluating proposed methods, rather than the data and findings. Editors then accept or reject articles largely based on the pre-data collection review. Accordingly, RRs reduce the incentive for researchers to exaggerate their findings, and they make any data-driven changes to the methods and analysis more conspicuous. They also reduce publication bias, ensuring studies with null or otherwise unfavorable results are published. RRs are being used in many fields to improve research practices and increase confidence in study findings. The authors suggest RRs ought to be the default way in which validation studies are conducted and reported in the forensic sciences. They produce more reliable findings, advance criminal justice values, and will lead to several efficiencies in the research process.

Forensic science evidence: Naive estimates of false positive error rates and reliability.

We do not know how often false positive reports are made in a range of forensic science disciplines. In the absence of this information it is important to understand the naive beliefs held by potential jurors about forensic science evidence reliability. It is these beliefs that will shape evaluations at trial. This descriptive study adds to our knowledge about naive beliefs by: (1) measuring jury-eligible (lay) perceptions of reliability for the largest range of forensic science disciplines to date, over three waves of data collection between 2011 and 2016 (n=674); (2) calibrating reliability ratings with false positive report estimates; and (3) comparing lay reliability estimates with those of an opportunity sample of forensic practitioners (n=53). Overall the data suggest that both jury-eligible participants and practitioners consider forensic evidence highly reliable. When compared to best or plausible estimates of reliability and error in the forensic sciences these views appear to overestimate reliability and underestimate the frequency of false positive errors. This result highlights the importance of collecting and disseminating empirically derived estimates of false positive error rates to ensure that practitioners and potential jurors have a realistic impression of the value of forensic science evidence.

Peer review in forensic science.

Peer review features prominently in the forensic sciences. Drawing on recent research and studies, this article examines different types of peer review, specifically: editorial peer review; peer review by the scientific community; technical and administrative review; and verification (and replication). The article reviews the different meanings of these quite disparate activities and their utility in relation to enhancing performance and reducing error. It explains how forensic practitioners should approach and use peer review, as well as how it should be described in expert reports and oral testimony. While peer review has considerable potential, and is a key component of modern quality management systems, its actual value in most forensic science settings has yet to be determined. In consequence, forensic practitioners should reflect on why they use specific review procedures and endeavour to make their actual practices and their potential value transparent to consumers; whether investigators, lawyers, jurors or judges. Claims that review increases the validity of a scientific technique or accuracy of opinions within a particular case should be avoided until empirical evidence is available to support such assertions.

Thinking forensics: Cognitive science for forensic practitioners.

Human factors and their implications for forensic science have attracted increasing levels of interest across criminal justice communities in recent years. Initial interest centred on cognitive biases, but has since expanded such that knowledge from psychology and cognitive science is slowly infiltrating forensic practices more broadly. This article highlights a series of important findings and insights of relevance to forensic practitioners. These include research on human perception, memory, context information, expertise, decision-making, communication, experience, verification, confidence, and feedback. The aim of this article is to sensitise forensic practitioners (and lawyers and judges) to a range of potentially significant issues, and encourage them to engage with research in these domains so that they may adapt procedures to improve performance, mitigate risks and reduce errors. Doing so will reduce the divide between forensic practitioners and research scientists as well as improve the value and utility of forensic science evidence.

On the likelihood of "encapsulating all uncertainty".

The assignment of personal probabilities to form a forensic practitioner's likelihood ratio is a mental operation subject to all the frailties of human memory, perception and judgment. While we agree that beliefs expressed as coherent probabilities are neither 'right' nor 'wrong' we argue that debate over this fact obscures both the requirement for and consideration of the 'helpfulness' of practitioner's opinions. We also question the extent to which a likelihood ratio based on personal probabilities can realistically be expected to 'encapsulate all uncertainty'. Courts cannot rigorously assess a forensic practitioner's bare assertions of belief regarding evidential strength. At a minimum, information regarding the uncertainty both within and between the opinions of practitioners is required.

Just truth? Carefully applying history, philosophy and sociology of science to the forensic use of CCTV images.

Using as a case study the forensic comparison of images for purposes of identification, this essay considers how the history, philosophy and sociology of science might help courts to improve their responses to scientific and technical forms of expert opinion evidence in ways that are more consistent with legal system goals and values. It places an emphasis on the need for more sophisticated models of science and expertise that are capable of helping judges to identify sufficiently reliable types of expert evidence and to reflexively incorporate the weakness of trial safeguards and personnel into their admissibility decision making.