Combined interpretation of objective firearm evidence comparison algorithms using Bayesian networks.

Traditionally, firearm and toolmark examiners manually evaluate the similarity of features on two bullets using comparison microscopy. Advances in microscopy have made it possible to collect 3D topographic data, and several automated comparison algorithms have been introduced for the comparison of bullet striae using these data. In this study, open-source approaches for cross-correlation, congruent matching profile segments, consecutive matching striations, and a random forest model were evaluated. A statistical characterization of these automated approaches was performed using four datasets of consecutively manufactured firearms to provide a challenging comparison scenario. Each automated approach was applied to all samples in a pairwise fashion, and classification performance was compared. Based on these findings, a Bayesian network was empirically learned and constructed to leverage the strengths of each individual approach, model the relationship between the automated results, and combine them into a posterior probability for the given comparison. The network was evaluated similarly to the automated approaches, and the results were compared. The developed Bayesian network classified 99.6% of the samples correctly, and the resultant probability distributions were significantly separated more so than the automated approaches when used in isolation.

The inconclusive category, entropy, and forensic firearm identification.

There has been extensive recent discussion of the difficulty in estimating meaningful error rates in forensic firearms examinations, and other areas of pattern evidence. The 2016 President's Council of Advisors on Science and Technology (PCAST) report was clear in criticizing many forensic disciplines as lacking the types of studies that would provide error rate measurements seen in other scientific fields. However, there is a substantial lack of consensus on the approach to measuring an "error rate" for fields such as forensic firearm examination that include in the conclusion scale the "inconclusive" category, as occurs in the Association of Firearm and Tool Mark Examiners (AFTE) Range of Conclusions and many other such fields. Many authors appear to assume the error rate calculated in the binary decision model is the only appropriate way to report errors, but there have been attempts made to adapt the error rate from the binary decision model to scientific fields in which the inconclusive category is viewed as a meaningful outcome of the examination process. In this study we present three neural networks of differing complexity and performance trained to classify the outlines of ejector marks on cartridge cases fired from different firearm models, as a model system for examining the performance of various metrics of error in systems using the inconclusive category. We also discuss an entropy, or information, based method to assess the similarity of classifications to ground truth that is applicable to range of conclusion scales, even when the inconclusive category is used.

Firearm examination: Examiner judgments and computer-based comparisons.

Forensic firearm examination provides the court of law with information about the source of fired cartridge cases. We assessed the validity of source decisions of a computer-based method and of 73 firearm examiners who compared breechface and firing pin impressions of 48 comparison sets. We also compared the computer-based method's comparison scores with the examiners' degree-of-support judgments and assessed the validity of the latter. The true-positive rate (sensitivity) and true-negative rate (specificity) of the computer-based method (for the comparison of both the breechface and firing pin impressions) were 94.4% and at least 91.7%, respectively. For the examiners, the true-positive rate was at least 95.3% and the true-negative rate was at least 86.2%. The validity of the source decisions improved when the evaluations of breechface and firing pin impressions were combined and for the examiners also when the perceived difficulty of the comparison decreased. The examiners were reluctant to provide source decisions for "difficult" comparisons even though their source decisions were mostly correct. The correlation between the computer-based method's comparison scores and the examiners' degree-of-support judgments was low for the same-source comparisons to negligible for the different-source comparisons. Combining the outcomes of computer-based methods with the judgments of examiners could increase the validity of firearm examinations. The examiners' numerical degree-of-support judgments for their source decisions were not well-calibrated and showed clear signs of overconfidence. We suggest studying the merits of performance feedback to calibrate these judgments.

Cognitive biases in the peer review of bullet and cartridge case comparison casework: A field study.

OBJECTIVE: Forensic judgments and their peer review are often the result of human assessment and are thus subjective and prone to bias. This study examined whether bias affects forensic peer review. HYPOTHESES: We hypothesized that the probability of disagreement between two forensic examiners about the proposed conclusion would be higher with "blind" peer review (reviewer saw only the first examiner's comparison photos) than with "non-blind" peer review (reviewer also saw the first examiner's interpretation and proposed conclusion). We also hypothesized that examiners with a higher perceived professional status would have a larger effect on the reported conclusion than examiners with a lower status. METHOD: We acquired data during a non-blind and a blind peer review procedure in a naturalistic, covert study with eight examiners (3-26 years of experience). We acquired 97 conclusions of bullet and cartridge case comparisons in the blind and 471 in the non-blind peer review procedure. RESULTS: The odds of disagreement between examiners about the evidential strength of a comparison were approximately five times larger (95%-CI [3.06, 8.50]) in the blind than in the non-blind procedure, with disagreement about 42.3% and 12.5% of the proposed conclusions, respectively. Also, the odds that their proposed conclusion was reported as the final conclusion were approximately 2.5 higher for the higher-status examiners than for lower-status examiners. CONCLUSIONS: Our results support both the hypothesis that bias occurs during non-blind forensic peer review and the hypothesis that higher-status examiners determine the outcome of a discussion more than lower-status examiners. We conclude that blind peer review may reduce the probability of bias and that status effects have an impact on the peer reviewing process.

Validity and reliability of forensic firearm examiners.

Forensic firearm examiners compare the features in cartridge cases to provide a judgment addressing the question about their source: do they originate from one and the same or from two different firearms? In this article, the validity and reliability of these judgments is studied and compared to the outcomes of a computer-based method. The features we looked at were the striation patterns of the firing pin aperture shear marks of four hundred test shots from two hundred Glock pistols, which were compared by a computer-based method. Sixty of the resulting 79,800 comparisons were shown to 77 firearm examiners. They were asked to judge whether the cartridge cases had the same source or a different source, and to indicate the degree of support the evidence provided for those judgments. The results show that the true positive rates (sensitivity) and the true negative rates (specificity) of firearm examiners are quite high. The examiners seem to be slightly less proficient at identifying same-source comparisons correctly, while they outperform the used computer-based method at identifying different-source comparisons. The judged degrees of support by examiners who report likelihood ratios are not well-calibrated. The examiners are overconfident, giving judgments of evidential strength that are too high. The judgments of the examiners and the outcomes of the computer-based method are only moderately correlated. We suggest to implement performance feedback to reduce overconfidence, to improve the calibration of degree of support judgments, and to study the possibility of combining the judgments of examiners and the outcomes of computer-based methods to increase the overall validity.

Performance Evaluation of a Registered Ballistic Database Using the Evofinder® System.

Firearms for police in China are registered along with their fired bullets and cartridge cases. A Registered Ballistic Database (RBD) of 1000 Norinco QSZ-92 pistols with registered ammunition was established and was evaluated through the Evofinder® system. In this research, 1000 bullets and 1000 cartridge cases were randomly selected and correlated against an RBD of 2996 bullets and 2999 cartridge cases. Examiners found that successful identifications all ranked 1st, supported with land (100%), groove (97%) engraved areas, and primary marks (85.6%) for bullets, and firing pin impressions (99.8%), and breech face marks (99.9%) for cartridge cases. Two known matches (KM) for the same pistol rank in the top two (100%). The distribution of similarity scores varies from marks; however, the Evofinder® system could still effectively distinguish known matches from known nonmatches (KNM) for either bullets or cartridge cases. This study demonstrates the efficiency of the RBD.