From data to a validated score-based LR system: A practitioner's guide.

Likelihood ratios (LRs) are a useful measure of evidential strength. In forensic casework consisting of a flow of cases with essentially the same question and the same analysis method, it is feasible to construct an 'LR system', that is, an automated procedure that has the observations as input and an LR as output. This paper is aimed at practitioners interested in building their own LR systems. It gives an overview of the different steps needed to get to a validated LR system from data. The paper is accompanied by a notebook that illustrates each step with an example using glass data. The notebook introduces open-source software in Python constructed by NFI (Netherlands Forensic Institute) data scientists and statisticians.

Measuring the rarity of core-delta distances in fingerprint patterns in the Dutch population.

In latent print examination, a conclusion on an individualization is primarily based on the presence of corresponding minutiae. Depending on the level of correspondence between the minutiae on the fingermark and on the reference print, a match is concluded. However, a fingermark is analyzed on all three levels: general pattern combined with core-delta distance, minutiae, and third-level detail. In the opinion of the authors, next to the conclusion based on minutiae, all information should be included in the final assessment of the evidence if present. In this study, we present the results of frequency data on the combination of fingerprint patterns and their core-delta distances. The frequency data allow examiners to substantiate their subjective opinions on the strength of corresponding patterns and core-delta distances based on measurements. The frequencies of the core-delta distances in this study were obtained by encoding core-delta distances in a collection of more than 20,000 fingerprints containing either a loop or a whorl pattern. The frequencies for the core-delta distances of loop patterns were subsequently compared with data from the National Institute of Standards and Technology. This study will demonstrate that core-delta distances in loop and whorl patterns contribute significantly to the overall evidential strength of fingerprint evidence. By adding pattern and core-delta distance information to the evaluation of fingerprint evidence, more challenging fingermarks could be added to fingerprint case reports.

Automated interpretation of comparison scores for firearm toolmarks on cartridge case primers.

An automated approach for evaluating the strength of the evidence of firearm toolmark comparison results is presented for a common source scenario. First, comparison scores are derived describing the similarity of marks typically encountered on the primer of fired cartridge cases: aperture shear striations as well as breechface and firing pin impressions. Subsequently, these scores are interpreted using reference distributions of comparison scores obtained for representative known matching (KM) and known non-matching (KNM) ballistic samples in a common source, score-based likelihood ratio (LR) system. We study various alternatives to set up such an LR system and compare them using qualitative and quantitative criteria known from the literature. As an example, results are applied to establish a system suitable for a firearm-ammunition combination often encountered in casework: Glock firearms with Fiocchi nickel primer ammunition. The system outputs an LR and a measure of LR uncertainty. The range of possible LR-values is limited to a minimum and maximum value in areas of the score domain with little reference data. Finally, the feasibility of combining LRs of different mark types into one LR for the entire primer is assessed. For the distribution models considered in this paper, different modeling approaches are optimal for different types of similarity scores. For the chosen firearm-ammunition combination, non-parametric Kernel Density Estimation (KDE) models perform best for similarity scores based on the correlation coefficient, whereas parametric models perform best for the Congruent Matching Cells (CMC) scores, assuming binomial and beta-binomial models for KM and KNM score distributions respectively. Finally, it is demonstrated that individual LRs of different mark types can be combined into one LR, to interpret a set of different marks on the primer as a whole.

The evidential strength of a combination of corresponding class features in tire examination.

Tire marks are an important type of forensic evidence as they are frequently encountered at crime scenes. When the tires of a suspect's car are compared, the evidence can be very strong if so-called 'acquired features' are observed to correspond. When only 'class characteristics' such as parts of the tire pattern are observed to correspond, it is obvious that many other tires will exist that also correspond, and so this evidence is usually considered very weak or is simply ignored. Like Benedict et al. (2014) we argue that such evidence can still be strong and should be taken into account. We describe a method for assessing the evidential strength of a set of corresponding class characteristics by presenting a case example from the Netherlands in which tire marks were obtained. Only part of two different tire patterns were visible, in combination with measurements on the axes width. Suitable databases were found already existing and accessible to forensic experts. We show how such data can be used to quantify the strength of evidence and how it can be reported. We also show how the risk of bias due to information surrounding the case may be minimized in cases like this. Our 'blind' procedure enables the expert to report a correspondence between class features in a more convincing way than standard procedures allow. In the particular exemplar case quite strong evidence was obtained, which was accepted and used by the Dutch court. We generalize this procedure for quantifying the evidential value of an expert's opinion of a correspondence. This examination procedure can be applied directly to other types of pattern evidence such as shoeprints, fingerprints, or images. Furthermore, it is 'blind' in the sense that the risk of contextual bias is minimized.

Why calibrating LR-systems is best practice. A reaction to "The evaluation of evidence for microspectrophotometry data using functional data analysis", in FSI 305.

In their paper "The evaluation of evidence for microspectrophotometry data using functional data analysis", in FSI 305, Aitken et al. present a likelihood-ratio (LR) system for their data. We show the values generated by this system cannot be interpreted as LRs: they are ill-calibrated and should be interpreted as discriminating scores. We demonstrate how to transform the scores to well-calibrated LRs using a post-hoc calibrating step. Also, we address criticisms of calibration posited by Aitken et al. We conclude by noting that ill-calibrated LR-values are misleadingly small or large. Therefore calibration should be measured and, if necessary, corrected for. The corrected LR-values (instead of the discriminating scores) can be used to update the prior odds in Bayes rule.

The influence of motion artefacts on magnetic resonance imaging of the clavicles for age estimation.

PURPOSE: To determine how motion affects stage allocation to the clavicle's sternal end on MRI. MATERIALS AND METHODS: Eighteen volunteers (9 females, 9 males) between 14 and 30 years old were prospectively scanned with 3-T MRI. One resting-state scan was followed by five intentional motion scans. Additionally, a control group of 72 resting-state scans were selected from previous research. Firstly, six observers allocated developmental stages to the clavicles independently. Secondly, they re-assessed the images, allocating developmental statuses (immature, mature). Finally, the resting-state scans of the 18 volunteers were assessed in consensus to decide on the "correct" stage/status. Results were compared between groups (control, prospective resting state, prospective motion), and between staging techniques (stages/statuses). RESULTS: Inter-observer agreement was low (Krippendorff α 0.23-0.67). The proportion of correctly allocated stages (64%) was lower than correctly allocated statuses (83%). Overall, intentional motion resulted in fewer assessable images and less images of sufficient evidential value. The proportion of correctly allocated stages did not differ between resting-state (64%) and motion scans (65%), while correctly allocated statuses were more prevalent in resting-state scans (83% versus 77%). Remarkably, motion scans did not render a systematically higher or lower stage/status, compared to the consensus. CONCLUSION: Intentional motion impedes clavicle MRI for age estimation. Still, in case of obvious disturbances, the forensic expert will consider the MRI unsuitable as evidence. Thus, the development of the clavicle as such and the staging technique seem to play a more important role in allocating a faulty stage for age estimation.

Influence of Muzzle Instability on Bullet Deflection after Perforating Laminated Particleboards.

The influence of muzzle instability on bullet deflection, when perforating laminated particleboards, was studied with three different handgun bullet types. The mean deflection angles of .32 Auto FMJ and .38 Special SJHP bullets were calculated to be 0.90° and 0.83°, respectively after perforating particleboards orthogonally at a 1 m muzzle-to-target distance. The mean angles dropped to 0.70° and 0.58° at a 15 m muzzle-to-target distance. The differences in deflection angles proved to be statistically significant (p < 0.05) with p-values of 0.023 and 0.001, respectively. The mean calculated deflection angles of .38 Special LRN bullets also dropped from 1.51° to 1.38° when the muzzle-to-target distance was increased from 1 to 15 m, but this difference was not significant (p-value of 0.357). The results support the hypothesis that muzzle instability has an influence on deflection. The possible implications for shooting incident reconstructions and for future research are discussed.

Measuring the Rarity of Fingerprint Patterns in the Dutch Population Using an Extended Classification Set.

Latent print examiners often use their experience and knowledge to reach a conclusion on the identity of the source. Their conclusion is primarily based on their personal opinion on the rarity of the matching fingerprint features. Fingerprint patterns, if present, can play a significant role in the final assessment of a match. The authors believe that statistical data on the rarity of fingerprint patterns strengthens the subjective evaluation of the corresponding information. In order to provide fingerprint examiners with additional numerical support, fingerprint patterns were manually classified in a set of 24,104 fingerprints. In this study the frequencies of occurrence of 35 different fingerprint patterns have been obtained. The frequency data presented in this study can be used in the ACE-V process applied in forensic casework, allowing for the assessment of the evidential strength related to a specific fingerprint pattern type.

Probability intervals of speed estimations from video images: The Markov Chain Monte Carlo approach.

Closed Circuit TV (CCTV) systems often record vehicle motion prior to incidents. From the footage an estimate of the average speed of the vehicle between two frames can be calculated. In a forensic investigation not only an estimate of the average speed is needed but also an estimation of the measurement error. In earlier papers an approach was explained how to estimate the average speed and the corresponding uncertainty in terms of a confidence interval. In practice confidence intervals are often wrongly interpreted as being probability intervals. In this paper we show how to use the Markov Chain Monte Carlo approach to derive probability intervals instead of confidence intervals. We show the robustness of the Markov Chain Monte Carlo approach and the numerical differences between both approaches. In casework the difference between confidence intervals and probability intervals turns out to be very limited. As a consequence the impact of confusion between confidence and probability intervals can also be expected to be limited.

Magazine Influence on Cartridge Case Ejection Patterns with Glock Pistols.

In this study, the cartridge case ejection patterns of six different Glock model pistols (one specimen per model) were compared under three conditions: firing with a loaded magazine, an empty magazine, and without magazine. The distances, covered by the ejected cartridge cases given these three conditions, were compared for each of the six models. A significant difference was found between the groups of data for each of the tested specimens. This indicates that it is important that, to reconstruct a shooting scene incident based on the ejection patterns of a pistol, test shots are fired with the same pistol type and under the correct magazine condition.

Implementation and assessment of a likelihood ratio approach for the evaluation of LA-ICP-MS evidence in forensic glass analysis.

For the comparative analysis of glass fragments, a method using Laser Ablation Inductively Coupled Plasma Mass Spectrometry (LA-ICP-MS) is in use at the NFI, giving measurements of the concentration of 18 elements. An important question is how to evaluate the results as evidence that a glass sample originates from a known glass source or from an arbitrary different glass source. One approach is the use of matching criteria e.g. based on a t-test or overlap of confidence intervals. An important drawback of this method is the fact that the rarity of the glass composition is not taken into account. A similar match can have widely different evidential values. In addition the use of fixed matching criteria can give rise to a "fall off the cliff" effect. Small differences may result in a match or a non-match. In this work a likelihood ratio system is presented, largely based on the two-level model as proposed by Aitken and Lucy [1], and Aitken, Zadora and Lucy [2]. Results show that the output from the two-level model gives good discrimination between same and different source hypotheses, but a post-hoc calibration step is necessary to improve the accuracy of the likelihood ratios. Subsequently, the robustness and performance of the LR system are studied. Results indicate that the output of the LR system is robust to the sample properties of the dataset used for calibration. Furthermore, the empirical upper and lower bound method [3], designed to deal with extrapolation errors in the density models, results in minimum and maximum values of the LR outputted by the system of 3.1×10-3 and 3.4×104. Calibration of the system, as measured by empirical cross-entropy, shows good behavior over the complete prior range. Rates of misleading evidence are small: for same-source comparisons, 0.3% of LRs support a different-source hypothesis; for different-source comparisons, 0.2% supports a same-source hypothesis. The authors use the LR system in reporting of glass cases to support expert opinion in the interpretation of glass evidence for origin of source questions.

Performance Study of a Score-based Likelihood Ratio System for Forensic Fingermark Comparison.

In this article, the performance of a score-based likelihood ratio (LR) system for comparisons of fingerprints with fingermarks is studied. The system is based on an automated fingerprint identification system (AFIS) comparison algorithm and focuses on fingerprint comparisons where the fingermarks contain 6-11 minutiae. The hypotheses under consideration are evaluated at the level of the person, not the finger. The LRs are presented with bootstrap intervals indicating the sampling uncertainty involved. Several aspects of the performance are measured: leave-one-out cross-validation is applied, and rates of misleading evidence are studied in two ways. A simulation study is performed to study the coverage of the bootstrap intervals. The results indicate that the evidential strength for same source comparisons that do not meet the Dutch twelve-point standard may be substantial. The methods used can be generalized to measure the performance of score-based LR systems in other fields of forensic science.

Quantifying Uncertainty in Estimations of the Total Weight of Drugs in Groups of Complex Matrices: Using the Welch-Satterthwaite Equation.

In this paper, a method is described to quantify estimations of the total amount of drugs in groups of seized items, including quantification of the precision. Previous work on this topic was based on the assumptions of normally distributed measurements and grouping of items with a common relative standard deviation. In practice, these assumptions are often violated, for example, for data with point masses at 0, or if certain items in a group have a very high standard deviation. The method described in this paper is based on work by Welch and Satterthwaite and does not assume constant relative standard deviations. Case examples are described for which the method is applied, and simulation studies are carried out for which both methods are applied. In the cases, both methods perform reasonably well. If the assumption of common relative standard deviations clearly does not apply, it is advised to use the method described.

Numerical likelihood ratios outputted by LR systems are often based on extrapolation: When to stop extrapolating?

A recent trend in forensic science is the development of objective, automated systems for the comparison of trace and reference material that give as output numerical likelihood ratios (LRs). For well discriminating LR systems, often the probability of the evidence given one or the other hypothesis depends on the density from the tail of a probability distribution. The models for probability distributions are trained by data. Since there is no proof of the applicability of the models beyond the data range, LR systems are sensitive to extrapolation errors. Given the unknown behavior in the tail region one may define the problem as when to stop extrapolating. When applied to LR systems, this leads to limit values of the likelihood ratio (e.g. a minimum and a maximum value of the LR outputted by the LR system), depending on the sizes of the validation datasets used. The solution proposed in this paper to determine these limits is based on the normalized Bayes error-rate [1] in combination with the introduction of misleading LRs with increasing strength.

Posterior distributions for likelihood ratios in forensic science.

Evaluation of evidence in forensic science is discussed using posterior distributions for likelihood ratios. Instead of eliminating the uncertainty by integrating (Bayes factor) or by conditioning on parameter values, uncertainty in the likelihood ratio is retained by parameter uncertainty derived from posterior distributions. A posterior distribution for a likelihood ratio can be summarised by the median and credible intervals. Using the posterior mean of the distribution is not recommended. An analysis of forensic data for body height estimation is undertaken. The posterior likelihood approach has been criticised both theoretically and with respect to applicability. This paper addresses the latter and illustrates an interesting application area.

The Influence of Wood Grain on the Bullet's Ricochet Behavior.

When a bullet ricochets from wood, various parameters will influence its behavior. In this study, the influence of the wood grain on the ricochet angle (ß) and deflection angle (γ) is assessed. Series of five .32 Auto bullets were fired at different angles of incidence (α) on boards of six wood types. The results confirm the previously shown effect that the mean ß-angles usually exceed α and increase when α increases. Overall, the maximum mean γ occurs when the angle of wood grain (ζ), in relation to the plane of impact, lies between 30° and 75° but differs per combination of wood and α. The results show the inclination of γ toward the left or right, depending on the bullets left or right rotation while also showing that the direction of ζ can enhance or counteract this effect considerably, especially when α is close to the critical ricochet angle.

An empirical study on the relation between the critical angle for bullet ricochet and the properties of wood.

The properties of a bullet, an object, and the incidence angle determine whether the bullet will penetrate, perforate, or ricochet. In this study, the critical angle for ricochet was established for .32 Auto and 9 mm Luger bullets on Abachi, Southern Yellow Pine, Beech, and Ipe wooden boards. Results show that the critical angle differs depending on caliber and wood type. The critical angle is higher for .32 Auto bullets than for 9 mm Luger bullets and increases with increasing wood density and Janka hardness. The established critical angles for ricochet on the lightest and softest wood (Abachi) are 10.4° and 10.3° for .32 Auto and 9 mm Luger, respectively. For the heaviest and hardest wood (Ipe), the angles were 45.0° and 33.4°, respectively. The combined results on the four tested woods show a strong linear relationship between both the density and the hardness and the critical angle for both calibers.

Quantifying uncertainty in estimations of the total weight of drugs in groups of complex matrices.

In case practice at forensic drug departments, multiple items from one seizure are sometimes sent in with the question: what is the total amount of drugs in the seizure? This may be complicated especially if impregnated material is involved such as clothes or rubber. Measurement uncertainty is typically stable on drug percentages, not weights, and subsampling may take place. It is recognized more and more that determination and reporting of uncertainty on estimators are an essential part of obtaining scientifically sound results in the forensic field. Methodology is described to quantify uncertainty on estimations of the total drug weight in groups of complex matrices, given simple statistical models, along a subdivision of five types of cases. Given each of these types, case examples are given where uncertainty is quantified in estimations of drug weights, by means of confidence intervals. The described models are statistically sound and relatively easy to implement.