An inter-laboratory comparison of probabilistic genotyping parameters and evaluation of performance on DNA mixtures from different laboratories.

Probabilistic genotyping (PG) is becoming the preferred standard for evidence interpretation, amongst forensic DNA laboratories, especially those in the United States. Various groups have expressed concern about reliability of PG systems, especially for mixtures beyond two contributors. Studies involving interlaboratory testing of known mixtures have been identified as ways to evaluate the reliability of PG systems. Reliability means different things in different contexts. However, it suffices here to think about it as a mixture of precision and accuracy. We might also consider whether a system is prone to producing misleading results - for example large likelihood ratios (LRs) when the POI is truly not a contributor, or small LRs when the POI is a truly a contributor. In this paper we show that the PG system STRmix™ is relatively unaffected by differences in parameter settings. That is, a DNA mixture that is analyzed in different laboratories using STRmix™ will result in different LRs, but less than 0.05% of these LRs would result in a different, or misleading conclusion as long as the LR is greater than 50. For the purposes of this study, we define LRs assigned using different parameters for the same mixtures as similar if the LR of the true POI is greater than the LRs generated for 99.9% of the general population. These findings are based on an interlaboratory study involving eight laboratories that provided twenty known DNA mixtures of two to four contributors and their individual laboratory STRmix™ parameters. The eight sets of laboratory parameters included differences in STR kits and PCR cycles as well as the peak, stutter, and locus specific amplification efficiency variances.

Developing an interpretation model for body fluid identification.

Criminal investigations, particularly sexual assaults, frequently require the identification of body fluid type in addition to body fluid donor to provide context. In most cases this can be achieved by conventional methods, however, in certain scenarios, alternative molecular methods are required. An example of this is the detection of menstrual fluid and vaginal material, which are not able to be identified using conventional techniques. Endpoint reverse-transcription PCR (RT-PCR) is currently used for this purpose to amplify body fluid specific messenger RNA (mRNA) transcripts in forensic casework. Real-time quantitative reverse-transcription PCR (RT-qPCR) is a similar method but utilises fluorescent markers to generate quantitative results in the form of threshold cycle (Cq) values. Despite the uncertainty surrounding body fluid identification, most interpretation guidelines utilise categorical statements. Probabilistic modelling is more realistic as it reflects biological variation as well as the known performance of the method. This research describes the application of various machine learning models to single-source mRNA profiles obtained by RT-qPCR and assesses their performance. Multinomial logistic regression (MLR), Naïve Bayes (NB), and linear discriminant analysis (LDA) were used to discriminate between the following body fluid categories: saliva, circulatory blood, menstrual fluid, vaginal material, and semen. We identified that the performance of MLR was somewhat improved when the quantitative dataset of the original Cq values was used (overall accuracy of approximately 0.95) rather than presence/absence coded data (overall accuracy of approximately 0.94). This indicates that the quantitative information obtained by RT-qPCR amplification is useful in assigning body fluid class. Of the three classification methods, MLR performed the best. When we utilised receiver operating characteristic curves to observe performance by body fluid class, it was clear that all methods found difficulty in classifying menstrual blood samples. Future work will involve the modelling of body fluid mixtures, which are common in samples analysed as part of sexual assault investigations.

A diagnosis of the primary difference between EuroForMix and STRmix™.

There is interest in comparing the output, principally the likelihood ratio, from the two probabilistic genotyping software EuroForMix (EFM) and STRmix™. Many of these comparison studies are descriptive and make little or no effort to diagnose the cause of difference. There are fundamental differences between EFM and STRmix™ that are causative of the largest set of likelihood ratio differences. This set of differences is for false donors where there are many instances of LRs just above or below 1 for EFM that give much lower LRs in STRmix™. This is caused by the separate estimation of parameters such as allele height variance and mixture proportion using MLE under Hp and Ha for EFM. This can result in very different estimations of these parameters under Hp and Ha . It results in a departure from calibration for EFM in the region of LRs just above and below 1.

Classification of firing pin impressions using HOG-SVM.

Crimes, such as robbery and murder, often involve firearms. In order to assist with the investigation into the crime, firearm examiners are asked to determine whether cartridge cases found at a crime scene had been fired from a suspect's firearm. This examination is based on a comparison of the marks left on the surfaces of cartridge cases. Firing pin impressions can be one of the most commonly used of these marks. In this study, a total of nine Ruger model 10/22 semiautomatic rifles were used. Fifty cartridges were fired from each rifle. The cartridge cases were collected, and each firing pin impression was then cast and photographed using a comparison microscope. In this paper, we will describe how one may use a computer vision algorithm, the Histogram of Orientated Gradient (HOG), and a machine learning method, Support Vector Machines (SVMs), to classify images of firing pin impressions. Our method achieved a reasonably high accuracy at 93%. This can be used to associate a firearm with a cartridge case recovered from a scene. We also compared our method with other feature extraction algorithms. The comparison results showed that the HOG-SVM method had the highest performance in this classification task.

The number of alleles in DNA mixtures with related contributors.

The maximum allele count (MAC) across loci and the total allele count (TAC) are often used to gauge the number of contributors to a DNA mixture. Computational strategies that predict the total number of alleles in a mixture arising from a certain number of contributors of a given population have been developed. Previous work considered the restricted case where all of the contributors to a mixture are unrelated. We relax this assumption and allow mixture contributors to be related according to a pedigree. We introduce an efficient computational strategy. This strategy based on first determining a probability distribution on the number of independent alleles per locus, and then conditioning on this distribution to compute a distribution of the number of distinct alleles per locus. The distribution of the number of independent alleles per locus is obtained by leveraging the Identical by Descent (IBD) pattern distribution which can be computed from the pedigree. We explain how allelic dropout and a subpopulation correction can be accounted for in the calculations.

Modeling allelic analyte signals for aSTRs in NGS DNA profiles.

We describe an adaption of Bright et al.'s work modeling peak height variability in CE-DNA profiles to the modeling of allelic aSTR (autosomal short tandem repeats) read counts from NGS-DNA profiles, specifically for profiles generated from the ForenSeq™ DNA Signature Prep Kit, DNA Primer Mix B. Bright et al.'s model consists of three key components within the estimation of total allelic product-template, locus-specific amplification efficiencies, and degradation. In this work, we investigated the two mass parameters-template and locus-specific amplification efficiencies-and used MLE (maximum likelihood estimation) and MCMC (Markov chain Monte Carlo) methods to obtain point estimates to calculate the total allelic product. The expected read counts for alleles were then calculated after proportioning some of the expected stutter product from the total allelic product. Due to preferential amplicon selection introduced by the sample purification beads, degradation is difficult to model from the aSTR outputs alone. Improved modeling of the locus-specific amplification efficiencies may mask the effects of degradation. Whilst this model could be improved by introducing locus specific variances in addition to locus specific priors, our results demonstrate the suitability of adapting Bright et al.'s allele peak height model for NGS-DNA profiles. This model could be incorporated into continuous probabilistic interpretation approaches for mixed DNA profiles.

Relaxing the assumption of unrelatedness in the numerator and denominator of likelihood ratios for DNA mixtures.

DNA mixtures will have multiple donors under both the prosecution and alternate propositions when assigning a likelihood ratio for forensic DNA evidence. These donors are usually assumed to be unrelated to each other. In this paper, we make a small, preliminary examination of the potential effect of relaxing this assumption. We consider the simple situation of a two-person mixture with no dropout and a two-person major/minor mixture with dropout of the minor contributor. We make no adjustment for subpopulation effects. Mixtures were simulated under two assumptions: 1. that the donors were siblings 2. or that they were unrelated. Both unresolvable and major/minor mixtures were considered. We compared the likelihood ratio assuming sibship with the likelihood ratio assuming no relatedness. The LR for hypotheses assuming no relatedness is less than the LR assuming relatedness approximately 95% of the time when relatives are present in the mixture.

Are low LRs reliable?

To answer the question "Are low likelihood ratios reliable?" requires both a definition of reliable and then a test of whether low likelihood ratios (LRs) meet that definition. We offer, from a purely statistical standpoint, that reliability can be determined by assessing whether the rate of inclusionary support for non-donors over many cases is not larger than expected from the LR value. Thus, it is not the magnitude of the LR alone that determines reliability. Turing's rule is used to inform the expected rate of non-donor inclusionary support, where the rate of non-donor inclusionary support is at most the reciprocal of the LR, i.e. Pr(LR > x|Ha) ≤1/x. There are parallel concerns about whether the value of the evidence can be communicated. We do not discuss that in depth here although it is an important consideration to be addressed with training. In this paper, we use a mixture of real and simulated data to show that the rate of non-donor inclusionary support for these data is significantly lower than the upper bound given by Turing's rule. We take this as strong evidence that low LRs are reliable.

Variability and additivity of read counts for aSTRs in NGS DNA profiles.

There has been an increase in the number of laboratories and researchers adopting new sequencing technologies, known as next-generation sequencing (NGS). An understanding of the behaviour of NGS DNA profiles is needed to enable for the development of probabilistic genotyping methods for the interpretation of such profiles. In this work, we investigate NGS analyte signal variation, specifically heterozygous balance and stutter variability from profiles generated using the ForenSeq™ DNA Signature Prep Kit, DNA Primer Mix B. We also investigate additivity of analyte signals in NGS profiles for overlapping allelic and stutter signals originating from the same or different contributors. We describe models that can be used to inform a continuous method for the interpretation of DNA profiling data.

Estimation of Y haplotype frequencies with lower order dependencies.

Estimating Y haplotype population frequencies is a demanding task in forensic genetics. Despite the suggestion of various methods, none these have yet reached a level of accuracy and precision that is acceptable to the forensic genetics community. At the basis of this problem is the complex dependency structure between the involved STR loci. Here, we approximate this structure by the use of specific graphical models, namely t-cherry junction trees. We apply trees of order three by which dependencies between three STR loci can be taken into account, thereby extending the Chow-Liu method which is restricted to pairwise dependencies. We show that the t-cherry tree method outperforms the Chow-Liu method as well as the well-established discrete Laplace method in estimation accuracy.

Testing whether stutter and low-level DNA peaks are additive.

Peaks in an electropherogram could represent alleles, stutter product, or a combination of allele and stutter. Continuous probabilistic genotyping (PG) systems model the heights of peaks in an additive manner: for a shared or composite peak, PG models assume that the peak height is the sum of the allelic component and the stutter component. In this work we examine the assumption that the heights of overlapping alleles from a minor contributor and stutter peaks from a major contributor are additive. Any peak below the analytical threshold is considered unobserved; hence, in any dataset and particularly in low-template DNA profiles, some or many peaks may be unobserved or missing. Using simulation and empirical data, we show that an additive model can explain the heights of overlapping alleles from a minor contributor and stutter peaks from a major contributor as long as missing data are carefully considered. We use a naive method of imputation for the missing data which appears to perform adequately in this case. If missing data are ignored then the sum of stutter and allelic peaks is expected to be an overestimate of the average height of the composite peaks, as was observed in this study.

Multi-element comparisons of tapes evidence using dimensionality reduction for calculating likelihood ratios.

Computing the likelihood ratio (LR), as a measure of weight of evidence, has traditionally been difficult for multi-element evidence. A solution based on multivariate random effects models has been adopted by the forensic community but suffers from instability and has a tendency toward extreme values. This problem is magnified by increasing the number of variables. In this study, we consider reducing the dimensionality of the problem using principal component analysis (PCA) and a post-hoc calibration step suggested by van Es et al. [1] and evaluate the performance of this method using multi-element data collected from electrical tapes with up to 18 elements measured. A set of 90 tapes known to originate from different sources were analyzed by LA-ICP-MS. We used additive log-ratio transformation with respect to the signal of 208Pb to transform the 18-dimensional data. This transformation altered the scale of the signals and more importantly, the transformed signals exhibited characteristics similar to a normal distribution. We used scores of the first five principal components (PCs) as input to the LR formula given by Aitken and Lucy [2] where we assumed multivariate normal between-sources distribution (LR MVN) to compare the tapes. We observed that the calculated LRs were extremely positive and negative and did not conform with the definition of well-calibrated LRs. Thus, we used the post-hoc calibration method given by van Es et al. [1] to calibrate the likelihood ratios. The calibrated LRs were obtained within an appropriate range. Five scenarios, each related to the number of principal components used to compare the samples formed part of this study. The first scenario made the comparisons using only the first PC, the second scenario used the first two PCs together and so on. The last scenario, LR5, used 5 PCs for the comparisons. Comparing the results of these 5 scenarios provided an understanding around sensitivity of the method based on the percentage of information used for the comparisons. The lowest false exclusion (Type I) and false inclusion (Type II) error rates were obtained for LR5 scenario in comparison to all the other scenarios. False inclusion and false exclusion error rates of 3.7% and 2.2% were reported by using only 5 out of 17 PCs. False exclusion error rates of 2.2% indicated that only two same-source comparisons had LR<1. The proposed method overcomes the problem of using highly-dimensional data for the comparisons, while using a high percentage of information present in the original data.

A response to "Likelihood ratio as weight of evidence: A closer look" by Lund and Iyer.

Recently, Lund and Iyer (L&I) raised an argument regarding the use of likelihood ratios in court. In our view, their argument is based on a lack of understanding of the paradigm. L&I argue that the decision maker should not accept the expert's likelihood ratio without further consideration. This is agreed by all parties. In normal practice, there is often considerable and proper exploration in court of the basis for any probabilistic statement. We conclude that L&I argue against a practice that does not exist and which no one advocates. Further we conclude that the most informative summary of evidential weight is the likelihood ratio. We state that this is the summary that should be presented to a court in every scientific assessment of evidential weight with supporting information about how it was constructed and on what it was based.

Importance sampling allows Hd true tests of highly discriminating DNA profiles.

Hd true testing is a way of assessing the performance of a model, or DNA profile interpretation system. These tests involve simulating DNA profiles of non-donors to a DNA mixture and calculating a likelihood ratio (LR) with one proposition postulating their contribution and the alternative postulating their non-contribution. Following Turing it is possible to predict that "The average LR for the Hd true tests should be one"[1]. This suggests a way of validating softwares. During discussions on the ISFG software validation guidelines [2] it was argued by some that this prediction had not been sufficiently examined experimentally to serve as a criterion for validation. More recently a high profile report [3] has emphasised large scale empirical examination. A limitation with Hd true tests, when non-donor profiles are generated at random (or in accordance with expectation from allele frequencies), is that the number of tests required depends on the discrimination power of the evidence profile. If the Hd true tests are to fully explore the genotype space that yields non-zero LRs then the number of simulations required could be in the 10s of orders of magnitude (well outside practical computing limits). We describe here the use of importance sampling, which allows the simulation of rare events to occur more commonly than they would at random, and then adjusting for this bias at the end of the simulation in order to recover all diagnostic values of interest. Importance sampling, whilst having been employed by others for Hd true tests, is largely unknown in forensic genetics. We take time in this paper to explain how importance sampling works, the advantages of using it and its application to Hd true tests. We conclude by showing that employing an importance sampling scheme brings Hd true testing ability to all profiles, regardless of discrimination power.

Admitting to uncertainty in the LR.

In this paper I argue that, given our current state of knowledge, reporting uncertainty in the likelihood ratio is best practice. This may in time be replaced by reporting a Bayes factor, but we are currently unable to do this in all but the simplest of examples.

Uncertainty in the number of contributors in the proposed new CODIS set.

The probability that multiple contributors are detected within a forensic DNA profile improves as more highly polymorphic loci are analysed. The assignment of the correct number of contributors to a profile is important when interpreting the DNA profiles. In this work we investigate the probability of a mixed DNA profile appearing as having originated from a fewer number of contributors for the African American, Asian, Caucasian and Hispanic US populations. We investigate a range of locus configurations from the proposed new CODIS set. These theoretical calculations are based on allele frequencies only and ignore peak heights. We show that the probability of a higher order mixture (five or six contributors) appearing as having originated from one less individual is high. This probability decreases as the number of loci tested increases.

The effect of wild card designations and rare alleles in forensic DNA database searches.

Forensic DNA databases are powerful tools used for the identification of persons of interest in criminal investigations. Typically, they consist of two parts: (1) a database containing DNA profiles of known individuals and (2) a database of DNA profiles associated with crime scenes. The risk of adventitious or chance matches between crimes and innocent people increases as the number of profiles within a database grows and more data is shared between various forensic DNA databases, e.g. from different jurisdictions. The DNA profiles obtained from crime scenes are often partial because crime samples may be compromised in quantity or quality. When an individual's profile cannot be resolved from a DNA mixture, ambiguity is introduced. A wild card, F, may be used in place of an allele that has dropped out or when an ambiguous profile is resolved from a DNA mixture. Variant alleles that do not correspond to any marker in the allelic ladder or appear above or below the extent of the allelic ladder range are assigned the allele designation R for rare allele. R alleles are position specific with respect to the observed/unambiguous allele. The F and R designations are made when the exact genotype has not been determined. The F and R designation are treated as wild cards for searching, which results in increased chance of adventitious matches. We investigated the probability of adventitious matches given these two types of wild cards.

A series of recommended tests when validating probabilistic DNA profile interpretation software.

There has been a recent push from many jurisdictions for the standardisation of forensic DNA interpretation methods. Current research is moving from threshold-based interpretation strategies towards continuous interpretation strategies. However laboratory uptake of software employing probabilistic models is slow. Some of this reluctance could be due to the perceived intimidating calculations to replicate the software answers and the lack of formal internal validation requirements for interpretation software. In this paper we describe a set of experiments which may be used to internally validate in part probabilistic interpretation software. These experiments included both single source and mixed profiles calculated with and without dropout and drop-in and studies to determine the reproducibility of the software with replicate analyses. We do this by way of example using three software packages: STRmix™, LRmix, and Lab Retriever. We outline and demonstrate the profile examples where the expected answer may be calculated and provide all calculations.

The variability in likelihood ratios due to different mechanisms.

Recently there has been a drive towards standardisation of forensic DNA interpretation methods resulting in the uptake of probabilistic interpretation software. Some of these software solutions utilise Markov chain Monte Carlo techniques (MCMC). They will not produce an identical answer after repeat interpretations of the same evidence profile because of the Monte Carlo aspect. This is a new source of variability within the forensic DNA analysis process. In this paper we explore the size of the MCMC variability within the interpretation software STRmix™ compared to other sources of variability in forensic DNA profiling including PCR, capillary electrophoresis load and injection, and the makeup of allele frequency databases. The MCMC variability within STRmix™ was shown to be the smallest source of variability in this process.