Integrating Validity Evidence to Revise a Child Abuse Knowledge Test for Early Childhood Education Providers: A Mixed Methods Approach.

Knowledge tests used to evaluate child protection training program effectiveness for early childhood education providers may suffer from threats to construct validity given the contextual variability inherent within state-specific regulations around mandated reporting requirements. Unfortunately, guidance on instrument revision that accounts for such state-specific mandated reporting requirements is lacking across research on evaluation practices. This study, therefore, explored how collection and integration of validity evidence using a mixed methods framework can guide the instrument revision process to arrive at a more valid program outcome measure.

Effects of using an abdominal simulator to develop palpatory competencies in 3rd year medical students.

BACKGROUND: Medical school faculty are hard pressed to provide clerkship students with sufficient opportunity to develop and practice their capacity to perform a competent clinical examination, including the palpatory examination of the abdomen. We evaluated the impact of training with an abdominal simulator, AbSim, designed to monitor the depth, location, and thoroughness of their palpation and to provide concurrent and summative feedback regarding their performance. METHODS: All third-year medical students were given the opportunity to develop their palpatory skills with the AbSim simulator during the family medicine rotation. The performance of those who studied with the simulator was measured by its sensors, before and after a training session that included visual feedback regarding the depth and coverage of the student's manual pressure. Additionally, all students reported their confidence in their evolving abdominal palpation skills at the beginning and end of the rotation. RESULTS: 119 (86.9%) of 137 students filled out the initial questionnaire, and 73 (61.3%) studied with the abdominal simulator. The training produced a highly significant improvement in their overall performance (4 measures, p's < 0.001). Pre-training performance (depth calibration and thoroughness of coverage) was not related to the number of months of previous clinical rotations nor to previous internal medicine or surgery rotations. There was little relation between students' confidence in their abdominal examination skills and objective measures of their palpatory performance; however, students who chose the training started with less confidence, and became more confident after training. CONCLUSIONS: Guided abdominal simulator practice increased medical students' capacity to perform an abdominal examination with more appropriate depth and thoroughness of palpation. Interpretation of changes in confidence are uncertain, because confidence was unrelated to objectively measured performance. However, students with low initial confidence in their abdominal examination seemed to be more likely to choose to study with the abdominal simulator.

COGNITIVE MAPPING FOR ILOOKOUT FOR CHILD ABUSE: AN ONLINE TRAINING PROGRAM FOR EARLY CHILDHOOD PROFESSIONALS.

This article delineates the theory and framework for an innovative child abuse training program for mandated reporters called 'iLookOut'. iLookOut is an online learning delivery system that utilizes mastery learning and self-determination theory in the Core Training program, along with spaced retrieval and retrieval practice in a follow-up micro-learning program that reinforces learning from the Core Training. A cognitive mapping model provides the structure for documenting and organizing the learning content in both the Core training and the follow-up micro-learning program. The article provides a conceptual framework for designing and implementing effective and efficient online learning programs.

Generalizing findings from a randomized controlled trial to a real-world study of the iLookOut, an online education program to improve early childhood care and education providers' knowledge and attitudes about reporting child maltreatment.

In recent years, real-world studies (RWS) are gaining increasing interests, because they can generate more realistic and generalizable results than randomized controlled clinical trials (RCT). In 2017, we published a RCT in 741 early childhood care and education providers (CCPs). It is the Phase I of our iLookOut for Child Abuse project (iLookOut), an online, interactive learning module about reporting suspected child maltreatment. That study demonstrated that in a RCT setting, the iLookOut is efficient at improving CCPs' knowledge of and attitudes towards child maltreatment reporting. However, the generalizability of that RCT's results in a RWS setting remains unknown. To address this question, we design and conduct this large RWS in 11,065 CCPs, which is the Phase II of the iLookOut. We hypothesize replication of the earlier RCT findings, i.e., the iLookOut can improve CCPs' knowledge of and attitudes toward child maltreatment reporting in a real world setting. In addition, this RWS also explores whether demographic factors affect CCPs' performance. Results of this RWS confirmed the generalizability of the previous RCT's results in a real world setting. It yielded similar effect sizes for knowledge and attitudes as were found in the earlier RCT. Cohen's d for knowledge improvement was 0.95 in that RCT, 0.96 in this RWS; Cohen's d for attitude improvement was 0.98 in that RCT, 0.80 in this RWS. Also, we found several significant differences in knowledge and attitude improvement with regard to age, race, education, and employment status. In conclusion, iLookOut improves knowledge and attitudes of CCPs about child maltreatment prevention and reporting in a real-world setting. The generalizability of the initial RCT findings to this RWS provides strong evidence that the iLookout will be effective in other real world settings. It can be a useful model for other interventions aimed at preventing child maltreatment. Clinical trial registration for the original RCT: NCT02225301 (ClinicalTrials.gov Identifier).

Effects of Biopsychosocial Education on the Clinical Judgments of Medical Students and GP Trainees Regarding Future Risk of Disability in Chronic Lower Back Pain: A Randomized Control Trial.

BACKGROUND: Chronic lower back pain (CLBP) is a major health care burden and often results in workplace absenteeism. It is a priority for appropriate management of CLBP to get individuals back to work as early as possible. Interventions informed by the flags approach, which integrates cognitive and behavioral approaches via identification of biopsychosocial barriers to recovery, have resulted in reduced pain-related work absences and increased return to work for individuals with CLBP. However, research indicates that physicians' adherence to biopsychosocial guidelines is low. OBJECTIVE: The current study examined the effects of a flags approach-based educational intervention on clinical judgments of medical students and general practitioner (GP) trainees regarding the risk of future disability of CLBP patients. DESIGN: Randomized controlled trial (trial registration number: ISRCTN53670726). SETTING: University classroom. SUBJECTS: Medical students and GP trainees. METHODS: Using 40 fictional CLBP cases, differences in clinical judgment accuracy, weighting, and speed (experimental N = 32) were examined pre- and postintervention, as were flags approach knowledge, pain attitudes and beliefs, and empathy, in comparison with a no-intervention control group (control N = 31). RESULTS: Results revealed positive effects of the educational intervention on flags approach knowledge, pain-related attitudes and beliefs, and judgment weighting of psychologically based cues; results are discussed in light of existing theory and research. CONCLUSIONS: Short flags approach-based educational video interventions on clinical judgment-making regarding the risk of future disability of CLBP patients may provide opportunities to gain biopsychosocial knowledge, overcome associated attitude barriers, and facilitate development of clinical judgment-making more aligned with psychological cues.

Alternative Lens Model Equations for Dichotomous Judgments about Dichotomous Criteria.

OBJECTIVE: The Brunswik lens model typically represents a judge's accuracy using parameters derived from linear regression. This is not optimal if the judgment or the ecological criterion is dichotomous. Alternative approaches, modeling dichotomies using logistic regression, or linearizing judgments with confidence ratings, have not been compared with the same data. METHOD: Four techniques for deriving lens model equation parameters were compared: 1) linear and 2) logistic regression applied to dichotomous patient outcomes and judgments; 3) linear regression with confidence-adjusted judgments but dichotomous patient outcomes; and 4) a hybrid with a linear model of the confidence-adjusted judgments and a logistic model of the patient outcomes. RESULTS: Judgment accuracy (ra) was slightly higher with confidence adjustment of the categorical judgments. The logistic lens model accounted for a higher proportion of ra than the linear lens model; the confident-linear and hybrid lens models were intermediate. For up to a quarter of participants, different methods identified different cues as most important. Display condition differences in achievement ra and in lens model components are similar with all lens model methods. CONCLUSION: Each of the three alternative lens model equation methods improves on the linear lens model equation's decomposition of the accuracy of dichotomous judgments. Confidence adjustment improves achievement although it requires additional work from the subjects. The logistic lens model equation explains the highest proportion of achievement, but with a small stimulus set it is more vulnerable to cue intercorrelations than either the linear or the confident linear lens model equation.

In patients suspected of cognitive decline, what is the most accurate in-office screening instrument to determine if there is dementia needing further evaluation and management?

While screening for dementia in patients without symptoms is not recommended by the United States Preventative Services Task Force (USPSTF), screening in those presenting with symptoms may help patients and caregivers prepare for the future. When selecting which screening tool to use in a primary care office, one needs to consider practicality, feasibility, applicability, and psychometric properties. The MIS, MiniCog, PhotoTest, and GPCOG have been found to have short administration times, good accuracy, and applicability for a broad range of patient education and backgrounds. The BAS, MiniCog, PhotoTest and MAT have been found to be the most accurate tests. However, the evidence is limited and more studies need to be done to accurately answer the presented question.

Effectiveness of a biopsychosocial e-learning intervention on the clinical judgements of medical students and GP trainees regarding future risk of disability in patients with chronic lower back pain: study protocol for a randomised controlled trial.

INTRODUCTION: Chronic lower back pain (CLBP) is a major healthcare problem with wide ranging effects. It is a priority for appropriate management of CLBP to get individuals back to work as early as possible. Interventions that identify biopsychosocial barriers to recovery have been observed to lead to successfully reduced pain-related work absences and increased return to work for individuals with CLBP. Modern conceptualisations of pain adopt a biopsychosocial approach, such as the flags approach. Biopsychosocial perspectives have been applied to judgements about future adjustment, recovery from pain and risk of long-term disability; and provide a helpful model for understanding the importance of contextual interactions between psychosocial and biological variables in the experience of pain. Medical students and general practitioner (GP) trainees are important groups to target with education about biopsychosocial conceptualisations of pain and related clinical implications. AIM: The current study will compare the effects of an e-learning intervention that focuses on a biopsychosocial model of pain, on the clinical judgements of medical students and trainees. METHODS AND ANALYSIS: Medical student and GP trainee participants will be randomised to 1 of 2 study conditions: (1) a 20 min e-learning intervention focused on the fundamentals of the flags approach to clinical judgement-making regarding risk of future pain-related disability; compared with a (2) wait-list control group on judgement accuracy and weighting (ie, primary outcomes); flags approach knowledge, attitudes and beliefs towards pain, judgement speed and empathy (ie, secondary outcomes). Participants will be assessed at preintervention and postintervention. ETHICS AND DISSEMINATION: The study will be performed in agreement with the Declaration of Helsinki and is approved by the National University of Ireland Galway Research Ethics Committee. The results of the trial will be published according to the CONSORT statement and will be presented at conferences and reported in peer-reviewed journals. TRIAL REGISTRATION NUMBER: ISRCTN53670726; Pre-results.

Rationality, practice variation and person-centred health policy: a threshold hypothesis.

Variation in practice of medicine is one of the major health policy issues of today. Ultimately, it is related to physicians' decision making. Similar patients with similar likelihood of having disease are often managed by different doctors differently: some doctors may elect to observe the patient, others decide to act based on diagnostic testing and yet others may elect to treat without testing. We explain these differences in practice by differences in disease probability thresholds at which physicians decide to act: contextual social and clinical factors and emotions such as regret affect the threshold by influencing the way doctors integrate objective data related to treatment and testing. However, depending on a theoretical construct each of the physician's behaviour can be considered rational. In fact, we showed that the current regulatory policies lead to predictably low thresholds for most decisions in contemporary practice. As a result, we may expect continuing motivation for overuse of treatment and diagnostic tests. We argue that rationality should take into account both formal principles of rationality and human intuitions about good decisions along the lines of Rawls' 'reflective equilibrium/considered judgment'. In turn, this can help define a threshold model that is empirically testable.

When is rational to order a diagnostic test, or prescribe treatment: the threshold model as an explanation of practice variation.

BACKGROUND: The threshold model represents an important advance in the field of medical decision-making. It is a linchpin between evidence (which exists on the continuum of credibility) and decision-making (which is a categorical exercise - we decide to act or not act). The threshold concept is closely related to the question of rational decision-making. When should the physician act, that is order a diagnostic test, or prescribe treatment? The threshold model embodies the decision theoretic rationality that says the most rational decision is to prescribe treatment when the expected treatment benefit outweighs its expected harms. However, the well-documented large variation in the way physicians order diagnostic tests or decide to administer treatments is consistent with a notion that physicians' individual action thresholds vary. METHODS: We present a narrative review summarizing the existing literature on physicians' use of a threshold strategy for decision-making. RESULTS: We found that the observed variation in decision action thresholds is partially due to the way people integrate benefits and harms. That is, explanation of variation in clinical practice can be reduced to a consideration of thresholds. Limited evidence suggests that non-expected utility threshold (non-EUT) models, such as regret-based and dual-processing models, may explain current medical practice better. However, inclusion of costs and recognition of risk attitudes towards uncertain treatment effects and comorbidities may improve the explanatory and predictive value of the EUT-based threshold models. CONCLUSIONS: The decision when to act is closely related to the question of rational choice. We conclude that the medical community has not yet fully defined criteria for rational clinical decision-making. The traditional notion of rationality rooted in EUT may need to be supplemented by reflective rationality, which strives to integrate all aspects of medical practice - medical, humanistic and socio-economic - within a coherent reasoning system.

The balance beam metaphor: a perspective on clinical diagnosis.

Understanding the impact of clinical findings in discriminating between possible causes of a patient's presentation is essential in clinical judgment. A balance beam is a natural physical analogue that can accurately represent the combination of several pieces of evidence with varying ability to discriminate between disease hypotheses. Calculation of Bayes' theorem using log(posterior odds) as a function of log(prior odds) and the logarithms of the evidence's likelihood ratios maps onto the physical forces affecting objects placed on a balance beam. We describe the rules governing the functioning of tokens representing clinical findings in the comparison of 2 competing diseases. The likelihood ratios corresponding to positive (LR+) or negative (LR-) observations for each symptom determine the lateral position at which the symptom's token is placed on the beam, using a weight if the finding is present and a helium balloon if it is absent. We discuss how a balance beam could represent concepts of dynamic specificity (due to changes in competitor diseases' probabilities) and dynamic sensitivity (due to class-conditional independence). Utility-based thresholds for acting on a diagnosis could be represented by moving the balance beam's fulcrum. It is suggested that a balance beam can be a useful aid for students learning clinical diagnosis, allowing them to build on existing intuitive understanding to develop an appreciation of how evidence combines to influence degree of belief. The balance beam could also facilitate exploration of the potential impact of available questions or investigations.

A balance beam aid for instruction in clinical diagnostic reasoning.

We describe a balance beam aid for instruction in diagnosis (BBAID) and demonstrate its potential use in supplementing the training of medical students to diagnose acute chest pain. We suggest the BBAID helps students understand the process of diagnosis because the impact of tokens (weights and helium balloons) attached to a beam at different distances from the fulcrum is analogous to the impact of evidence to the relative support for 2 diseases. The BBAID presents a list of potential findings and allows students to specify whether each is present, absent, or unknown. It displays the likelihood ratios corresponding to a positive (LR+) or negative (LR-) observation for each symptom, for any pair of diseases. For each specified finding, a token is placed on the beam at a location whose distance from the fulcrum is proportional to the finding's log(LR): a downward force (a weight) if the finding is present and a lifting force (a balloon) if it is absent. Combining the physical torques of multiple tokens is mathematically identical to applying Bayes' theorem to multiple independent findings, so the balance beam is a high-fidelity metaphor. Seven first-year medical students and 3 faculty members consulted the BBAID while diagnosing brief patient case vignettes. Student comments indicated the program is usable, helpful for understanding pertinent positive and negative findings' usefulness in particular situations, and welcome as a reference or self-test. All students attended the effect of the tokens on the beam, although some stated they did not use the numerical statistics. Faculty noted the BBAID might be particularly helpful in reminding students of diseases that should not be missed and identifying pertinent findings to ask for.

Figure and ground in physician misdiagnosis: metacognition and diagnostic norms.

Meta-cognitive awareness, or self reflection informed by the "heuristics and biases" theory of how experts make cognitive errors, has been offered as a partial solution for diagnostic errors in medicine. I argue that this approach is not as easy nor as effective as one might hope. We should also promote mastery of the basic principles of diagnosis in medical school, continuing medical education, and routine reflection and review. While it may seem difficult to attend to both levels simultaneously, there is more to be gained from attending to both than from focusing only on one.

Improving colorectal cancer screening in family medicine: an Oklahoma Physicians Resource/Research Network (OKPRN) study.

PURPOSE: The purpose of this study was to describe colorectal cancer screening (CRCS) practices across a variety of primary care clinics and identify the methods used by primary care physicians (PCPs) with higher rates of CRCS ("exemplars"). METHODS: Physician questionnaires, structured interviews, medical record abstractions, and practice observations were conducted for 48 PCPs in 25 practices within a regional practice-based research network followed by secondary in-depth interviews to further investigate the practices of PCPs in the top quartile of CRCS rates ("exemplars"). RESULTS: We abstracted 3596 medical records (mean of 75 records per PCP). Overall, exemplars had higher CRCS rates (median, 57.2% vs. 27.6%; P < .001). Patients of exemplars had higher screening rates for fecal occult blood testing (FOBT) and colonoscopy but not for flexible sigmoidoscopy or double-contrast barium enemas. Exemplars adopted few of the system-based innovations proposed by researchers to improve CRCS. Colonoscopy was promoted as the preferred CRCS method. FOBT was recommended for patients who could not afford or did not want colonoscopy. Flexible sigmoidoscopy or barium enemas were rarely recommended. Exemplars used brief CRCS promotion scripts that informally paralleled theory-driven counseling techniques. CONCLUSIONS: Experienced PCPs use brief CRCS promotion scripts including counseling techniques that improve CRCS performance. Future research should be directed toward whether these techniques can be used to create an intervention aimed at PCPs to improve CRCS.

The effect of improving task representativeness on capturing nurses' risk assessment judgements: a comparison of written case simulations and physical simulations.

BACKGROUND: The validity of studies describing clinicians' judgements based on their responses to paper cases is questionable, because - commonly used - paper case simulations only partly reflect real clinical environments. In this study we test whether paper case simulations evoke similar risk assessment judgements to the more realistic simulated patients used in high fidelity physical simulations. METHODS: 97 nurses (34 experienced nurses and 63 student nurses) made dichotomous assessments of risk of acute deterioration on the same 25 simulated scenarios in both paper case and physical simulation settings. Scenarios were generated from real patient cases. Measures of judgement 'ecology' were derived from the same case records. The relationship between nurses' judgements, actual patient outcomes (i.e. ecological criteria), and patient characteristics were described using the methodology of judgement analysis. Logistic regression models were constructed to calculate Lens Model Equation parameters. Parameters were then compared between the modeled paper-case and physical-simulation judgements. RESULTS: Participants had significantly less achievement (ra) judging physical simulations than when judging paper cases. They used less modelable knowledge (G) with physical simulations than with paper cases, while retaining similar cognitive control and consistency on repeated patients. Respiration rate, the most important cue for predicting patient risk in the ecological model, was weighted most heavily by participants. CONCLUSIONS: To the extent that accuracy in judgement analysis studies is a function of task representativeness, improving task representativeness via high fidelity physical simulations resulted in lower judgement performance in risk assessments amongst nurses when compared to paper case simulations. Lens Model statistics could prove useful when comparing different options for the design of simulations used in clinical judgement analysis. The approach outlined may be of value to those designing and evaluating clinical simulations as part of education and training strategies aimed at improving clinical judgement and reasoning.

Interpreting the psychometric properties of the components of primary care instrument in an elderly population.

OBJECTIVE: To determine the psychometric properties of the Components of Primary Care Instrument (CPCI) in a patient population aged 65 or older. MATERIALS AND METHODS: 795 participants in the OKLAHOMA Studies, a longitudinal population-based study of predominantly Caucasian, elderly patients, completed the CPCI. Reliability analysis and confirmatory factor analysis were done to provide psychometric properties for this elderly sample. Models were constructed and tested to determine the best fit for the data including the addition of a method factor for negatively worded items. RESULTS: Cronbach's alphas were comparable to values reported in prior studies. The confirmatory factor analysis with factor inter-correlations and a method factor each improved the fit of the factor model to the data. The combined model's fit approached the level conventionally recognized as adequate. CONCLUSION: CPCI appears to be a reliable tool for describing patient perceptions of the quality of primary care for patients over age 65.