A Critical Appraisal of the Evidence Supporting the Factor Structure of Extant Coping Instruments.

This paper evaluated the evidence supporting the factor structure of extant coping instruments based on modern psychometric standards. Our literature search identified nine coping instruments that are routinely used to measure coping strategies in adult populations. While nearly 10 thousand papers have been published using these instruments, only 39 studies have investigated their psychometric validity. Our findings revealed that the majority of these studies did not follow current psychometric recommendations for establishing internal validity in part because they did not account for the ordinal nature of the data. Further, studies employing exploratory factor analysis used methods for identifying the number of factors to retain that have been found to have a low accuracy in a simulation study while those employing confirmatory factor analysis reported model fit statistics that did not meet widely accepted benchmarks. Hence, conflicting results were found within and across the nine coping instruments. Recommendations are made for improving future validation studies.

Measuring Maternal Gatekeeping: A Rasch Analysis of the Parental Regulation Inventory.

To understand factors that may influence father involvement, researchers have increasingly considered maternal gatekeeping, or the extent to which mothers might attempt to regulate (i.e., encourage, discourage) fathers' involvement in childrearing. Although several theoretical models of maternal gatekeeping have been advanced in recent years, maternal gatekeeping measurement has lagged significantly behind developments in gatekeeping theory. Rasch analysis offers a useful framework for conducting item-level analyses to evaluate measurement validity and identify areas of improvement for measurement scales. In the present study, Rasch analysis techniques were implemented to 1) illustrate how modern psychometric methods can be applied to validate measures in family psychology and 2) examine the validity of the Parental Regulation Inventory, a commonly used maternal gatekeeping measure (PRI; Van Egeren, 2000). Results indicated that the PRI exhibited adequate construct validity; however, measurement could be improved by including additional items on the PRI subscales. In particular, Rasch analyses indicated floor effects on fathers' reports of maternal gate closing, floor and ceiling effects on fathers' reports of maternal gate opening, and floor and ceiling effects on fathers' reports of maternal communication at 3- and 9- months postpartum. Recommendations for improving maternal gatekeeping measurement and implications for maternal gatekeeping theory are discussed.

Psychometric properties of the itch numeric rating scale, skin pain numeric rating scale, and atopic dermatitis sleep scale in adult patients with moderate-to-severe atopic dermatitis.

BACKGROUND: The Itch Numeric Rating Scale (NRS), Skin Pain NRS, and Atopic Dermatitis Sleep Scale (ADSS) are self-administered patient-reported outcome (PRO) instruments developed to assess symptoms in patients with atopic dermatitis (AD). The objective of this study was to evaluate the psychometric properties (reliability, validity, and responsiveness) and interpretability thresholds of these PROs using data from three pivotal Phase 3 studies in adults. METHODS: BREEZE-AD1, BREEZE-AD2, and BREEZE-AD5 evaluated the safety and efficacy of baricitinib in adults with moderate-to-severe AD. Clinician-reported outcomes and other PROs commonly assessed in patients with AD were used to estimate meaningful changes and evaluate test-retest reliability, convergent and divergent validity, known-groups validity, responsiveness, and meaningful change thresholds (MCTs) of the Itch NRS, Skin Pain NRS, and ADSS. RESULTS: The test-retest reliability of the Itch NRS, Skin Pain NRS, and ADSS was evidenced by generally large intraclass correlation coefficients (> 0.7) in stable groups of patients between baseline and Week 1 and Weeks 4 and 8. Moderate-to-large correlations (r > 0.4) at baseline and Week 16 were generally observed between each measure and other PROs measuring the same concept, supporting convergent validity. Small-to-moderate correlations with clinician-reported outcomes demonstrated divergent validity. Each instrument was able to distinguish between known groups of disease severity as assessed using other indicators of AD severity. The responsiveness of the Itch NRS, Skin Pain NRS, and ADSS scales was demonstrated through significant differences in their change scores from baseline to Week 16 between categories of change in another PRO also from baseline to Week 16. Thresholds for interpreting meaningful change were estimated as - 4.0 for the 0-10 Itch and Skin Pain NRS items; - 1.25 for the 0-4 ADSS Items 1 and 3 and; - 1.50 for the 0-29 ADSS Item 2, these equivalent to moderate degrees of change. CONCLUSIONS: Results of this study demonstrate that the psychometric properties of the Itch NRS, Skin Pain NRS, and ADSS are good to excellent. These findings support the use of these instruments in daily assessment of AD symptoms in adults with moderate-to-severe AD. Trial registration ClinicalTrials.gov numbers: NCT03334396, NCT03334422, and NCT03435081.

Prediction of Cognitive Ability With Social Determinants in Children of Low Birth Weight.

BACKGROUND: Despite strong prevention efforts and advances in neonatal care in recent decades, low birth weight remains a serious public health problem in the United States, and survivors remain at increased risk for lifelong problems including cognitive deficits. Current regional and local strategies for referral often rely on variable thresholds for birth weight and gestational age that may be poor analogues to cognitive risk. Improving early referral criteria offers many benefits, including improved cognitive outcomes for children and improved cost-effectiveness and resource utilization in resource-limited communities. OBJECTIVES: We hypothesized that social determinants measurable at birth or at birth hospital discharge, when combined with birth weight and gestational age, would offer an improvement over birth weight and gestational age alone in predicting cognitive test scores in school-aged children with low birth weight. METHODS: We conducted a secondary analysis using a birth cohort of children from the Fragile Families and Child Wellbeing Study. We created a panel of maternal, familial, and community-level social determinant indicators from the data and examined associations with cognitive measures assessed at age of 9 years. RESULTS: The final social determinant model was statistically significant and explained 35% of the total variance in composite test scores. The "standard care" model (birth weight and gestational age) only explained 9% of the variance. DISCUSSION: Assessment of social determinants may offer improvement over traditional referral criteria to identify children most at risk of cognitive deficits after low birth weight.

Assessing health worker competence to deliver a brief psychological treatment for depression: development and validation of a scalable measure.

Increased interest in disseminating and implementing psychological treatments has focused on the need for evidence-based training programs for providers, especially those without specialized training. To evaluate provider-training programs, validated outcome measures are necessary; however, the scalable measurement of training outcomes has been largely overlooked. Current methods of assessing providers' ability to deliver psychological treatments are generally time-consuming and costly, representing a major bottleneck in scaling up mental health care for commonly occurring disorders such as depression. The present study describes the development and initial validation of a scalable measure for assessing provider competence in delivering a brief behavioral activation treatment for depression, called the Healthy Activity Program, adapted for primary care settings. The measure focuses on testing knowledge about the treatment and applied knowledge regarding how to skillfully deliver the treatment, both essential features of competence. The measure was tested on a sample of 531 respondents with a variety of educational levels and professional backgrounds and found to meet the requirements of the Rasch model. Three versions of the measure each of equal difficulty were derived to allow repeat testing of training outcomes over time. A scalable measure of provider competence is an essential first step towards supporting the wider dissemination and implementation of brief psychological interventions for depression, especially in low-resource settings.

Genetic Risk Factors for Poor Cognitive Development in Children With Low Birth Weight.

Low birth weight is an ongoing public health problem with severe consequences for those affected, including early morbidity and mortality and elevated risk for lifelong deficits in cognitive function. These deficits can be ameliorated by early intervention in many cases. To contribute to criteria for earlier identification of at-risk children prior to the onset of delays or deficits, we examined relationships between three gene candidates-SLC6A4, BDNF, COMT-and cognitive outcomes at school age in a secondary analysis of existing data from a nationally representative cohort. Single nucleotide polymorphism rs4074134, a variant of BDNF, and a rare insertion/deletion in the intron region of SLC6A4 were significant predictors of cognitive performance. Our final model predicted 17% of the variance in composite cognitive test scores among children with low birth weight at school age (F = 96.36, p < .001, R2 = .17). Specifically, children homozygous for cytosine at rs4074134 scored .62 standard deviations higher on a measure of global cognition than children with one or more thymine. Similarly, children with an extra-long copy number variant of SLC6A4 scored .88 standard deviations higher than children who had one or more short forms of the gene. These findings support the potential for an approach to identifying children with low birth weights who are most at need of early intervention services. Future research should focus on validation of these findings in an independent sample and confirmation of the biological mechanisms through which these genes influence cognitive development.

A Rasch Analysis Validation of the Maslach Burnout Inventory-Student Survey with Preclinical Medical Students.

Construct: Burnout is a psychological construct characterized by emotional exhaustion that arises from an excess of physical, emotional, and social demands over an extended period. Symptoms of burnout include withdrawal or disengagement from work. Burnout has become an important public health concern due to its association with severe negative consequences across numerous professions. BACKGROUND: The most widely used instrument for measuring burnout is the Maslach Burnout Inventory (MBI). An adaptation of the MBI, the MBI-Student Survey (MBI-SS), was developed for college students. The MBI-SS consists of 15 items covering 3 domains of burnout: exhaustion, cynicism (CY), and professional efficacy (PE). Although studies have confirmed the validity of the MBI-SS for college student populations, studies of its use with medical students are limited. The purpose of this study was to employ the Rasch model to examine the psychometric properties of the MBI-SS when used with a population of preclinical medical students. APPROACH: Data were collected from 787 medical students who answered the MBI-SS at the conclusion of their 1st year. A maximum likelihood exploratory factor analysis for ordinal data confirmed the hypothesized three factor structure of the MBI-SS. Subsequently, a Rasch analysis was employed to further evaluate the measurement properties of MBI-SS. We used the Rasch Rating Scale model to investigate the extent to which the three MBI subscales conformed to proper measurement characteristics, including comprehensive coverage of person ability and item difficulty along the latent continuum. RESULTS: Most of the 15 items on the MBI-SS effectively fit the Rasch Rating Scale Model, with minimal measurement error. Respondents effectively used the full range of the rating scale for all 15 items. Two subscales (PE and CY) contained items that were difficult for respondents to endorse, resulting in significant gaps along the measurement continuum. The CY subscale exhibited a slight floor effect. The 3 subscales showed good person reliability, good real-item reliability, and good person separation. CONCLUSIONS: The Rasch analysis confirmed that the MBI-SS works well for measuring burnout among preclinical medical students. However, the Rasch analysis was able to identify that additional items are needed to improve the performance of MBI-SS. New items would be targeted at reducing the floor effect for the CY subscale and filling the other gaps in measurement along the latent continuum for the PE and CY subscales.

Ohio Modified Arm-Motor Ability Test (OMAAT): An Optimized Measure of Upper Extremity Functional Limitation in Hemiparetic Stroke.

This secondary analysis quantified the psychometric properties of the Ohio Modified Arm-Motor Ability Test (OMAAT) in a sample of neurologically stable chronic stroke survivors (n = 67, 40 men; mean age 59.8 yr, standard deviation = 12.8; 42 White, 23 Black, 2 other; 92.5% right-sided lesion; 44 ischemic stroke). Findings indicate high OMAAT internal consistency (Cronbach's α = .97, ordinal α = .98, Gugiu's bootstrap reliability = .97), unidimensionality, and strong positive factor loadings for all 20 OMAAT items. Convergent validity between OMAAT and Action Research Arm Test total scores was strong (r = .90, p < .0001). The OMAAT is the first short measure of upper extremity functional limitation available to clinicians and researchers that includes an administration manual and that has been examined using nonparametric psychometrics. A detailed administration manual is provided as a supplement to this article.

Development and Psychometric Evaluation of the Vocational Fit Assessment (VFA).

OBJECTIVE: The objective of this study was to determine the psychometric properties of the Vocational Fit Assessment (VFA) by examining its factor structure and subscale reliability. METHOD: This prospective cross-sectional study used two surveys (one for worker abilities and one for job demands) to collect the data needed for the psychometric evaluation of the VFA. Latent parallel analysis and ordinal exploratory factor analysis were used to iteratively refine VFA subscales. RESULTS: Ten unidimensional subscales emerged from factor analysis of VFA items: (1) Cognitive Abilities, (2) Communication Skills, (3) Computer Skills, (4) Higher Task-Related Abilities, (5) Interpersonal Skills, (6) Lower Task-Related Abilities, (7) Physical Abilities, (8) Safety, (9) Self-Determination, and (10) Work Structure. Subscale internal consistency (ordinal α) was ≥.86 for VFA for worker abilities and ≥.77 for VFA for job demands. CONCLUSION: The unidimensional structure of VFA subscales and estimates of internal consistency lend initial evidence in support of their reliability and validity.

Rasch Analysis of the Wrist and Hand Fugl-Meyer: Dimensionality and Item-Level Characteristics.

BACKGROUND AND PURPOSE: Clinical administration of the wrist stability, wrist mobility, and hand items of the upper-extremity Fugl-Meyer (W/H UE FM) may provide a rigorous, easily administered, bedside measure of motor impairment in mildly impaired stroke survivors. The purpose of this study was to evaluate the item structure of the W/H UE FM to better understand its measurement properties using Rasch analysis. METHODS: This was a secondary analysis of W/H UE FM data arising from clinical trials of mildly impaired stroke survivors using latent parallel analysis, ordinal factor analysis, and partial credit model Rasch analyses. RESULTS: Latent parallel analysis and ordinal factor analysis indicated that all W/H UE FM items represent a single unidimensional construct, wrist and hand motor ability. Rasch analysis of data from 150 mildly impaired stroke survivors (94 men; mean age, 57.1 ± 11.4 years; mean time since stroke, 19.5 months) revealed that the W/H UE FM operated as a reliable, valid, and effective measure of wrist and hand motor ability. These data were compatible with Rasch model assumptions and are consistent with previous W/H UE FM research. Mass flexion and extension movements were the least difficult W/H UE FM items while the radial and hook grasp items were the most difficult. DISCUSSION AND CONCLUSION: The W/H UE FM is well suited to mildly impaired stroke survivors who exhibit the ability to perform mass flexion and mass extension movements. The full-scale UE FM may be preferable for stroke survivors with lower levels of ability.Video abstract available for additional insight from the authors (Supplemental Digital Content 1, http://links.lww.com/JNPT/A108).

Hierarchy of evidence and appraisal of limitations (HEAL) grading system.

Despite more than 30 years of effort that has been dedicated to the improvement of grading systems for evaluating the quality of research study designs considerable shortcomings continue. These shortcomings include the failure to define key terms, provide a comprehensive list of design flaws, demonstrate the reliability of such grading systems, properly value non-randomized controlled trials, and develop theoretically-derived systems for penalizing and promoting the evidence generated by a study. Consequently, in light of the importance of grading guidelines in evidence-based medicine, steps must be taken to remedy these deficiencies. This article presents two methods--a grading system and a measure of methodological bias--for evaluating the quality of evidence produced by an efficacy study.

The translational research impact scale: development, construct validity, and reliability testing.

Increasing emphasis is being placed on measuring return on research investment and determining the true impacts of biomedical research for medical practice and population health. This article describes initial progress on development of a new standardized tool for identifying and measuring impacts across research sites. The Translational Research Impact Scale (TRIS) is intended to provide a systematic approach to assessing impact levels using a set of 72 impact indicators organized into three broad research impact domains and nine subdomains. A validation process was conducted with input from a panel of 31 experts in translational research, who met to define and standardize the measurement of research impacts using the TRIS. Testing was performed to estimate the reliability of the experts' ratings. The reliability was found to be high (ranging from .75 to .94) in all of the domains and most of the subdomains. A weighting process was performed assigning item weights to the individual indicators, so that composite scores can be derived.

An application of a new evidence grading system to research on the chronic care model.

Methodological quality undergirds all evidence-based medicine because without strong evidence supporting or refuting the efficacy of an intervention, the movement toward basing medical decisions and practice on scientific evidence is not sustainable. Recently, the consensus that had existed regarding the hierarchy of evidence produced by a study design was challenged on the basis that existing guidelines failed to properly define key terms, weight the merits of certain non-randomized controlled trials, and employ a comprehensive list of study design limitations to render evaluative conclusions, to name a few of the challenges. The present study introduces a new grading system that overcomes, or at the very least greatly diminishes, these challenges. This new method is applied to the literature on the Chronic Care Model and the results are then compared to several of the most popular grading guidelines currently in use. These results revealed substantial differences between the guidelines in accordance with previous research that challenged existing methods. Furthermore, the present study lends support to the proposed grading guideline although further research into its validity and reliability is needed.

A critical appraisal of standard guidelines for grading levels of evidence.

Over the past 30 years, a general consensus has emerged within the medical community regarding the essential role served by grading guidelines in evaluating the quality of evidence produced by a medical research study. Specifically, consensus exists regarding the hierarchy of evidence, where randomized controlled trials (RCTs) are considered the ''gold standard'' followed by nonrandomized controlled trials (non-RCTs) and uncontrolled trials. As guidelines have become more sophisticated, processes have been developed for downgrading poorly conducted studies and upgrading strong studies. Lists of threats to internal validity have been disseminated, thereby assisting reviewers in grading studies. However, despite these many accomplishments, considerable issues remain unresolved with respect to how to evaluate the strength of evidence produced by flawed RCTs versus well-conducted non-RCTs. The purpose of this article is to evaluate existing evidence-based grading guidelines and to offer suggestions for how such guidelines may be improved.

Development and evaluation of the short version of the Patient Assessment of Chronic Illness Care instrument.

OBJECTIVES: The Patient Assessment of Chronic Illness Care (PACIC) was developed for measuring the extent to which patients receive care congruent with the chronic care model (CCM). The purpose of this study was to develop a short version of the PACIC with better psychometric properties than the original instrument. METHODS: Two samples of 529 and 361 type 2 diabetic patients completed a modified PACIC. A short-version PACIC instrument was developed and validated using parallel analysis to determine the number of factors, confirmatory factor analysis (CFA) within an exploratory factor analysis framework (E/CFA) was conducted to explore the measurement structure of the full instrument, and a CFA was performed to confirm the hypothesized structure. RESULTS: The results demonstrated that the PACIC is unidimensional and that it can be reduced to 11 items with no loss in psychometric properties. No demographic variables or clinical assays were found to be related to the PACIC. DISCUSSION: A short-version PACIC is now available and ready for use in research with diabetic patients. Its use is encouraged in future research, particularly in the exploration of its validity against actual CCM services delivered and long-term clinical outcomes.

Semi-structured interview protocol for constructing logic models.

This paper details a semi-structured interview protocol that evaluators can use to develop a logic model of a program's services and outcomes. The protocol presents a series of questions, which evaluators can ask of specific program informants, that are designed to: (1) identify key informants basic background and contextual information, (2) generate logic model elements, (3) model program inputs, activities, outputs, and outcomes, (4) build a rational theory, (5) develop a program theory, (6) prioritize logic model elements, and (7) build a graphical or tabular logic model. The paper will also provide an example of how this approach was used to develop a logic model for a youth mentoring program. It is our hope and belief that with this interview protocol, novice evaluators will be able to generate comprehensive logic models like seasoned professional evaluators.