Training the next generation of learning health system scientists.

Introduction: The learning health system (LHS) aligns science, informatics, incentives, stakeholders, and culture for continuous improvement and innovation. The Agency for Healthcare Research and Quality and the Patient-Centered Outcomes Research Institute designed a K12 initiative to grow the number of LHS scientists. We describe approaches developed by 11 funded centers of excellence (COEs) to promote partnerships between scholars and health system leaders and to provide mentored research training. Methods: Since 2018, the COEs have enlisted faculty, secured institutional resources, partnered with health systems, developed and implemented curricula, recruited scholars, and provided mentored training. Program directors for each COE provided descriptive data on program context, scholar characteristics, stakeholder engagement, scholar experiences with health system partnerships, roles following program completion, and key training challenges. Results: To date, the 11 COEs have partnered with health systems to train 110 scholars. Nine (82%) programs partner with a Veterans Affairs health system and 9 (82%) partner with safety net providers. Clinically trained scholars (n = 87; 79%) include 70 physicians and 17 scholars in other clinical disciplines. Non-clinicians (n = 29; 26%) represent diverse fields, dominated by population health sciences. Stakeholder engagement helps scholars understand health system and patient/family needs and priorities, enabling opportunities to conduct embedded research, improve outcomes, and grow skills in translating research methods and findings into practice. Challenges include supporting scholars through roadblocks that threaten to derail projects during their limited program time, ranging from delays in access to data to COVID-19-related impediments and shifts in organizational priorities. Conclusions: Four years into this novel training program, there is evidence of scholars' accomplishments, both in traditional academic terms and in terms of moving along career trajectories that hold the potential to lead and accelerate transformational health system change. Future LHS training efforts should focus on sustainability, including organizational support for scholar activities.

Development of patient-reported outcome measures of children's oral health aesthetics.

OBJECTIVES: To develop and evaluate the psychometric properties of child- and parent-proxy measures of oral health aesthetics. METHODS: Items that describe children's perceptions of their oral attractiveness and its impact on social, emotional, and behavioural functioning were developed based on a systematic review of existing measures, clinician feedback (n = 13) and child semi-structured interviews (n = 27). The tools' content validity was assessed in cognitive interviews with 21 children. Items were administered to socio-demographically diverse samples of 998 children aged 8-17 years and 626 parents of children aged 5-17 years. Psychometric methods were used to finalize and calibrate item banks, generate short questionnaire forms, and evaluate the tools' reliability, precision and validity. RESULTS: The item banks and their short forms provide precise measurement across a wide range of oral health aesthetic states. They measure relevant and meaningful positive and negative experiences using terminology that most children as young as 8 years of age can understand. Known-group comparisons and convergence with existing measures of oral health-related quality of life, global health and body image provide evidence of construct validity. The scores are interpretable relative to the US general population. CONCLUSIONS: The oral health aesthetic item banks and short forms provide precise and valid assessments of children's satisfaction with their oral appearance. They may be useful for targeting and evaluating paediatric dental and orthodontic care in clinical practice and research settings.

Assessing Children's Eudaimonic Well-Being: The PROMIS Pediatric Meaning and Purpose Item Banks.

OBJECTIVE: To describe the development of the Patient-Reported Outcome Measurement Information System (PROMIS) Pediatric Meaning and Purpose item banks, child-report and parent-proxy editions. METHODS: Data were collected from two samples. The first comprised 1,895 children (8-17 years old) and 927 parents of children 5-17 years old recruited from an Internet panel, medical clinics, and schools. The second comprised a nationally representative sample of 990 children 8-17 years old and 1,292 parents of children 5-17 years old recruited from a different Internet panel. Item pool evaluation was done with Sample 1 and analyses were used to support decisions about item retention. The combined sample was used for item response theory (IRT) calibration of the item bank. Both samples were used in validation studies. RESULTS: Eleven items were deleted from the item pool because of poor psychometric performance. The final versions of the scales showed excellent reliability (>0.90). Short form scales (4 or 8 items) had a high degree of precision across over 4 SD units of the latent variable. The item bank positively correlated with extant measures of positive psychological functioning, and negatively correlated with measures of emotional distress, pessimism, and pain. Lower meaning and purpose scores were associated with adolescence and presence of a special healthcare need. CONCLUSION: The PROMIS Pediatric Meaning and Purpose item banks and their short forms are ready for use in clinical research and practice. They are measures of children's eudaimonic well-being and indicative of children's hopefulness, optimism, goal-directedness, and feelings that life is worth living.

Evaluation of the PROMIS pediatric global health scale (PGH-7) in children with asthma.

OBJECTIVE: To evaluate the reliability and validity of the PROMIS Pediatric Global Health scale, a 7-item measure of perceived physical, mental, and social health, in children with asthma. METHODS: From February 2014 to February 2015, convenience samples of 8-17 year-old children (n = 182) and parents of 5-17 year-old children (n = 328) visiting an emergency department for treatment of asthma were enrolled. The Asthma Control Test was used to characterize children as controlled versus not controlled, and the PROMIS Asthma Impact Scale was used to assess asthma symptoms' impact on functional status. We conducted longitudinal analyses among 92 children and 218 parents at 3 weeks, and 74 children and 171 parents at 8 weeks after enrollment. RESULTS: The PGH-7 reliability ranged from 0.66 to 0.81 for child-report and 0.76 to 0.82 for parent-proxy. In cross-sectional analyses, children with controlled asthma had PGH-7 scores 0.40-0.95 standard deviation units higher than those who were uncontrolled. The PGH-7 was responsive to changes in overall general health between time points, with moderate effect sizes (0.5-0.6 standard deviation units). In longitudinal analyses, PGH-7 scores were no different between those who stayed uncontrolled versus became controlled at 3 weeks of follow-up; however, by 8 weeks of follow-up, the differences between these groups were 0.7-0.8 standard deviation units, indicative of large effects. CONCLUSIONS: The PGH-7 is a reliable and valid patient-reported outcome for assessing general health among children with asthma. It is a useful complement to other asthma-specific outcome measures.

Development and Evaluation of the PROMIS® Pediatric Positive Affect Item Bank, Child-Report and Parent-Proxy Editions.

OBJECTIVE: The purpose of this study is to describe the psychometric evaluation and item response theory calibration of the PROMIS Pediatric Positive Affect item bank, child-report and parent-proxy editions. METHODS: The initial item pool comprising 53 items, previously developed using qualitative methods, was administered to 1,874 children 8-17 years old and 909 parents of children 5-17 years old. Analyses included descriptive statistics, reliability, factor analysis, differential item functioning, and construct validity. A total of 14 items were deleted, because of poor psychometric performance, and an 8-item short form constructed from the remaining 39 items was administered to a national sample of 1,004 children 8-17 years old, and 1,306 parents of children 5-17 years old. The combined sample was used in item response theory (IRT) calibration analyses. RESULTS: The final item bank appeared unidimensional, the items appeared locally independent, and the items were free from differential item functioning. The scales showed excellent reliability and convergent and discriminant validity. Positive affect decreased with children's age and was lower for those with a special health care need. After IRT calibration, we found that 4 and 8 item short forms had a high degree of precision (reliability) across a wide range of the latent trait (>4 SD units). CONCLUSION: The PROMIS Pediatric Positive Affect item bank and its short forms provide an efficient, precise, and valid assessment of positive affect in children and youth.

Development and psychometric evaluation of the PROMIS Pediatric Life Satisfaction item banks, child-report, and parent-proxy editions.

PURPOSE: To describe the psychometric evaluation and item response theory calibration of the PROMIS Pediatric Life Satisfaction item banks, child-report, and parent-proxy editions. METHODS: A pool of 55 life satisfaction items was administered to 1992 children 8-17 years old and 964 parents of children 5-17 years old. Analyses included descriptive statistics, reliability, factor analysis, differential item functioning, and assessment of construct validity. Thirteen items were deleted because of poor psychometric performance. An 8-item short form was administered to a national sample of 996 children 8-17 years old, and 1294 parents of children 5-17 years old. The combined sample (2988 children and 2258 parents) was used in item response theory (IRT) calibration analyses. RESULTS: The final item banks were unidimensional, the items were locally independent, and the items were free from impactful differential item functioning. The 8-item and 4-item short form scales showed excellent reliability, convergent validity, and discriminant validity. Life satisfaction decreased with declining socio-economic status, presence of a special health care need, and increasing age for girls, but not boys. After IRT calibration, we found that 4- and 8-item short forms had a high degree of precision (reliability) across a wide range (>4 SD units) of the latent variable. CONCLUSIONS: The PROMIS Pediatric Life Satisfaction item banks and their short forms provide efficient, precise, and valid assessments of life satisfaction in children and youth.

Concurrent validity of the PROMIS® pediatric global health measure.

PURPOSE: To evaluate the concurrent validity of the PROMIS Pediatric Global Health measure (PGH-7), child-report and parent-proxy versions. METHODS: Surveys were administered via home computer on two separate occasions (December, 2011 and August/September, 2012) to a convenience sample of 4636 children 8-17 years old and 2609 parents who participated in a national Internet panel. Data analysis included: (1) evaluations of differences in PGH-7 scores between groups defined by sociodemographics, clinical characteristics, and access to health care; (2) associations with 15 PROMIS pediatric measures; and (3) correlations with two health-related quality-of-life instruments, the KIDSCREEN-10 and PedsQL-15. RESULTS: PGH-7 scores were lower for children with chronic conditions, Hispanic ethnicity, low socioeconomic status, and barriers to accessing health care. The PGH-7 showed excellent convergent and discriminant validity with PROMIS pediatric measures of physical, mental, and social health. The PGH-7 was strongly correlated with the KIDSCREEN-10, which assesses positive health, and moderately correlated with the PedsQL-15, which assesses problems with a child's health. CONCLUSIONS: The PGH-7 measures global health, summarizing a child's physical, mental, and social health into a single score. These properties make it a useful clinical, population health, and research tool for applications that require an efficient, precise, and valid summary measure of a children's self-reported health status. Future research should prospectively evaluate the PGH-7's capacity to detect change that results from alterations in clinical status, transformations of the healthcare delivery system, and children's health development.

Medicaid and CHIP retention among children in 12 states.

OBJECTIVE: Though stable insurance is important to support optimal child health, the reproducibility of metrics to assess child health insurance retention at the state and county level has not been examined. We sought to determine reproducibility of public insurance retention rates for children using 3 different metrics at the state and county level. METHODS: Public health insurance retention for children was assessed using 3 different metrics calculated from 2006-2009 Medicaid Analytic Extract data from 12 selected states. The metrics were: 1) Duration: a prospective metric that quantifies the number of newly enrolled children continuously enrolled in public insurance 6, 12, and 18 months after initial enrollment during a selected period; (2) Infant Duration: assesses Duration only among infants born during a selected period; (3) Coverage: a prospective metric that quantifies the average percentage of time a selected population is enrolled over an 18-month interval. Reproducibility of the metrics was assessed using a range of sample sizes with resampling and determining changes in relative rankings of states/counties by retention rate. RESULTS: All 3 metrics demonstrated reproducible estimates at the state level with sample sizes of 2000, 5000, and 10,000. Reproducibility of relative rankings for child health insurance retention of counties within states were sensitive to county child population size and the amount of variability in retention rates within the county and at the state level. CONCLUSIONS: As health care reform unfolds, the complete set of these 3 reproducible metrics can be used to evaluate multipronged and multilevel strategies to retain eligible children in public health insurance.

Developing measures for pediatric quality: methods and experiences of the CHIPRA pediatric quality measures program grantees.

BACKGROUND: Monitoring quality is an important way of understanding how the health care system is serving children and families. The Children's Health Insurance Program Reauthorization Act of 2009 (CHIPRA) Pediatric Quality Measures Program (PQMP) funded efforts to develop and enhance measures to assess care for children and adolescents. We describe the processes used by the PQMP grantees to develop measures to assess the health care of children and adolescents in Medicaid and the Children's Health Insurance Program. METHODS: Key steps in the measures development process include identifying concepts, reviewing and synthesizing evidence, prioritizing concepts, defining how measures should be calculated, and measure testing. Stakeholder engagement throughout the process is critical. Case studies illustrate how PQMP grantees adapted the process to respond to the nature of measures they were charged to develop and overcome challenges encountered. RESULTS: PQMP grantees used varied approaches to measures development but faced common challenges, some specific to the field of pediatrics and some general to all quality measures. Major challenges included the limited evidence base, data systems difficult or unsuited for measures reporting, and conflicting stakeholder priorities. CONCLUSIONS: As part of the PQMP, grantees were able to explore innovative methods to overcome measurement challenges, including new approaches to building the evidence base and stakeholder consensus, integration of alternative data sources, and implementation of new testing methods. As a result, the PQMP has developed new quality measures for pediatric care while also building an infrastructure, expertise, and enhanced methods for measures development that promise to provide more relevant and meaningful tools for improving the quality of children's health care.

Patient reported outcomes as indicators of pediatric health care quality.

Health care reform has increased demand for pediatric health care quality evaluations, particularly those that assess the impact of care on patient and population health outcomes. Many of today's most common childhood conditions are characterized by symptoms, behaviors, and functional limitations that are best assessed as patient reported outcomes (PROs). Although they remain greatly underutilized, PROs have the potential to improve pediatric health care quality assessment at the point of care and through system-level performance evaluations. The functions, benefits, and challenges of these PRO applications are described and illustrated in case examples. Although challenges remain, numerous methodological and technical innovations facilitate the use of PROs as health care quality metrics. These include advances in PRO measure development methodologies, the integration of PRO measures into electronic health records, and developing consensus among providers that PROs provide valuable information that can be used to enhance patient care. Although additional work is needed to address remaining methodological challenges, pediatric PROs are increasingly recognized as valuable indicators of health care quality in the clinical environment and as measures of organization- and provider-level performance.

Development of the PROMIS ® pediatric global health (PGH-7) measure.

PURPOSE: To develop a practical, efficient, and reliable pediatric global health (PGH) measure that would be useful for clinical, quality improvement, and research applications. METHODS: Using the Patient Reported Outcome Measurement Information System (PROMIS) mixed-methods approach for item bank development, we identified an item pool that was well understood by children as young as age 8 years and tested its psychometric properties in an internet panel sample of 3,635 children 8-17 years old and 1,807 parents of children 5-17 years old. RESULTS: The final version of the PGH measure included 7 items assessing general, physical, mental, and social health. Four of these items had the same wording as the PROMIS adult global health measure. Internal consistency was 0.88 for the child-report form and 0.84 for the parent form; both had excellent test-retest reliability. The measures showed factor invariance across age categories. There was no differential item functioning by age, gender, race, or ethnicity. Because the measure includes the general health rating question, it is possible to estimate the PGH scale using this widely used single item. CONCLUSIONS: The PROMIS PGH-7 measure is a brief and reliable 7-item measure of a child's global health.

Continuity of public insurance coverage: a systematic review of the literature.

Publicly financed insurance programs are tasked with maintaining coverage for eligible children, but published measures to assess coverage have not been evaluated. Therefore, we sought to identify and categorize measures of health insurance continuity for children and adolescents. We conducted a systematic review of Medline and HealthStar databases, review of reference lists of eligible articles, and contact with experts. We categorized measures into 8 domains based on a conceptual framework. We identified 147 measures from 84 eligible articles. Most measures evaluated the following domains: always insured (41%), repeatedly uninsured (36%), and transition out of coverage (29%), while fewer assessed single gap in coverage, always uninsured, transition into coverage, change in coverage, and eligibility. Only 18% of measures assessed associations between continuity of coverage and child and adolescent health outcomes. These results suggest that a number of measures of continuity of coverage exist, but few measures have assessed impact on outcomes.

Subjective well-being measures for children were developed within the PROMIS project: presentation of first results.

OBJECTIVES: The aims of this Patient Reported Outcome Measurement Information System (PROMIS) study were to (1) conceptualize children's subjective well-being (SWB) and (2) produce item pools with excellent content validity for calibration and use in computerized adaptive testings (CATs). STUDY DESIGN AND SETTING: Children's SWB was defined through semistructured interviews with experts, children (aged 8-17 years), parents, and a systematic literature review to identify item concepts comprehensively covering the full spectrum of SWB. Item concepts were transformed into item expressions and evaluated for comprehensibility using cognitive interviews, reading level analysis, and translatability review. RESULTS: Children's SWB comprises affective (positive affect) and global evaluation components (life satisfaction). Input from experts, children, parents, and the literature indicated that the eudaimonic dimension of SWB-that is, a sense of meaning and purpose-could be evaluated. Item pools for life satisfaction (56 items), positive affect (53 items), and meaning and purpose (55 items) were produced. Small differences in comprehensibility of some items were observed between children and adolescents. CONCLUSION: The SWB measures for children are the first to assess both the hedonic and eudaimonic aspects of SWB. Both children and youth seem to understand the concepts of a meaningful life, optimism, and goal orientation.

Conceptual and methodological advances in child-reported outcomes measurement.

Increasingly, clinical, pharmaceutical and translational research studies use patient-reported outcomes as primary and secondary end points. Obtaining this type of information from children themselves is now possible, but effective assessment requires developmentally sensitive conceptual models of child health and an appreciation for the rapid change in children's cognitive capacities. To overcome these barriers, outcomes researchers have capitalized on innovations in modern measurement theory, qualitative methods for instrument development and new computerized technologies to create reliable and valid methods for obtaining self-reported health data among 8-17-year-old children. This article provides a developmentally focused framework for selecting child-report health assessment instruments. Several generic health-related quality of life instruments and the assessment tools developed by the NIH-sponsored Patient-Reported Outcome Measurement Information System network are discussed to exemplify advances in the measurement of children's self-reported health, illness, wellbeing and quality of life.