Costs of Implementing Electronic Context Factor Assessments and Patient-reported Outcomes in Pain Clinic Settings.

BACKGROUND: The Healing Encounters and Attitudes Lists (HEALs) patient-reported measures, consisting of 6 separate context factor questionnaires, predict patients' pain improvements. Our Patient-centered Outcomes Research Initiative-funded implementation project demonstrated success in using HEAL data during clinic consultations to enhance patient engagement, improve patient outcomes, and reduce opioid prescribing. OBJECTIVE: We aimed to determine the resources needed for additional sites to implement HEAL to improve pain care treatment. RESEARCH DESIGN: An observational study from March 1 to November 30, 2021, assessing implementation cost data from invoices, time and salary requirements for clinic personnel training, estimates of non-site-based costs, and one-time resource development costs. SUBJECTS: Unique patients eligible to complete a HEAL survey (N=24,018) and 74 clinic personnel. MEASURES: The Stages of Implementation Completion guided documentation of preimplementation, implementation, and sustainment activities of HEAL pain clinic operations. These informed the calculations of the costs of implementation. RESULTS: The total time for HEAL implementation is 7 months: preimplementation and implementation phases (4 mo) and sustainment (3 mo). One hour of HEAL implementation involving a future clinical site consisting of 2 attending physicians, 1 midlevel provider, 1 nurse manager, 1 nurse, 1 radiology technician, 2 medical assistants, and 1 front desk staff will cost $572. A 10-minute time increment for all clinic staff is $95. Total implementation costs based on hourly rates over 7 months, including non-site-based costs of consultations, materials, and technology development costs, is $28,287. CONCLUSIONS: Documenting our implementation costs clarifies the resources needed for additional new sites to implement HEAL to measure pain care quality and to engage patients and clinic personnel.

Validation of the Three-Item Screener for Personality Disorders From the Inventory of Interpersonal Problems (IIP-3).

We previously developed a three-item screener for identifying respondents with any personality disorder (PD) using the Inventory of Interpersonal Problems (IIP). The current goal was to examine the convergent validity of the IIP-3 with other PD screeners and diagnostic tools and to investigate its relationship to measures of adult attachment and emotion regulation. The sample consisted of participants from five studies (total N = 852), with data from collateral informants available for a subsample (N = 353). Despite its brevity, the IIP-3 showed moderate to strong relationships with other longer PD screeners, with PD symptom scores from the Structured Interview for DSM-IV Personality (SIDP-IV), and with a global rating of PD severity. It was most sensitive to the stylistic aspects of PD typical of the traditional DSM cluster B (dramatic, expressive) PDs. These results emerged with data from both participants and informants, although correlations using informant data were generally smaller.

Using formative evaluation methods to improve clinical implementation efforts: Description and an example.

Formative evaluation, a rigorous assessment process to identify potential and actual influences on the implementation process, is a necessary first step prior to launching any implementation effort. Without formative evaluation, intervention studies may fail to translate into meaningful patient care or public health outcomes or across different contexts. Formative evaluation usually consists of qualitative methods, but may involve quantitative or mixed methods. A unique aspect of formative evaluation is that data are shared with the implementation team during the study in order to adapt and improve the process of implementation during the course of the study or improvement activity. In implementation science, and specifically within formative evaluation, it is imperative that a theory or conceptual model or framework guide the selection of the various individual, organizational or contextual factors to be assessed. Data from these theory-based constructs can translate into the development and specification of implementation strategies to support the uptake of the intervention. In this article, we describe different types of formative evaluations (developmental, implementation-focused, progress-focused, and interpretive), and then present a formative evaluation case study from a real-world implementation study within several academic pain clinics, guided by the Theory of Diffusion of Innovation.

Screening for Personality Disorders: A Three-Item Screener From the Inventory of Interpersonal Problems (IIP-3).

To encourage screening for personality disorders (PDs), we developed (in previous work) self-report scales for PDs using the Inventory of Interpersonal Problems (IIP). The combined score from three of the scales-inter-personal sensitivity, interpersonal ambivalence, and aggression-requiring 15 items (IIP-15) did the best job of distinguishing between respondents with any versus no PD. The goals for the present work were (a) to cross-validate the IIP-15 by examining its performance using receiver operating characteristics (ROC) analyses in a new sample (N = 410), and (b) to investigate the utility of a brief three-item variant (IIP-3). The present results again documented the good operating characteristics of the IIP-15. Sensitivity, specificity, and positive and negative predictive values were all above. 70. The operating characteristics of the IIP-3 were nearly as good despite its brevity and support its use as an initial screen for PDs.

An Item Bank for Abuse of Prescription Pain Medication from the Patient-Reported Outcomes Measurement Information System (PROMIS®).

OBJECTIVE: There is a need to monitor patients receiving prescription opioids to detect possible signs of abuse. To address this need, we developed and calibrated an item bank for severity of abuse of prescription pain medication as part of the Patient-Reported Outcomes Measurement Information System (PROMIS ® ). METHODS: Comprehensive literature searches yielded an initial bank of 5,310 items relevant to substance use and abuse, including abuse of prescription pain medication, from over 80 unique instruments. After qualitative item analysis (i.e., focus groups, cognitive interviewing, expert review, and item revision), 25 items for abuse of prescribed pain medication were included in field testing. Items were written in a first-person, past-tense format, with a three-month time frame and five response options reflecting frequency or severity. The calibration sample included 448 respondents, 367 from the general population (ascertained through an internet panel) and 81 from community treatment programs participating in the National Drug Abuse Treatment Clinical Trials Network. RESULTS: A final bank of 22 items was calibrated using the two-parameter graded response model from item response theory. A seven-item static short form was also developed. The test information curve showed that the PROMIS ® item bank for abuse of prescription pain medication provided substantial information in a broad range of severity. CONCLUSION: The initial psychometric characteristics of the item bank support its use as a computerized adaptive test or short form, with either version providing a brief, precise, and efficient measure relevant to both clinical and community samples.

Individuals with knee impairments identify items in need of clarification in the Patient Reported Outcomes Measurement Information System (PROMIS®) pain interference and physical function item banks - a qualitative study.

BACKGROUND: The content and wording of the Patient Reported Outcome Measurement Information System (PROMIS) Physical Function and Pain Interference item banks have not been qualitatively assessed by individuals with knee joint impairments. The purpose of this investigation was to identify items in the PROMIS Physical Function and Pain Interference Item Banks that are irrelevant, unclear, or otherwise difficult to respond to for individuals with impairment of the knee and to suggest modifications based on cognitive interviews. METHODS: Twenty-nine individuals with knee joint impairments qualitatively assessed items in the Pain Interference and Physical Function Item Banks in a mixed-methods cognitive interview. Field notes were analyzed to identify themes and frequency counts were calculated to identify items not relevant to individuals with knee joint impairments. RESULTS: Issues with clarity were identified in 23 items in the Physical Function Item Bank, resulting in the creation of 43 new or modified items, typically changing words within the item to be clearer. Interpretation issues included whether or not the knee joint played a significant role in overall health and age/gender differences in items. One quarter of the original items (31 of 124) in the Physical Function Item Bank were identified as irrelevant to the knee joint. All 41 items in the Pain Interference Item Bank were identified as clear, although individuals without significant pain substituted other symptoms which interfered with their life. CONCLUSIONS: The Physical Function Item Bank would benefit from additional items that are relevant to individuals with knee joint impairments and, by extension, to other lower extremity impairments. Several issues in clarity were identified that are likely to be present in other patient cohorts as well.

Validation of the alcohol use item banks from the Patient-Reported Outcomes Measurement Information System (PROMIS).

BACKGROUND: The Patient-Reported Outcomes Measurement Information System (PROMIS) includes five item banks for alcohol use. There are limited data, however, regarding their validity (e.g., convergent validity, responsiveness to change). To provide such data, we conducted a prospective study with 225 outpatients being treated for substance abuse. METHODS: Assessments were completed shortly after intake and at 1-month and 3-month follow-ups. The alcohol item banks were administered as computerized adaptive tests (CATs). Fourteen CATs and one six-item short form were also administered from eight other PROMIS domains to generate a comprehensive health status profile. After modeling treatment outcome for the sample as a whole, correlates of outcome from the PROMIS health status profile were examined. RESULTS: For convergent validity, the largest correlation emerged between the PROMIS alcohol use score and the Alcohol Use Disorders Identification Test (r=.79 at intake). Regarding treatment outcome, there were modest changes across the target problem of alcohol use and other domains of the PROMIS health status profile. However, significant heterogeneity was found in initial severity of drinking and in rates of change for both abstinence and severity of drinking during follow-up. This heterogeneity was associated with demographic (e.g., gender) and health-profile (e.g., emotional support, social participation) variables. CONCLUSIONS: The results demonstrated the validity of PROMIS CATs, which require only 4-6 items in each domain. This efficiency makes it feasible to use a comprehensive health status profile within the substance use treatment setting, providing important prognostic information regarding abstinence and severity of drinking.

Clinical validity of PROMIS Depression, Anxiety, and Anger across diverse clinical samples.

OBJECTIVES: The purpose of this study was to evaluate the responsiveness to change of the PROMIS negative affect measures (depression, anxiety, and anger) using longitudinal data collected in six chronic health conditions. STUDY DESIGN AND SETTING: Individuals with major depressive disorder (MDD), back pain, chronic obstructive pulmonary disease (COPD), chronic heart failure (CHF), and cancer completed PROMIS negative affect instruments as computerized adaptive test or as fixed-length short form at baseline and a clinically relevant follow-up interval. Participants also completed global ratings of health. Linear mixed effects models and standardized response means (SRM) were estimated at baseline and follow-up. RESULTS: A total of 903 individuals participated (back pain, n = 218; cancer, n = 304; CHF, n = 60; COPD, n = 125; MDD, n = 196). All three negative affect instruments improved significantly for treatments of depression and pain. Depression improved for CHF patients (anxiety and anger not administered), whereas anxiety improved significantly in COPD groups (stable and exacerbation). Response to treatment was not assessed in cancer. Subgroups of patients reporting better or worse health showed a corresponding positive or negative average SRM for negative affect across samples. CONCLUSION: This study provides evidence that the PROMIS negative affect scores are sensitive to change in intervention studies in which negative affect is expected to change. These results inform the estimation of meaningful change and enable comparative effectiveness research.

Measuring nonspecific factors in treatment: item banks that assess the healthcare experience and attitudes from the patient's perspective.

PURPOSE: Nonspecific factors that accompany healthcare treatments, such as patients' attitudes and expectations, are important parts of the experience of care and can influence outcomes. However, no precise, concise, and generalizable instruments to measure these factors exist. We report on the development and calibration of new item banks, titled the Healing Encounters and Attitudes Lists (HEAL), that assess nonspecific factors across a broad range of treatments and conditions. METHODS: The instrument development methodology of the Patient-Reported Outcomes Measurement Information System (PROMIS(®)) was used. Patient focus groups and clinician interviews informed our HEAL conceptual model. Literature searches of eight databases yielded over 500 instruments and resulted in an initial item pool of several thousand items. After qualitative item analysis, including cognitive interviewing, 296 items were included in field testing. The calibration sample included 1657 respondents, 1400 obtained through an Internet panel and 257 from conventional and integrative medicine clinics. Following exploratory and confirmatory factor analyses, the HEAL item banks were calibrated using item response theory. RESULTS: The final HEAL item banks were Patient-Provider Connection (57 items), Healthcare Environment (25 items), Treatment Expectancy (27 items), Positive Outlook (27 items), and Spirituality (26 items). Short forms were also developed from each item bank. A six-item short form, Attitudes toward Complementary and Alternative Medicine (CAM), was also created. CONCLUSIONS: HEAL item banks provided substantial information across a broad range of each construct. HEAL item banks showed initial evidence of predictive and concurrent validity, suggesting that they are suitable for measuring nonspecific factors in treatment.

Evaluating PROMIS® instruments and methods for patient-centered outcomes research: Patient and provider voices in a substance use treatment setting.

PURPOSE: Our work as a primary research site of the Patient-Reported Outcomes Measurement Information System (PROMIS®), combined with support from the Patient-Centered Outcomes Research Institute, allowed us to evaluate the real-world applicability and acceptability of PROMIS measures in an addiction medicine setting. METHODS: As part of a 3-month prospective observational study, 225 outpatients at a substance abuse treatment clinic completed PROMIS item banks for alcohol use (as well as 15 additional item banks from 8 other PROMIS domains, including emotional distress, sleep, and pain), with assessments at intake, 1-month follow-up, and 3-month follow-up. A subsample of therapists and their patients completed health domain importance ratings and qualitative interviews to elicit feedback regarding the content and format of the patients' assessment results. RESULTS: The importance ratings revealed that depression, anxiety, and lack of emotional support were rated highest of the non-alcohol-related domains among both patients and clinicians. General alcohol use was considered most important by both patients and clinicians. Based on their suggestions, changes were made to item response feedback to facilitate comprehension and communication. CONCLUSIONS: Both therapists and patients agreed that their review of the graphical display of scores, as well as individual item responses, helped them to identify areas of greatest concern and was useful for treatment planning. The results of our pilot work demonstrated the value and practicality of incorporating a comprehensive health assessment within a substance abuse treatment setting.

Item banks for substance use from the Patient-Reported Outcomes Measurement Information System (PROMIS(®)): Severity of use and positive appeal of use.

BACKGROUND: Two item banks for substance use were developed as part of the Patient-Reported Outcomes Measurement Information System (PROMIS(®)): severity of substance use and positive appeal of substance use. METHODS: Qualitative item analysis (including focus groups, cognitive interviewing, expert review, and item revision) reduced an initial pool of more than 5300 items for substance use to 119 items included in field testing. Items were written in a first-person, past-tense format, with 5 response options reflecting frequency or severity. Both 30-day and 3-month time frames were tested. The calibration sample of 1336 respondents included 875 individuals from the general population (ascertained through an internet panel) and 461 patients from addiction treatment centers participating in the National Drug Abuse Treatment Clinical Trials Network. RESULTS: Final banks of 37 and 18 items were calibrated for severity of substance use and positive appeal of substance use, respectively, using the two-parameter graded response model from item response theory (IRT). Initial calibrations were similar for the 30-day and 3-month time frames, and final calibrations used data combined across the time frames, making the items applicable with either interval. Seven-item static short forms were also developed from each item bank. CONCLUSIONS: Test information curves showed that the PROMIS item banks provided substantial information in a broad range of severity, making them suitable for treatment, observational, and epidemiological research in both clinical and community settings.

Validation of the depression item bank from the Patient-Reported Outcomes Measurement Information System (PROMIS) in a three-month observational study.

The Patient-Reported Outcomes Measurement Information System (PROMIS) is an NIH Roadmap initiative devoted to developing better measurement tools for assessing constructs relevant to the clinical investigation and treatment of all diseases-constructs such as pain, fatigue, emotional distress, sleep, physical functioning, and social participation. Following creation of item banks for these constructs, our priority has been to validate them, most often in short-term observational studies. We report here on a three-month prospective observational study with depressed outpatients in the early stages of a new treatment episode (with assessments at intake, one-month follow-up, and three-month follow-up). The protocol was designed to compare the psychometric properties of the PROMIS depression item bank (administered as a computerized adaptive test, CAT) with two legacy self-report instruments: the Center for Epidemiological Studies Depression scale (CESD; Radloff, 1977) and the Patient Health Questionnaire (PHQ-9; Spitzer et al., 1999). PROMIS depression demonstrated strong convergent validity with the CESD and the PHQ-9 (with correlations in a range from .72 to .84 across all time points), as well as responsiveness to change when characterizing symptom severity in a clinical outpatient sample. Identification of patients as "recovered" varied across the measures, with the PHQ-9 being the most conservative. The use of calibrations based on models from item response theory (IRT) provides advantages for PROMIS depression both psychometrically (creating the possibility of adaptive testing, providing a broader effective range of measurement, and generating greater precision) and practically (these psychometric advantages can be achieved with fewer items-a median of 4 items administered by CAT-resulting in less patient burden).

Using peer review to improve research and promote collaboration.

OBJECTIVE: The declining success rate of National Institutes of Health (NIH) grant applications highlights the need for interdisciplinary work within a large, diverse department to improve chances of federal funding success. The authors demonstrate how systematic peer review promotes two goals: enhancing the quality of research proposals and cultivating a collaborative departmental culture. METHODS: Changes to the Research Review Committee (RRC) in the Department of Psychiatry at the University of Pittsburgh were instituted to accommodate the increasingly interdisciplinary nature of grant applications, integrate revisions to NIH grant application processes, and incorporate advances in computer technology. RESULTS: The internal peer review process is associated with success in obtaining research support and with significant levels of collaborative scientific work reflected in both grant applications and peer-reviewed publications. CONCLUSIONS: A rich collaborative environment promoted through a rigorous internal peer review system has many benefits for both the quality of scholarly work and the collegiality of the research environment.

Item banks for alcohol use from the Patient-Reported Outcomes Measurement Information System (PROMIS): use, consequences, and expectancies.

BACKGROUND: We report on the development and calibration of item banks for alcohol use, negative and positive consequences of alcohol use, and negative and positive expectancies regarding drinking as part of the Patient-Reported Outcomes Measurement Information System (PROMIS). METHODS: Comprehensive literature searches yielded an initial bank of more than 5000 items from over 200 instruments. After qualitative item analysis (including focus groups and cognitive interviewing), 141 items were included in field testing. Items for alcohol use and consequences were written in a first-person, past-tense format with a 30-day time frame and 5 response options reflecting frequency. Items for expectancies were written in a third-person, present-tense format with no time frame specified and 5 response options reflecting intensity. The calibration sample included 1407 respondents, 1000 from the general population (ascertained through an internet panel) and 407 from community treatment programs participating in the National Institute on Drug Abuse (NIDA) Clinical Trials Network (CTN). RESULTS: Final banks of 37, 31, 20, 11, and 9 items (108 total items) were calibrated for alcohol use, negative consequences, positive consequences, negative expectancies, and positive expectancies, respectively, using item response theory (IRT). Seven-item static short forms were also developed from each item bank. CONCLUSIONS: Test information curves showed that the PROMIS item banks provided substantial information in a broad range of severity, making them suitable for treatment, observational, and epidemiological research.

Development of short forms from the PROMIS™ sleep disturbance and Sleep-Related Impairment item banks.

This article reports on the development of short forms from the Patient-Reported Outcomes Measurement Information System (PROMIS™) Sleep Disturbance (SD) and Sleep-Related Impairment (SRI) item banks. Results from post-hoc computerized adaptive testing (CAT) simulations, item discrimination parameters, item means, and clinical judgments were used to select the best-performing 8 items for SD and SRI. The final 8-item short forms provided less test information than the corresponding full banks, but correlated strongly with the longer forms. The short forms had greater measurement precision than the Pittsburgh Sleep Quality Index (PSQI) and the Epworth Sleepiness Scale (ESS), as indicated by larger test information values across the continuum of severity, despite having fewer total items--a major advantage for both research and clinical settings.

Development and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments.

STUDY OBJECTIVES: To develop an archive of self-report questions assessing sleep disturbance and sleep-related impairments (SRI), to develop item banks from this archive, and to validate and calibrate the item banks using classic validation techniques and item response theory analyses in a sample of clinical and community participants. DESIGN: Cross-sectional self-report study. SETTING: Academic medical center and participant homes. PARTICIPANTS: One thousand nine hundred ninety-three adults recruited from an Internet polling sample and 259 adults recruited from medical, psychiatric, and sleep clinics. INTERVENTIONS: None. MEASUREMENTS AND RESULTS: This study was part of PROMIS (Patient-Reported Outcomes Information System), a National Institutes of Health Roadmap initiative. Self-report item banks were developed through an iterative process of literature searches, collecting and sorting items, expert content review, qualitative patient research, and pilot testing. Internal consistency, convergent validity, and exploratory and confirmatory factor analysis were examined in the resulting item banks. Factor analyses identified 2 preliminary item banks, sleep disturbance and SRI. Item response theory analyses and expert content review narrowed the item banks to 27 and 16 items, respectively. Validity of the item banks was supported by moderate to high correlations with existing scales and by significant differences in sleep disturbance and SRI scores between participants with and without sleep disorders. CONCLUSIONS: The PROMIS sleep disturbance and SRI item banks have excellent measurement properties and may prove to be useful for assessing general aspects of sleep and SRI with various groups of patients and interventions.