Development and Testing of the First-Episode Psychosis Services Fidelity Scale.

OBJECTIVE: The purpose of this study was to test the reliability and validity of the First-Episode Psychosis Services Fidelity Scale (FEPS-FS) and compare it with similar scales. METHODS: A fidelity scale was developed from previously identified essential components of first-episode psychosis services. The scale was tested in six programs in two countries and compared with three existing scales. RESULTS: Program data collection from multiple sources indicated the feasibility and reliability of the FEPS-FS (intraclass correlation coefficient for interrater reliability=.842; 95% confidence interval=.795-.882). Satisfactory programs scored an average of 86% of the maximum total score; the single unsatisfactory program scored 70%. Compared with the other scales, the FEPS-FS has fewer items, but it has the highest proportion of items common to all scales. CONCLUSIONS: The FEPS-FS is a feasible, compact, reliable, and valid measure of adherence to evidence-based practices for first-episode psychosis services that can be applied to any first-episode psychosis service.

Relationship between relapse and hospitalization in first-episode psychosis.

OBJECTIVE: Relapse is a frequently used outcome measure in schizophrenia research. However, difficulties in reliably measuring relapse diminish its usefulness. Hospitalization is a potential alternative, but its relationship to relapse has not been assessed. METHODS: This study used data from a two-year, prospective study to examine associations between relapse and hospitalization in a cohort of 200 Canadian patients with first-episode psychosis. First, the relationship between relapse and hospitalization was assessed by cross-tabulating relapse and hospitalization. Next, survival curves of time to first relapse or hospitalization were compared. Finally, to examine the convergent validity of relapse and hospitalization, the predictive capacity of three predictors were examined: a substance use disorder diagnosis, prior hospitalization, and medication adherence. RESULTS: Rates of both relapse and hospitalization were similar. During the two-year follow-up, 37% of the patients experienced a relapse, and 26% were hospitalized. As an indicator of relapse, hospitalization had a low sensitivity (47%) and high specificity (87%). A higher risk of hospitalization and relapse was associated with prior hospitalization, a substance use disorder diagnosis, and medication nonadherence. CONCLUSIONS: Results indicated that relapse and hospitalization are separate but related outcome measures. They had similar frequencies and were found to have similar relationships with some predictors. Relapse is a more useful outcome measure in smaller clinical studies in which routine standardized clinical measures can be used. Hospitalization is more relevant in larger studies or as a quality indicator for studies using administrative databases, and it serves as a good measure for quality management in health systems.

Essential evidence-based components of first-episode psychosis services.

OBJECTIVE The purpose of this study was to identify essential evidence-based components of first-episode psychosis services. METHODS The study was conducted in two stages. In the first stage a systematic review of both peer-reviewed and gray literature (January 1980 to April 2010) was conducted. Databases searched included MEDLINE, PsycINFO, and EMBASE. In the second stage, a consensus-building technique, the Delphi, was used with an international panel of experts. The panelists were presented the evidence-based components identified in the review, together with the level of supporting evidence for each component. They rated the importance of each component on a 5-point scale. A score of 5 was required to determine that a component was essential. RESULTS The review identified 1,020 citations; abstracts were reviewed for relevance. A total of 280 peer-reviewed articles met criteria for relevance. Two researchers independently reviewed these articles and identified 75 unique service components. Each component was assigned a level of supporting evidence. Twenty-seven experts completed the first Delphi round, of whom 23 participated in the second. Consensus was achieved in two rounds, with 32 components rated as essential. CONCLUSIONS The two-step process yielded a manageable list of 32 evidence-based components of first-episode psychosis services. Given the proliferation of such services and the absence of an evidence-based fidelity scale, this list can form a foundation for developing a fidelity scale for such services. It may also be helpful to funders and providers as a summary of essential services.

Development of a core set of performance measures for evaluating schizophrenia treatment services.

OBJECTIVE: The purpose of study was to identify a list of performance measures for schizophrenia treatment services and to assemble a multistakeholder group to reach consensus on a core list. METHODS: The study was conducted in two stages: first, a systematic review of the literature was conducted to identify a comprehensive list of measures; second, a consensus-building technique, the Delphi process, was used with participants from six groups of stakeholders: schizophrenia experts, mental health clinicians, mental health administrators, the payer (the Alberta Ministry of Health and Wellness), patients, and family members. Thirty stakeholders participated in three rounds of self-completed questionnaires. The degree of consensus achieved in the Delphi process was defined as the semi-interquartile range for each measure. RESULTS: Ninety-seven measures were identified in the literature review. The Delphi method reduced the list to 36 measures rated as essential. The measures address eight domains of service-level evaluation: acceptability, accessibility, appropriateness, competence, continuity, effectiveness, efficiency, and safety. Despite the diversity in backgrounds of the stakeholder groups, the Delphi technique was effective in moving participants' ratings toward consensus through successive questionnaire rounds. The resulting measures reflected the interests of all stakeholders. CONCLUSIONS: Several further steps are required before these measures are implemented and include working toward reliability and validity of specific measures, assessing the feasibility and cost-effectiveness of collecting the data, and finally, undertaking risk adjustment for outcome measures.

Validity of hospital admission as an outcome measure of services for first-episode psychosis.

OBJECTIVE: The objective of this study was to assess the external validity of hospitalization as an outcome measure for first-episode psychosis services. METHODS: A two-year, prospective cohort design was used to examine the associations between hospitalization, general psychopathology, and quality of life in a cohort of 606 participants treated for a first episode of psychosis. RESULTS: In bivariate analyses, general psychopathology and quality of life were each significantly associated with hospitalization at both time points. After analyses controlled for all covariates in the logistic models, both clinical measures continued to be significantly correlated with hospitalization at both one and two years. CONCLUSIONS: These results validate hospitalization as an outcome measure for first-episode psychosis services, and they extend the existing research showing a relationship between hospitalization and process measures to include other valid and reliable clinical measures.

Predictors of admission in first-episode psychosis: developing a risk adjustment model for service comparisons.

OBJECTIVE: The aim of this study was to develop a risk adjustment model based on hospital admissions that would enable comparison between services for patients with a first episode of psychosis. METHODS: Candidate predictor variables for hospital admission were identified in a literature review, from which an expert panel selected 12 potential risk adjustment variables by using a structured process, the Template for Risk Adjustment Information Transfer. Multivariable logistic regression modeling with the 12 variables was used to develop models in one cohort of first-episode psychosis patients (N=297); these models were validated with data from a second cohort (N=309). The C statistic, a measure of model discrimination, was calculated to assess model performance. RESULTS: In the data from the development sample, prior hospitalization was the only significant predictor of hospital admissions within one year of enrollment in the first-episode psychosis program (odds ratio [OR]=1.88, p=.05). Hospital admissions after two and three years from admission to the program were significantly associated with higher levels of initial positive symptoms (OR=1.07, p=.02; OR=1.06, p=.02, respectively), and prior hospitalizations (OR=2.72, p=.001; OR=3.34, p<.001, respectively). The logistic models performed well, with C statistics ranging from .72 to .74 for the three outcomes, where a value of 1.0 implies perfect model discrimination. In the validation data the C statistics were slightly lower, ranging from .67 to .72. CONCLUSIONS: According to the C statistic estimates, the model developed provided good discrimination and was relatively robust in predicting hospitalization of first-episode psychosis patients.