Measuring mental health outcomes with pre-post designs.

The pre-post design has been the workhorse of outcome evaluations for many years. Using data from a study of 984 treated children (ages 5 to 17 years), this article argues that there are two structural problems with the pre-post evaluation of outcome: (1) excessively large intervals of uncertainty for individual outcomes and (2) paradoxical inconsistencies in the evaluation of groups. These problems can be solved by designs with three or more repeated measures analyzed with longitudinal multilevel analytic models.

Looking for the disorder in conduct disorder.

Besides their well-known externalizing behavior, children with conduct disorder (CD) often have additional impairments outside the criteria for the CD diagnosis. In a 5-year study of 984 treated children (ages 5-17 years), those with CD had an average of 2.2 primary diagnoses. Children with CD showed the worst problem and impairment scores in comparison with 11 common diagnoses. Compared with other treated children, children with CD achieved worse scores on 14 of 15 syndromes, including internalizing problems such as withdrawal and major depression. The average child with CD had larger relapse scores in the 1.5- to 3-year period after admission to treatment. This pattern, pervasive at intake and chronic in course, resembles a global disability more than a circumscribed problem managed with a narrow range of treatments specific to it.

The co-occurrence of psychiatric and substance use diagnoses in adolescents in different service systems: frequency, recognition, cost, and outcomes.

The frequency, severity, recognition, cost, and outcomes of adolescent substance use comorbidity were analyzed in the Fort Bragg Demonstration Project. Comorbidity was defined as the co-occurrence of substance use disorder (SUD) with other psychiatric diagnosis. The sample consisted of 428 adolescent clients whose providers' diagnoses were compared with research diagnoses. The project identified 59 clients (13.8%) with SUD, all with additional psychiatric diagnoses. Providers recognized only 21 of these 59 comorbid cases. The frequency and severity of comorbidity did not differ between service system samples, although recognition did. Comorbid clients had more behavior problems and more functioning impairment, and their average treatment cost ($29,057) was more than twice as high as that of noncomorbid clients ($13,067). Mental health outcomes were not influenced by type of service system, comorbid diagnosis, or treatment. Screening for and prevention of SUD are discussed as a potential cost-savings opportunity in mental health services.

The Fort Bragg continuum of care for children and adolescents: mental health outcomes over 5 years.

Controversial early results of the Fort Bragg mental-health-effectiveness study indicated that the continuum of care did not produce better outcomes (i.e., children's rate of improvement was the same in both the demonstration and comparison sites). The present study considered outcomes at 5-year follow-up to examine long-term effects from the continuum of care. A random regression longitudinal model analyzed 10 key outcome variables measured 7 times. Long-term outcomes in continuum-treated children were no better than those of comparison children; results are consistent with those of earlier studies.

Dose effect in child psychotherapy: outcomes associated with negligible treatment.

OBJECTIVE: To compare the outcomes of children who received negligible amounts of outpatient treatment to children receiving more treatment. METHOD: A random regression longitudinal model was used to analyze outcomes of children (aged 5-17 years) from the Fort Bragg Evaluation Project. RESULTS: In examining several outcomes, the results show no statistically significant dose effect. CONCLUSIONS: Children receiving substantial amounts of treatment showed no better mental health outcomes than those receiving negligible amounts of treatment. The results do not support the existence of a dose effect consistent enough to guide clinicians, administrators, or policymakers.

Refining the costs analyses of the Fort Bragg evaluation: the impact of cost offset and cost shifting.

A key aim of the evaluation of the Fort Bragg Demonstration was to determine whether delivering services through a continuum of care lowered expenditures on mental health services. The evaluation clearly showed that expenditures were actually higher in the Demonstration. Critics of the evaluation claimed that the evaluation's perspective on costs was too narrow-in particular, that the Demonstration produced cost shifting and cost offset that were not captured by the evaluation. New data allow us to include a broader array of costs: mental health services received outside the catchment areas, general medical services for the children themselves, and mental health services used by family members. Results showed that reductions in other costs do partially offset higher expenditures on mental health services for children at the Fort Bragg Demonstration. However, even when broader costs are included, total family expenditures are still substantially higher at the Demonstration.

Child sexual abuse II: treatment.

OBJECTIVE: To evaluate the scientific literature concerning the treatment of child sexual abuse. METHOD: A critical review of the scientific literature. RESULTS: There are only nine published research studies in which subjects were randomly assigned to an index treatment or treatments and a comparison treatment or no-treatment control group. In seven of the studies, the index treatment exceeded the control or comparison group in regard to treatment outcome; in two studies it did not. The successful treatments involved group therapy, combined individual and group play therapy and cognitive behaviour therapy. CONCLUSIONS: Treatment should be based on an explicit conceptual model of the psychopathology of sexual abuse. The University of Queensland Sexual Abuse Treatment Project, which is based on a transactional model, is described.

Long-term effects of a system of care on children and adolescents.

This study evaluates an exemplary system of care designed to provide comprehensive mental health services to children and adolescents. It was believed that the system would lead to more improvement in the functioning and symptoms of clients compared to those receiving care as usual. The project employed a randomized experimental five-wave longitudinal design with 350 families. While access to care, type of care, and the amount of care were better in the system of care, there were no differences in clinical outcomes compared to care received outside the system. In addition, children who did not receive any services, regardless of experimental condition, improved at the same rate as treated children. Similar to the Fort Bragg results, the effects of systems of care are primarily limited to system-level outcomes but do not appear to affect individual outcomes such as functioning and symptomatology.

Dose-effect relationship in children's psychotherapy services.

Concerns about session limits imposed by behavioral health care organizations (BHOs) have prompted an examination of the dose-effect relationship in psychotherapy. This study of the dose-effect relationship in child psychotherapy used multiple outcome measures and standardized data-collection points and addressed the confound between dose, effect, and initial severity. Data obtained from 567 outpatient children who participated in the Fort Bragg Evaluation Project were used. Probit and longitudinal growth curve analyses found no evidence of a general dose-effect relationship. Implications of these findings are discussed, especially the need for BHOs to justify use limits and suggestions for clinicians to justify use requests.

Rejoinder to Mordock's critique of the Fort Bragg Evaluation Project: the sample is generalizable and the outcomes are clear.

The Fort Bragg Evaluation Project (FBEP) showed that children in a well implemented and expensive continuum of care had no better clinical outcomes than those experiencing more traditional and fragmented services. In an article published in this journal that was critical of the evaluation, Mordock argued that the FBEP results be viewed with skepticism because of what he perceived to be methodological, design, measurement, and analytic failures of this study. We think it is important to respond to Mordock's critique since it contributes to the great reluctance to seriously consider the study's findings and their implications.

Clinical outcome, consumer satisfaction, and ad hoc ratings of improvement in children's mental health.

Mental health clinics and managed care organizations assess treatment effectiveness with consumer satisfaction measures and ad hoc measures of improvement obtained from a single informant; some of these measures are as simple as asking clients whether they improved during treatment. In the present correlational study of 199 treated adolescents, we used a multitrait-multimethod analysis to examine psychometrically measured pathology change (pre- and postassessment of symptoms and functioning), consumer satisfaction, and perceived improvement reported by multiple informants. Confirmatory factor-analytic results indicate that (a) outcome variance due to multiple informants cannot be ignored, (b) consumer satisfaction is unrelated to pathology change, and (c) parent-reported perceived improvement ratings are more akin to satisfaction than to pathology change.

Comparative outcomes of emotionally disturbed children and adolescents in a system of services and usual care.

OBJECTIVE: This study compared six-month functional and symptom outcomes of children and adolescents with serious emotional disturbance who received services in an exemplary system of care with outcomes of children who received traditional care. The system of care offers a comprehensive and coordinated network of mental health and other necessary services. METHODS: The study used a randomized longitudinal experimental design. Baseline data on symptoms, functioning, and family characteristics were collected from 350 families selected from among those who sought services for children from community agencies in Stark County, Ohio. The families were randomly assigned to either the experimental group, which received services from the system of care, or the control group, which received usual care in the community. Six-month outcome measures of children's symptoms and functioning were compared for the two groups. RESULTS: Although access to care and the amount of care received increased under the system of care, no differences in clinical or functional outcomes were found between the group served in the system of care and the group who received usual care. CONCLUSIONS: The effects of systems of care are primarily limited to system-level outcomes such as access to and cost of care and do not appear to affect clinical outcomes such as functioning and symptoms.

Children's mental health in a continuum of care: clinical outcomes at 18 months for the Fort Bragg demonstration.

This article uses data collected at 18 months from the evaluation of the continuum of mental health services of the Fort Bragg Child and Adolescent Mental Health Demonstration Project (the Demonstration) to address the hypotheses that longer term follow-up (beyond 12 months) will show that the Demonstration is more effective and more successful for children with serious emotional disturbance (SED). The effects of the Demonstration are examined in comparison to those of traditional care by analyzing 12 key mental health outcomes with a random regression model, and the potential impact of attrition on results is explored. Results show neither hypothesis is supported, and the attrition analysis showed that the influence of missing data on the outcome analyses is negligible. Implications of these results for mental health policy are discussed.

Clinician reliability and accuracy in judging appropriate level of care.

Accurately assigning children to the most appropriate level of care is widely recognized as important. Managed care companies conduct utilization reviews in which they monitor the level of care to which clients are assigned using written placement criteria. However, no research has examined the ability of clinicians to perform this task. In the present study, 47 child and adolescent clinical profiles consisting of 48 variables were developed. Eighteen clinicians, trained to use their agency's level-of-care guidelines, made level-of-care decisions on these profiles. Their interjudge reliability in assigning a child to an appropriate level of care was close to zero (kappa = .07). There was a small, statistically significant correlation between client placement and actual placement, but chance-corrected agreement between client placement and actual placement was very low (kappa = .09). Implications of these findings for clinical research, practice, policy, and training are discussed.

Measuring quality in mental health services.

This article introduces the special issue on measuring quality of care. The overall purpose of this issue is to encourage an increased presence of evaluators in quality-of-care research. The sections of this special issue are devoted to conceptual issues concerning quality and the future of quality measurement. They include a description of current governmental and private efforts in measuring quality, a brief report on quality measurement efforts in Great Britain, and four quality-of-care research reports.

Validating quality indicators. Quality as relationship between structure, process, and outcome.

Theory and research have not kept pace with the growing interest in evaluating quality of mental health care, resulting in the use of unvalidated quality indicators. A framework for validating quality indicators is offered by which quality is viewed as the relationship between service structures, processes, and outcomes. Adoption of this framework will facilitate the measurement of quality using valid indicators and should be useful to agencies in their continuous quality improvement efforts. Valid information about the quality of mental health care services will help purchasers and consumers make more informed health care decisions.

A continuum of care. More is not always better.

This article describes an $80-million project designed to test whether a continuum of mental health and substance abuse services for children and adolescents is more cost-effective than services delivered in the more typical fragmented system. The study showed that an integrated continuum was successfully implemented that had better access, greater continuity of care, more client satisfaction, and treated children in less restrictive environments. However, the cost was higher, and clinical outcomes were no better than those at the comparison site. The article concludes that reform of mental health systems alone is unlikely to affect clinical outcomes. Cooperation is needed between mental health providers and researchers to better understand how to improve services delivered in the community.

Who gets hospitalized in a continuum of care?

OBJECTIVE: To compare children and adolescents hospitalized under a continuum of care with those hospitalized under traditional insurance coverage. METHOD: With comprehensive data, logistic regressions were used to predict hospitalization and to identify its determinants. RESULTS: As expected, the probability of being hospitalized was much higher under traditional care. In addition, the predictors of hospitalization differed by site. Accuracy of predictions was high. CONCLUSIONS: Different kinds of children were hospitalized under a continuum of care than under a traditional insurance system. Hospitalizations under both systems were highly predictable.