Coming to grips with negative evidence for the Comprehensive System for the Rorschach: a comment on Gacono, Loving, and Bodholdt; Ganellen; and Bornstein.

The Comprehensive System (CS; Exner, 1991, 1993) for the Rorschach is currently engulfed in controversy. This comment article responds to 3 articles by Rorschach proponents in this issue of the Journal of Personality Assessment. Contrary to the claims of Gacono, Loving, and Bodholdt (this issue), CS scores do not bear a well-demonstrated relationship to psychopathy, antisocial personality disorder, or conduct disorder. Contrary to the claims of Ganellen (this issue), both the original and the revised CS Depression Index (Exner, 1993) bear little or no relationship to depression diagnoses. Furthermore, the scoring reliability of some CS scores is problematic. Although we agree with Bornstein (this issue) that Rorschach scores generally bear little or no relation to psychiatric diagnoses or self-report questionnaires, we believe this lack of relationship tends to disconfirm hypotheses concerning the validity of the Rorschach. In the spirit of the philosopher Sir Karl Popper, the Rorschach community should not minimize negative evidence or engage in post hoc arguments to immunize the CS against falsification.

Cognitive complexity and the validity of clinicians' judgments.

Efforts to explain why some neuropsychologists are more accurate than others have been unsuccessful. In the present study, the relation between the cognitive complexity of clinicians and the validity of their judgments was investigated. Forty-two neuropsychologists completed a measure of cognitive complexity. The hypothesis that neuropsychologists who are high in cognitive complexity will be more accurate than those who are low in cognitive complexity was not supported. It is not clear why cognitive complexity was not significantly related to the validity of diagnostic ratings: One explanation is that the measure of cognitive complexity used in this study was inadequate but another possibility is that neuropsychologists who are low in cognitive complexity may still be able to achieve high levels of accuracy if they attend to normative data when they interpret test results.

Toward a resolution of the Rorschach controversy.

Comments are made about the articles comprising the first round of the Special Series on the Rorschach. G. Stricker and J. R. Gold (1999) and D. J. Viglione (1999) praised the Rorschach, but they consistently failed to cite negative findings. R. M. Dawes (1999) obtained results that provide modest support for the Rorschach, but one of his data sets is flawed. J. B. Hiller, R. Rosenthal, R. F. Bornstein, D. T. R. Berry, and S. Brunell-Neuleib (1999) reported the results of a meta-analysis, but, among other problems, their coders were not blind to the results of all the studies. J. Hunsley and J. M. Bailey (1999) made a strong case for concluding that there is no scientific basis for using the Rorschach. Recommendations are made for resolving the Rorschach controversy.

The Rorschach test in clinical diagnosis: a critical review, with a backward look at Garfield (1947).

The present article comments on a classic study by Garfield (1947) then reviews research on the Rorschach and psychiatric diagnoses. Despite a few positive findings, the Rorschach has demonstrated little validity as a diagnostic tool. Deviant verbalizations and bad form on the Rorschach, and indices based on these variables, are related to Schizophrenia and perhaps to Bipolar Disorder and Schizotypal Personality Disorder. Patients with Borderline Personality Disorder also seem to give an above-average number of deviant verbalizations. Otherwise the Rorschach has not shown a well-demonstrated relationship to these disorders or to Major Depressive Disorder, Posttraumatic Stress Disorder (PTSD), anxiety disorders other than PTSD, Dissociative Identity Disorder, Dependent, Narcissistic, or Antisocial Personality Disorders, Conduct Disorder, or psychopathy.

Projective techniques and the detection of child sexual abuse.

Projective techniques (e.g., the Rorschach, Human Figure Drawings) are sometimes used to detect child sexual abuse. West recently conducted a meta-analysis on this topic, but she systematically excluded nonsignificant results. In this article, a reanalysis of her data is presented. The authors conclude that projective techniques should not be used to detect child sexual abuse. Many of the studies purportedly demonstrating validity are flawed, and none of the projective test scores have been well replicated.

The Scientific Status of Projective Techniques.

Although projective techniques continue to be widely used in clinical and forensic settings, their scientific status remains highly controversial. In this monograph, we review the current state of the literature concerning the psychometric properties (norms, reliability, validity, incremental validity, treatment utility) of three major projective instruments: Rorschach Inkblot Test, Thematic Apperception Test (TAT), and human figure drawings. We conclude that there is empirical support for the validity of a small number of indexes derived from the Rorschach and TAT. However, the substantial majority of Rorschach and TAT indexes are not empirically supported. The validity evidence for human figure drawings is even more limited. With a few exceptions, projective indexes have not consistently demonstrated incremental validity above and beyond other psychometric data. In addition, we summarize the results of a new meta-analysis intended to examine the capacity of these three instruments to detect child sexual abuse. Although some projective instruments were better than chance at detecting child sexual abuse, there were virtually no replicated findings across independent investigative teams. This meta-analysis also provides the first clear evidence of substantial file drawer effects in the projectives literature, as the effect sizes from published studies markedly exceeded those from unpublished studies. We conclude with recommendations regarding the (a) construction of projective techniques with adequate validity, (b) forensic and clinical use of projective techniques, and (c) education and training of future psychologists regarding projective techniques.

Introduction to the special section on the use of computers for making judgments and decisions.

Computers have the potential to greatly increase the accuracy of judgments and decisions that are made in the area of psychological assessment. The validity of present-day computer programs are described in the first 2 articles of this Special Section. Both computer-based test interpretation programs and mechanical prediction rules (e.g., statistical prediction rules) are described. Limitations of present-day computer programs are described in the 3rd article along with methodological recommendations for building more powerful rules. In the 4th article, a statistical analysis (neural network modeling) is described that may allow us to build better rules.

Computers will become increasingly important for psychological assessment: not that there's anything wrong with that!

Though one can expect that computer programs will become increasingly important for psychological assessment, current automated assessment programs and statistical-prediction rules are of limited value. Validity has not been clearly established for many automated assessment programs. Statistical-prediction rules are of limited value because they have typically been based on limited information that has not been demonstrated to be optimal and they have almost never been shown to be powerful. Recommendations are made for building and evaluating new computer programs. Finally, comments are made about the ethics of using computers to make judgments.

Call for a moratorium on the use of the Rorschach Inkblot Test in clinical and forensic settings.

A call is issued for a moratorium on the use of the Rorschach Inkblot Test in clinical and forensic (but not research) settings. The moratorium should last until we have determined which Rorschach scores are valid and which ones are invalid. Unfortunately, for most Rorschach scores, results from meta-analyses have been uninformative. Also, incremental validity has not been studied for most Rorschach scores. Furthermore, positive findings for Rorschach scores have rarely been independently replicated. Finally, selective reporting of results has been a problem: Some investigators report significant results but not nonsignificant results. The magnitude of this problem has not been determined. Unless a moratorium is adopted, clinicians will continue to interpret invalid scores along with valid scores.

Judgement research and neuropsychological assessment: a narrative review and meta-analyses.

The authors review judgment research in the area of neuropsychological assessment. Topics include the reliability and validity of judgments, the appropriateness of confidence ratings, the value of training and experience, the cognitive processes of neuropsychologists, and the use of decision aids, including automated assessment programs and statistical prediction rules. Most of the research is on reliability and validity. The results indicate that neuropsychologists frequently make reliable and moderately valid judgments. However, for several tasks, experimenters have not studied the reliability and validity of judgments.

Clinical judgment, clinical training, and professional experience.

Reviews studies on training, experience, and clinical judgment. The results on the validity of judgments generally fail to support the value of on-the-job experience in mental health fields. The validity results do provide limited support for the value of training. Other results suggest that experienced clinicians are better than less experienced judges at knowing which of their judgments are likely to be correct and which are likely to be wrong. Reasons why clinicians have trouble learning from experience are given. Recommendations are made for improving training and clinical practice.

A conjoint measurement analysis of clinical predictions.

A lack of impressive theories in clinical psychology has been attributed to an excessive reliance on significance testing. Conjoint measurement is a worthwhile alternative to significance testing. Conjoint measurement determines whether data obey the ordinal properties of a model that contains main effect terms, interaction terms, or a combination of the two. Unlike hypothesis testing, with conjoint measurement an increase in statistical power leads to a greater risk of refuting a theory. In this study, conjoint measurement was used to describe how systems-oriented therapists predict the occurrence of suicide. Although the systems approach emphasizes the interaction between an individual and a system, all 10 clinicians were described by an additive model.