Experimental studies to improve the reliability and validity of regulatory judgments on health care in the Netherlands: a randomized controlled trial and before and after case study.

RATIONALE, AIMS AND OBJECTIVES: We examined the effect of two interventions on both the reliability and validity of regulatory judgments: adjusting the regulatory instrument and attending a consensus meeting. METHOD: We adjusted the regulatory instrument. With a randomized controlled trial (RCT) we examined the effect of the adjustments we made to the instrument. In the consensus meeting inspectors discussed cases and had to reach consensus about the order of the cases. We used a before and after case study to assess the effect of the consensus meeting. We compared the judgments assigned in the RCT with the unadjusted instrument with the judgments assigned with the unadjusted instrument after the consensus meeting. Moreover we explored the effect of increasing the number of inspectors per regulatory visit based on the estimates of the two interventions. RESULTS: The consensus meeting improved the agreement between inspectors; the variance between inspectors was smallest (0.03) and the reliability coefficient was highest (0.59). Validity is assessed by examining the relation between the assigned judgments and the corporate standard and expressed by a correlation coefficient. This coefficient was highest after the consensus meeting (0.48). Adjustment of the instrument did not increase reliability and validity coefficients. CONCLUSIONS: Participating in a consensus meeting improved reliability and validity. Increasing the number of inspectors resulted in both higher reliability and validity values. Organizing consensus meetings and increasing the number of inspectors per regulatory visit seem to be valuable interventions for improving regulatory judgments.

Evaluating instruments for regulation of health care in the Netherlands.

OBJECTIVES: Reliable and valid judgments are necessary for regulatory authorities to merit confidence from care institutions and society and preserve authority. Moreover, limited reliability and validity of regulatory judgments increase the risk of limited improvement of the quality of health care. The goal of the study is to obtain insight in (dis) advantages of different regulatory instruments for regulation of health care. METHOD: In this study, the reliability and validity of judgments generated by a lightly structured and highly structured regulatory instrument used by the Dutch Health Care Inspectorate are compared. RESULTS: Results indicate that the lightly structured instrument causes a large variety in discussed topics in regulatory visits: indicators pointing out potential risks in care are not always part of these discussions, by which incentives to improve care remain unjustly undone. Both types of instruments show variations in the meaning of judgments, indicating validity problems. CONCLUSION: The results of our study suggest that regulation of health care requires thorough appraisal of instruments. Several requirements are identified: first, an instrument that justifies the complexity of care with an accompanying explicit set of standards is necessary. Second, commitment of inspectors to the instrument is essential. And third, training of inspectors is indispensable.