Project quality rating by experts and practitioners: experience with Preffi 2.0 as a quality assessment instrument.

Preffi 2.0 is an evidence-based Dutch quality assessment instrument for health promotion interventions. It is mainly intended for both planning and assessing one's own projects but can also be used to assess other people's projects (external use). This article reports a study on the reliability of Preffi as an external quality assessment instrument. Preffi is used to assess quality at three levels: (i) specific criteria, (ii) clusters of criteria and (iii) entire projects. The study compared Preffi-based assessments of 20 projects by three practitioners with their intuitive assessments of the same projects and with assessments by three experts, which were to be used as external criteria. The intuitive assessments only related to the cluster and project levels. Our main hypothesis was that intuitive assessments by practitioners would be less reliable and accurate than their Preffi-based assessments and the experts' assessments. On the whole, we failed to confirm this hypothesis: the experts' assessments proved less reliable and accurate than the practitioners' intuitive and Preffi-based assessments and differed too much from each other to be used as external criteria. The Preffi-based assessments by the practitioners had an acceptable generalizability coefficient (G) and accuracy (standard error of measurement). At the level of the entire project, two assessors are needed to produce sufficiently reliable and accurate assessments, whereas three are needed for assessment at cluster level. The study also showed that different assessors use different perspectives and base their assessment on a variety of aspects. This was regarded as inevitable and even useful by the assessors themselves. Discussions between assessors are important to achieve consensus. The article suggests some improvements to Preffi to further increase its reliability.

Implementation of a quality assurance instrument (Preffi 1.0) to improve the effectiveness of health promotion in The Netherlands.

This paper describes the design and outcomes of implementing Preffi 1.0, a quality assurance instrument for health promotion (HP) interventions, among Dutch HP professionals. The Preffi instrument promotes a systematic way of working that is driven by evidence, which is expected to lead to high-quality projects and better outcomes. Implementation interventions included nationwide activities aimed at awareness of and positive attitudes toward the instrument, and an intensive 5-day programme for a self-selected sample aimed to enhance self-efficacy and use. Effects of the nationwide activities were measured in two independent representative samples of Dutch HP professionals (N = 120 and 316, respectively), while a cohort design was used to measure the effects of the training programme. The Theory of Planned Behavior was used to guide both the interventions and the study. While the nationwide implementation activities raised awareness of Preffi and contributed to a more positive attitude towards the instrument, a significant increase in its use required a more intensive training programme. Attitude was especially influenced by the perceived value and necessity of a systematic approach, and the usefulness of the instrument. Health managers in The Netherlands failed to stimulate a systematic implementation of Preffi, even though the embeddedness of Preffi in the quality system of a HP team was found to be a major predictor of its implementation. The study showed that determinants of the implementation stages vary by stage.