How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations.

UNLABELLED: CONSTRUCT: The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low. BACKGROUND: Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear. APPROACH: Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates. RESULTS: Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats. CONCLUSIONS: The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.

Inter-rater Reliability of Criteria-Based Content Analysis of Children's Statements of Abuse.

The evaluation of children's statements of sexual abuse cases in forensic cases is critically important and must and reliable. Criteria-based content analysis (CBCA) is the main component of the statement validity assessment (SVA), which is the most frequently used approach in this setting. This study investigated the inter-rater reliability (IRR) of CBCA in a forensic context. Three independent raters evaluated the transcripts of 95 statements of sexual abuse. IRR was calculated for each criterion, total score, and overall evaluation. The IRR was variable for the criteria, with several being unsatisfactory. But high IRR was found for the total CBCA scores (Kendall's W=0.84) and for overall evaluation (Kendall's W=0.65). Despite some shortcomings, SVA remains a robust method to be used in the comprehensive evaluation of children's statements of sexual abuse in the forensic setting. However, the low IRR of some CBCA criteria could justify some technical improvements.

When the evaluated becomes evaluator: what can we learn from students' experiences during clerkships?

PURPOSE: To identify aspects that influence students' evaluation of the overall quality of clerkships and learning in clinical settings. METHOD: The authors analyzed 2,450 questionnaires dated 1997 through 2005 that evaluated clerkships of seven medical specialties (internal medicine, surgery, pediatrics, psychiatry, community medicine, emergency medicine, and obstetrics-gynecology). Students rated 22 questionnaire items addressing clerkships' global evaluation and domains related to structure, supervision, and clinical and problem-solving learning (PSL) activities using a five-point Likert scale. The authors performed statistical analysis using principal component analysis and regression analysis of items associated with students' global evaluation of clerkships. RESULTS: Correlation between clerkships' global ratings and ratings derived from the evaluation questionnaire was 0.871 (P < .0001). Clerkships' quality was mainly related to their organization, students' integration into clerkship, improvement of clinical skills, supervision, and residents' availability (r = 0.405; P < .0001). Among learning activities, opportunities for clinical practice predominated as the contributing factor to the overall perceived quality of most clerkships, but less than PSL activities in psychiatry (r = 0.070 versus 0.261, respectively; P < .001) and community medicine (r = 0.126 versus 0.298, respectively; P < .001); in surgery, both clinical practice and PSL activities contributed minimally to the clerkships' perceived quality (r = 0.150 and 0.148, respectively; P > .05). CONCLUSIONS: Factors influencing students' evaluation of a clerkship vary among medical specialties and depend not only on the teaching and teacher but also on the clerkship's organization, supervision, and learning activities. For clerkships where direct and multiple access to patients is more difficult, written case-based PSL activities proved complementary to direct patient encounter activities.