Attitudes of health professionals to using routinely collected clinical data for performance feedback and personalised professional development.

OBJECTIVES: To learn the attitudes of health professionals, health informaticians and information communication technology professionals to using data in electronic health records (eHRs) for performance feedback and professional development. DESIGN: Qualitative research in a co-design framework. Health professionals' perceptions of the accessibility of data in eHRs, and barriers to and enablers of using these data in performance feedback and professional development were explored in co-design workshops. Audio recordings of the workshops were transcribed, de-identified, and thematically analysed. SETTING, PARTICIPANTS: A total of nine co-design workshops were held in two major public hospitals in Sydney: three for nursing staff (ten participants), three for doctors (15 participants), and one each for information communication technology professionals (six participants), health informaticians (four participants), and allied health professionals (13 participants). MAIN OUTCOME MEASURES: Key themes related to attitudes of participants to the secondary use of eHR data for improving health care practice. RESULTS: Six themes emerged from the discussions in the workshops: enthusiasm for feeding back clinical data; formative rather than punitive use; peer comparison, benchmarking, and collaborative learning; data access and use; capturing complex clinical narratives; and system design challenges. Barriers to secondary use of eHR data included access to information, measuring performance on the basis of eHR data, and technical questions. CONCLUSIONS: Our findings will inform the development of programs designed to utilise routinely collected eHR data for performance feedback and professional development.

Bridging the gap: a five stage approach for developing specialty-specific entrustable professional activities.

BACKGROUND: Entrustable Professional Activities (EPAs) are increasingly used as a focus for assessment in graduate medical education (GME). However, a consistent approach to guide EPA design is currently lacking, in particular concerning the actual content (knowledge, skills and attitude required for specific tasks) for EPAs. This paper describes a comprehensive five stage approach, which was used to develop two specialty-specific EPAs in emergency medicine focused on the first year of GME. METHODS: The five stage approach was used to gain consensus on the task, content and entrustment scale for two specialty-specific EPAs in emergency medicine. The participants consisted of twelve clinical supervisors working in the emergency department. The five stages were: 1) Selecting the EPA topic; 2) Developing the EPA content by collecting data from participants using focus group and individual interviews; 3) Drafting the EPAs based on analysis of collected data; 4) Seeking feedback on the draft EPAs from the participants and other stakeholders; 5) Refining and finalising the EPAs based on feedback. RESULTS: Two specialty-specific EPAs were developed using the five stage approach. The participants reached consensus on the specific tasks and criteria for performance for the two EPAs. They also agreed that both day-to-day (ad hoc) and formal (summative) entrustment decisions were put into practice through the intensity of supervision provided to PGY1 doctors. As a result, a three level entrustment and supervision scale consisting of direct active, indirect active, passive was developed reflecting the shift in the intensity of supervision from close supervision to minimal supervision. CONCLUSIONS: The five stage approach described in this paper was used successfully to develop two specialty-specific EPAs in emergency medicine along with a three level entrustment scale.We propose that the five stage approach is transferable to a range of medical training contexts to design specialty-specific EPAs.

A review of prevocational medical trainee assessment in New South Wales.

OBJECTIVE: To evaluate the effectiveness of formal assessments of prevocational medical trainees (ie, interns, first-year residents and international medical graduates undergoing supervised training). DESIGN AND SETTING: Retrospective review of structured assessment forms that are completed by all prevocational trainees and their supervisors in New South Wales public hospitals. We reviewed the first 3390 assessment forms, representing assessments of 1072 trainees at 43 training sites (83% of all prevocational trainees in NSW) since January 2009. MAIN OUTCOME MEASURES: (i) Trainee ratings on 19 assessment items by self-assessment and by supervisor assessment; and (ii) quantity and quality of written comments provided in assessments. RESULTS: At the end of term, 43% of trainees assessed themselves as performing "at expected level" on all 19 rating items. Nearly 99% of trainees were assessed by their supervisors as performing at or above the expected level on all assessment items. Written comments by supervisors were generally short and encouraging, but lacked specific feedback that trainees might use to guide improvements in performance. CONCLUSIONS: As currently used by trainees and supervisors, the assessment forms may underreport trainee underperformance, do not discriminate strongly between different levels of performance of trainees or the training system, and do not provide trainees with enough specific feedback to guide their professional development.