Supporting international medical graduates in rural Australia: a mixed methods evaluation.

INTRODUCTION: In Australia, international medical graduates (IMGs) make a substantial contribution to rural medical workforces. They often face significant communication, language, professional and cultural barriers, in addition to the other challenges of rural clinical practice. The Gippsland Inspiring Professional Standards among International Experts (GIPSIE) program was designed to provide educational support to IMGs across a large geographical region using innovative educational methods to ultimately build capacity in the provision of rural medical education. GIPSIE offered 5 sessions over 3 months. Simulation-based training was a prominent theme and addressed clinical knowledge, attitudes and skills and included a range of activities (eg procedural skills training with benchtop models, management of the acutely ill patient with SimMan, patient assessment skills with simulated patients). Diverse clinical communication skills were explored (eg teamwork, handover, telephone, critical information). Audiovisual review of performance was enabled through the use of iPod nano devices. GIPSIE was underpinned by a website offering diverse learning resources. Content experts were invited to lead sessions that integrated knowledge and skills reflecting local practice. METHODS: IMGs were recruited from hospitals (n = 15) and general practices (n = 2) across the region. It was aimed to evaluate the impact of GIPSIE on the clinical practice of IMG participants. Evaluation measures included pre- and post-program 15 item multisource feedback (MSF), post-program questionnaires and, in order to address retention, telephone interviews exploring participants' responses 3 months after the program finished. RESULTS: Fifteen participants completed GIPSIE and rated the program highly, especially the simulation-based activities with feedback and later audiovisual review on iPods and the GIPSIE website. Suggestions were made to improve several aspects of the program. Participants reported increased knowledge, skills and professionalism after the program. Although overall MSF scores showed no statistically significant changes, there were positive directional changes for the items 'technical skills appropriate to current practice', 'willingness and effectiveness when teaching/training colleagues' and 'communication with carers and family'. These developments were also supported in free-text comments. Learning was reported to be sustained 3 months after the program. CONCLUSIONS: GIPSIE was highly valued by participants who reported improvements in clinical knowledge and skills. A range of professional issues were raised and addressed. GIPSIE seemed to provide a platform for further development. Although new to many participants, simulation was embraced as an educational method. The relationship between regional clinicians and the medical school was pivotal to success. A feature of the study was tracking improvements in clinical practice as a consequence of participating in the GIPSIE program. Future work needs to focus on further promoting the transfer of learning to the workplace. However the sustainability of these programs requires significant commitment.

Implementation of a multi-level evaluation strategy: a case study on a program for international medical graduates.

Evaluation of educational interventions is often focused on immediate and/or short-term metrics associated with knowledge and/or skills acquisition. We developed an educational intervention to support international medical graduates working in rural Victoria. We wanted an evaluation strategy that included participants' reactions and considered transfer of learning to the workplace and retention of learning. However, with participants in distributed locations and limited program resources, this was likely to prove challenging. Elsewhere, we have reported the outcomes of this evaluation. In this educational development report, we describe our evaluation strategy as a case study, its underpinning theoretical framework, the strategy, and its benefits and challenges. The strategy sought to address issues of program structure, process, and outcomes. We used a modified version of Kirkpatrick's model as a framework to map our evaluation of participants' experiences, acquisition of knowledge and skills, and their application in the workplace. The predominant benefit was that most of the evaluation instruments allowed for personalization of the program. The baseline instruments provided a broad view of participants' expectations, needs, and current perspective on their role. Immediate evaluation instruments allowed ongoing tailoring of the program to meet learning needs. Intermediate evaluations facilitated insight on the transfer of learning. The principal challenge related to the resource intensive nature of the evaluation strategy. A dedicated program administrator was required to manage data collection. Although resource-intensive, we recommend baseline, immediate, and intermediate data collection points, with multi-source feedback being especially illuminating. We believe our experiences may be valuable to faculty involved in program evaluations.