Rating of physiotherapy student clinical performance in a paediatric setting: are assessors consistent in their rating of a simulated clinical student performance?

BACKGROUND: During workplace-based clinical placements, best practice assessment states students should expect consistency between assessors rating their performance. To assist clinical educators (CEs) to provide consistent assessment of physiotherapy student performance, nine paediatric vignettes depicting various standards of simulated student performance, as assessed by the Assessment of Physiotherapy Practice (APP), were developed. The APP defines adequate on the global rating scale (GRS) as the minimally acceptable standard for an entry-level physiotherapist. The project aimed to evaluate consistency of paediatric physiotherapy educators assessing simulated student performance using the APP GRS. METHODS: Three paediatric scenarios representing neurodevelopment across three age ranges, infant, toddler and adolescent, were developed and scripted that depicted a 'not adequate', 'adequate' and 'good-excellent' performance based on the APP GRS. An expert panel (n = 9) conducted face and content validation. Once agreement was reached for all scripts, each video was filmed. A purposive sample of physiotherapists providing paediatric clinical education in Australia were invited to participate in the study. Thirty-five CEs, with minimum 3-years clinical experience and had supervised a student within the past year, were sent three videos at four-week intervals. Videos depicted the same clinical scenario, however performance varied with each video. Participants rated the performance on the four categories: 'not adequate', 'adequate', 'good' and 'excellent' Consistency among raters was assessed using percentage agreement to establish reliability. RESULTS: The vignettes were assessed a combined total of 59 times. Across scenarios, percentage agreement at the not adequate level was 100%. In contrast, the adequate scenarios for the Infant, Toddler and Adolescent video failed to meet the 75% agreement level. However, when combining adequate or good-excellent, percentage agreement was > 86%. The study demonstrated strong consensus when comparing not adequate to adequate or better performance. Importantly, no performance scripted as not adequate was passed by any assessor. CONCLUSIONS: Experienced educators demonstrate consistency in identifying not adequate from adequate or good-excellent performance when assessing a simulated student performance using the APP. Recommendation for practice: These validated video vignettes will be a valuable training tool to improve educator consistency when assessing student performance in paediatric physiotherapy.

Acceptance of the 'Assessment of Physiotherapy Practice (Chinese)' as a standardised evaluation of professional competency in Chinese physiotherapy students: an observational study.

BACKGROUND: Development of an entry-level physiotherapy curriculum in China currently follows the World Confederation for Physical Therapy (WCPT) guidelines, however there is no standard, validated, assessment tool for physiotherapy practice in use in China. This article reports the process of translation of the "Assessment of Physiotherapy Practice" (APP), a validated assessment instrument adopted by all universities in Australia and New Zealand, into Chinese (APP-Chinese) and its implementation by Chinese physiotherapy clinical educators (CEs) and students during clinical placements. METHODS: The process of forward and backward translation of the APP was undertaken by a team of academics from universities in Shanghai, Hong Kong, United States and Australia. An APP-Chinese version was produced and used for assessment of the clinical performance of 4th year students at a university in Shanghai. Feedback on the implementation of the APP-Chinese was solicited from students and CEs using the same two questionnaires employed to assess implementation of the original APP. RESULTS: All CEs agreed that the rules used to score the APP-Chinese were helpful in assessing student performance. Over 90% of the CEs considered the APP-Chinese was pragmatic for use in the clinical environment in China. All students agreed with the rating of their performance on the APP-Chinese marked by their educators, and that the performance indicators were useful in guiding their expected performance behaviour. CONCLUSION: The APP-Chinese is the first standardised assessment tool for evaluation of clinical performance of physiotherapy students in China and was shown to be well accepted by both students and CEs in the clinical education unit and university involved in this study.

Fostering patient uptake of recommended health services and self-management strategies for musculoskeletal conditions: A Delphi study of clinician attributes.

INTRODUCTION: Successful management of musculoskeletal conditions depends on active patient engagement and uptake of recommended health services and self-management strategies. Clinicians have a strong influence on patient uptake behaviours. Both clinicians and educators need to recognise the clinician's influence on patient uptake as a specific clinical skillset to be professionally developed. To inform professional development strategies this study explored priority clinician attributes that underpin the clinical skillset of fostering patient uptake. METHODS: A three-round Delphi process engaged relevant stakeholders including a professional panel (clinicians, health managers, education providers) and a patient panel. Panel members deliberated and reached consensus regarding key attributes required by allied health clinicians who manage patients with musculoskeletal disorders to optimize patient uptake behaviours. In the final round, panel members rated the importance of each attribute on a numerical rating scale. RESULTS: Overall 12 attributes were finalised. Both the professional and the patient panel provided a high rating of importance for all finalised attributes with 'patient centred communication' rated the highest importance (median scores 9.5-10/10) and 'contemporary electronics and media' rated the lowest (median scores 6-7/10). CONCLUSIONS: There appears to be agreement on a basic inventory of clinician attributes which positively influence patient uptake when managing musculoskeletal conditions. Professional development and training programs for clinicians managing musculoskeletal conditions may need to consider discipline relevant aspects of these attributes to advance the development of clinicians in this aspect of professional practice to attain better patient outcomes.

Simulation-based education improves student self-efficacy in physiotherapy assessment and management of paediatric patients.

BACKGROUND: The Australian Physiotherapy Council mandates that physiotherapy clinical education be sufficient to produce graduates who are competent to practice across the lifespan. Due to a lack of opportunities for paediatric clinical placements, there is a risk of graduates not having the opportunity to develop competency in paediatric physiotherapy. To address this risk, simulation-based education (SBE) has been proposed as an educational strategy to address the placement shortfall. Despite encouraging evidence for its use in physiotherapy education, there is limited evidence supporting its use specifically in paediatric populations. The aims of this research were to investigate the effect of SBE on student self-efficacy in the physiotherapy assessment and management of paediatric clients, and to determine student satisfaction with SBE as a learning strategy. METHODS: Three interactive SBE sessions were run during the undergraduate paediatric physiotherapy unit at the campus of one Australian university. Self-efficacy was surveyed before and after each session, to determine confidence in clinical skills, clinical decision-making, treatment preparation and planning, communication skills; evaluating and modifying interventions, and interprofessional practice. Student satisfaction with SBE as a learning strategy was surveyed after the final SBE session. RESULTS: For the 164 participants included in this study, self-efficacy survey response rate varied from 77 to 96% for each session. Significant increases in mean student self-efficacy were recorded for all questions (p <  0.001). A total of 139 (85%) responded to the learning reactionnaire with 78.6% indicating they were very satisfied with SBE as a learning strategy. Written comments from 41 participants identified 'experience' as the primary theme. CONCLUSION: SBE had a significant positive effect on student self-efficacy in the physiotherapy assessment and management of paediatric patients. Students also perceived SBE to be a valuable learning experience. Future research is needed to investigate whether the improvement in self-efficacy achieved through SBE translates into improved student performance during workplace-based clinical placements.

Physiotherapy students' DiSC behaviour styles can be used to predict the likelihood of success in clinical placements.

BACKGROUND: Behaviour can be defined as the internally coordinated responses (actions or inactions) of whole living organisms (individuals or groups) to internal and/or external stimuli, excluding responses more easily understood as developmental changes. Unlike personality traits, that are thought to be biologically consistent, behaviour, through the application of cognition and reasoning is open to change across time and circumstance, although most humans will display preferred ways of behaving. The objective of this study was to: i) identify the behaviour styles of physiotherapy students and investigate if there is a relationship (predictive or otherwise) between students' unique behaviour patterns and their clinical placement grades and; ii) examine if this relationship differs when student's in a Master's level program as well as student's in a Bachelor's level program are explored separately. METHODS: This cross-sectional study with 132 (F = 78, M = 54) physiotherapy students was conducted across two Australian university settings. Measures included Everything DiSC Workplace profile, Assessment of Physiotherapy Practice (APP). RESULTS: Physiotherapy students (n = 133) profiled the following ways: Dominance (D) style n = 20 (15%), Influence (i) style n = 33 (25%), Steadiness (S) style n = 36 (27%) and Conscientiousness (C) n = 44 (33%). Students with the individual DiSC styles of i and Conscientiousness / Steadiness (CS) were in the lowest APP quartile for clinical grades and the D style was in the highest quartile. Binary logistic regressions revealed students with an i DiSC style had 3.96 times higher odds, and students with a CS DiSC style had 4.34 times higher odds, of failing a clinical placement. When explored independently, the same trend remained for Master's level students. Bachelor's level students with DiSC styles of S and C had failed placements, however these styles were not significantly associated with failure (DiSC S Style: Exp(B) 1.667, p = 0.713 (CI: 0.109 to 25.433), DiSC C Style: Exp(B) 11.00, p = 0.097 (CI: 0.646 to 187.166)). CONCLUSION: Physiotherapy students with DiSC styles i and CS appear to be more likely to fail physiotherapy clinical placements. Further research with larger undergraduate samples is required to establish if relations differ for undergraduate versus postgraduate students.

Rating of physiotherapy student clinical performance: is it possible to gain assessor consistency?

BACKGROUND: Reliable interpretation of the Assessment of Physiotherapy Practice (APP) tool is necessary for consistent assessment of physiotherapy students in the clinical setting. However, since the APP was implemented, no study has reassessed how consistently a student performance is evaluated against the threshold standards. Therefore, the primary aim of this study was to determine the consistency among physiotherapy educators when assessing a student performance using the APP tool. METHODS: Physiotherapists (n = 153) from Australia with a minimum 3 years clinical experience and who had supervised a physiotherapy student within the past 12-months were recruited. Three levels of performance (not adequate, adequate, good/excellent) were scripted and filmed across outpatient musculoskeletal, neurorehabilitation, cardiorespiratory and inpatient musculoskeletal. In the initial phase of the study, scripts were written by academic staff and reviewed by an expert panel (n = 8) to ensure face and content validity as well as clinical relevance prior to filming. In the second phase of the study, pilot testing of the vignettes was performed by clinical academics (n = 16) from Australian universities to confirm the validity of each vignette. In the final phase, study participants reviewed one randomly allocated vignette, in their nominated clinical area and rated the student performance including a rationale for their decision. Participants were blinded to the performance level. Percentage agreement between participants was calculated for each vignette with an a priori percentage agreement of 75% considered acceptable. RESULTS: Consensus among educators across all areas was observed when assessing a performance at either the 'not adequate' (97%) or the 'good/excellent' level (89%). When assessing a student at the 'adequate' level, consensus reduced to 43%. Similarly, consensus amongst the 'not adequate' and 'good/excellent' ranged from 83 to 100% across each clinical area; while agreement was between 33 and 46% for the 'adequate' level. Percent agreement between clinical educators was 89% when differentiating 'not adequate' from 'adequate' or better. CONCLUSION: Consistency is achievable for 'not adequate' and 'good/excellent' performances, although, variability exists at an adequate level. Consistency remained when differentiating an 'adequate' or better from a 'not adequate' performance.

Using Interprofessional Learning for Continuing Education: Development and Evaluation of the Graduate Certificate Program in Health Professional Education for Clinicians.

INTRODUCTION: Health professionals may be expert clinicians but do not automatically make effective teachers and need educational development. In response, a team of health academics at an Australian university developed and evaluated the continuing education Graduate Certificate in Health Professional Education Program using an interprofessional learning model. METHODS: The model was informed by Collins interactional expertise and Knowles adult learning theories. The team collaboratively developed and taught four courses in the program. Blended learning methods such as web-based learning, face-to-face workshops, and online discussion forums were used. Twenty-seven multidisciplinary participants enrolled in the inaugural program. Focus group interview, self-report questionnaires, and teacher observations were used to evaluate the program. RESULTS: Online learning motivated participants to learn in a collaborative virtual environment. The workshops conducted in an interprofessional environment promoted knowledge sharing and helped participants to better understand other discipline roles, so they could conduct clinical education within a broader health care team context. Work-integrated assessments supported learning relevance. The teachers, however, observed that some participants struggled because of lack of computer skills. DISCUSSION: Although the interprofessional learning model promoted collaboration and flexibility, it is important to note that consideration be given to participants who are not computer literate. We therefore conducted a library and computer literacy workshop in orientation week which helped. An interprofessional learning environment can assist health professionals to operate outside their "traditional silos" leading to a more collaborative approach to the provision of care. Our experience may assist other organizations in developing similar programs.

Measuring safety climate in acute hospitals: Rasch analysis of the safety attitudes questionnaire.

BACKGROUND: The Safety Attitudes Questionnaire (SAQ) is commonly used to assess staff perception of safety climate within their clinical environment. The psychometric properties of the SAQ have previously been explored with confirmatory factor analysis and found to have some issues with construct validity. This study aimed to extend the psychometric evaluations of the SAQ by using Rasch analysis. METHODS: Assessment of internal construct validity included overall fit to the Rasch model (unidimensionality), response formats, targeting, differential item functioning (DIF) and person-separation index (PSI). RESULTS: A total of 420 nurses completed the SAQ (response rate 60 %). Data showed overall fit to a Rasch model of expected item functioning for interval scale measurement. The questionnaire demonstrated unidimensionality confirming the appropriateness of summing the items in each domain. Score reliabilities were appropriate (internal consistency PSI 0.6-0.8). However, participants were not using the response options on the SAQ in a consistent manner. All domains demonstrated suboptimal targeting and showed compromised score precision towards higher levels of safety climate (substantial ceiling effects). CONCLUSION: There was general support for the reliability of the SAQ as a measure of safety climate although it may not be able to detect small but clinically important changes in safety climate within an organisation. Further refinement of the SAQ is warranted. This may involve changing the response options and including new items to improve the overall targeting of the scale. TRIAL REGISTRATION: This study was registered with the Australian New Zealand Clinical Trials Registry, number ACTRN12611000332921 (21 March 2011).

Identifying educator behaviours for high quality verbal feedback in health professions education: literature review and expert refinement.

BACKGROUND: Health professions education is characterised by work-based learning and relies on effective verbal feedback. However the literature reports problems in feedback practice, including lack of both learner engagement and explicit strategies for improving performance. It is not clear what constitutes high quality, learner-centred feedback or how educators can promote it. We hoped to enhance feedback in clinical practice by distinguishing the elements of an educator's role in feedback considered to influence learner outcomes, then develop descriptions of observable educator behaviours that exemplify them. METHODS: An extensive literature review was conducted to identify i) information substantiating specific components of an educator's role in feedback asserted to have an important influence on learner outcomes and ii) verbal feedback instruments in health professions education, that may describe important educator activities in effective feedback. This information was used to construct a list of elements thought to be important in effective feedback. Based on these elements, descriptions of observable educator behaviours that represent effective feedback were developed and refined during three rounds of a Delphi process and a face-to-face meeting with experts across the health professions and education. RESULTS: The review identified more than 170 relevant articles (involving health professions, education, psychology and business literature) and ten verbal feedback instruments in health professions education (plus modified versions). Eighteen distinct elements of an educator's role in effective feedback were delineated. Twenty five descriptions of educator behaviours that align with the elements were ratified by the expert panel. CONCLUSIONS: This research clarifies the distinct elements of an educator's role in feedback considered to enhance learner outcomes. The corresponding set of observable educator behaviours aim to describe how an educator could engage, motivate and enable a learner to improve. This creates the foundation for developing a method to systematically evaluate the impact of verbal feedback on learner performance.

Validation of the Australian Nursing Standards Assessment Tool (ANSAT): A pilot study.

BACKGROUND: Workplace-based learning and assessment are essential elements of all nursing education programs. There is, however, limited evidence of validity and reliability available regarding assessment instruments, based on the professional competency standards for nursing in Australia. OBJECTIVE: The aims of this project were to advance the assessment properties of a new instrument, the Australian Nursing Standards Assessment Tool (ANSAT) and investigate the acceptability of this instrument when applied to the evaluation of the professional competence of nursing students in authentic practice settings. METHODS: A validation study of ANSAT was conducted by 23 clinical assessors from two universities, completing 220 instruments for second and third year undergraduate nursing students undertaking clinical placement in an authentic practice setting. 'Think Aloud' interviews and a post-test survey provided additional data to determine the validity and acceptability of the ANSAT. The pilot study occurred in parallel with completion of existing university assessment instruments. RESULTS: Principal components analysis extracted one factor: professional practice competence. Comparison of total instrument scores between year levels demonstrated a significant difference in each of the clinical domains (p=0.000), suggesting that the instrument is sensitive to differing levels of performance across different year levels. The ANSAT demonstrated high internal consistency with a Cronbach's alpha coefficient of 0.976. Post-test evaluation completed by assessors demonstrated high usability and acceptability for use in common practice settings. DISCUSSION: Results of the pilot study provided preliminary support for the ANSAT instrument. It is recommended that testing of the instrument be undertaken on a larger cohort to ensure the generalisability of the findings to other nursing populations. CONCLUSION: There is a need for valid and reliable instruments that assess the performance of nursing in the authentic practice setting. With the addition of supportive behavioural cues, the ANSAT enables clarity, consistency and collaboration in workplace-based assessment.

Assessing Physical Therapy Students' Performance during Clinical Practice.

PURPOSE: To assess the feasibility and acceptability of using the Assessment of Physiotherapy Practice (APP) instrument to assess physiotherapy students' clinical competencies. METHODS: A convenience sample of clinical educators (CEs) and students from the University of British Columbia (UBC) in Vancouver, Canada, completed the instrument currently in use, the Physical Therapist Clinical Performance Instrument (PT-CPI), and the APP. A cross-sectional survey of CEs and physiotherapy students was conducted from 2011 to 2012; the survey included questions worded to elicit opinions about the two instruments when used in the clinical environment with students at different stages of training. Questions addressed various aspects of the instruments, including ease of use, provision of feedback, and completion time. RESULTS: Data were analyzed from 63 CEs from a variety of settings; sufficient data were recorded on 71 student PT-CPI and APP forms. A grading comparison between the PT-CPI and the APP demonstrated equivalence of entry-to-practice standard. Mean completion time was 80 (SD 53) minutes for the PT-CPI and 23 (13) minutes for the APP; mean time difference was 57 (95% CI, 39-75). Students would prefer (82%) that the APP be used to provide feedback and assess their performance on clinical placements. CONCLUSIONS: It is feasible and acceptable to use the APP to assess physiotherapy students' clinical competencies at the University of British Columbia.

Objectif : Évaluer la faisabilité d'utiliser les instruments d'évaluation de la pratique de la physiothérapie (EPP) et leur accessibilité afin d'évaluer les compétences cliniques des étudiants en physiothérapie. Méthodes : Un échantillon de commodité constitué de formateurs cliniciens (FC) et d'étudiants de l'Université de la Colombie-Britannique (UCB) à Vancouver, au Canada, a rempli l'instrument utilisé actuellement, soit l'Instrument d'évaluation du rendement clinique des physiothérapeutes (ERC-PT) et l'EPP. On a réalisé, de 2011 à 2012, un sondage transversal auprès de formateurs cliniciens et d'étudiants en physiothérapie qui comportait des questions visant à réunir des opinions sur les deux instruments utilisés dans l'environnement clinique avec des étudiants à différents stades de la formation. Les questions portaient sur divers aspects des instruments, y compris la facilité d'utilisation, la formulation de commentaires et le temps nécessaire pour y répondre. Résultats : On a analysé des données provenant de 63 formateurs cliniciens de toutes sortes de contextes et 71 formulaires d'ERC-PT et d'EPP d'étudiants contenaient suffisamment de données. Une comparaison des cotes entre l'instrument ERC-PT et l'EPP a démontré une équivalence des normes au niveau débutant. Il a fallu en moyenne 80 (ET 53) minutes pour remplir l'ERC-PT et 23 (13) minutes pour répondre à l'EPP. La différence moyenne s'est établie à 57 (IC à 95%, 39 à 75). Les étudiants préféreraient (82%) qu'on utilise l'EPP pour formuler des commentaires et évaluer leur rendement au cours des stages cliniques. Conclusions : Il est possible et acceptable d'utiliser l'EPP pour évaluer les compétences cliniques des étudiants en physiothérapie à l'Université de la Colombie-Britannique.

The assessment of physiotherapy practice (APP) is a reliable measure of professional competence of physiotherapy students: a reliability study.

QUESTION: What is the inter-rater reliability of the Assessment of Physiotherapy Practice (APP) instrument, and what is the error associated with individual scores? DESIGN: Cross-sectional inter-rater reliability study. Thirty pairs of clinical educators each assessed one student after observing student practice over a 5-week clinical placement. PARTICIPANTS: Sixty clinical educators from five Australian universities formed 30 independent pairs of assessors. OUTCOME MEASURES: Each pair completed two independent assessments of one student, providing 60 completed APP assessments and an associated Global Rating Scale score for analysis. ANALYSIS: Correlational coefficients and measurement error expressed in APP scale units were computed to provide a comprehensive analysis of the likely utility of APP scores and to enable score and change score interpretation. RESULTS: Percentage of agreement between assessors for each item ranged from 56% (Item 19, evidence-based practice) to 83% (Item 20, risk management) and across all items averaged 70% (SD 7). The ICC(2,1) was 0.92 (95% CI 0.84 to 0.96) for the total APP score and 0.72 (95% CI 0.50 to 0.86) for the Global Rating Scale. The standard error of measurement for the total score (scale width 0-80) was 3.2 APP points and the MDC(90) was 7.86 representing 9% of the scale width. Bland-Altman analyses identified no systematic differences between raters. CONCLUSION: Clinical educators demonstrated a high level of reliability when using the APP instrument to assess physiotherapy students' level of professional competence in workplace-based practice.

The Assessment of Physiotherapy Practice (APP) is a valid measure of professional competence of physiotherapy students: a cross-sectional study with Rasch analysis.

QUESTION: Is the Assessment of Physiotherapy Practice (APP) a valid instrument for the assessment of entry-level competence in physiotherapy students? DESIGN: Cross-sectional study with Rasch analysis of initial (n=326) and validation samples (n=318). Students were assessed on completion of 4, 5, or 6-week clinical placements across one university semester. PARTICIPANTS: 298 clinical educators and 456 physiotherapy students at nine universities in Australia and New Zealand provided 644 completed APP instruments. RESULTS: APP data in both samples showed overall fit to a Rasch model of expected item functioning for interval scale measurement. Item 6 (Written communication) exhibited misfit in both samples, but was retained as an important element of competence. The hierarchy of item difficulty was the same in both samples with items related to professional behaviour and communication the easiest to achieve and items related to clinical reasoning the most difficult. Item difficulty was well targeted to person ability. No Differential Item Functioning was identified, indicating that the scale performed in a comparable way regardless of the student's age, gender or amount of prior clinical experience, and the educator's age, gender, or experience as an educator, or the type of facility, university, or clinical area. The instrument demonstrated unidimensionality confirming the appropriateness of summing the scale scores on each item to provide an overall score of clinical competence and was able to discriminate four levels of professional competence (Person Separation Index=0.96). Person ability and raw APP scores had a linear relationship (r(2)=0.99). CONCLUSION: Rasch analysis supports the interpretation that a student's APP score is an indication of their underlying level of professional competence in workplace practice.

The objective structured clinical examination (OSCE): optimising its value in the undergraduate nursing curriculum.

This article explores the use of the objective structured clinical examination (OSCE) in undergraduate nursing education. The advantages and limitations of this assessment approach are discussed and various applications of the OSCE are described. Attention is given to the complexities of evaluating some psychosocial competency components. The issues are considered in an endeavour to delineate the competency components, or skill sets, that best lend themselves to assessment by the OSCE. We conclude that OSCEs can be used most effectively in nurse undergraduate curricula to assess safe practice in terms of performance of psychomotor skills, as well as the declarative and schematic knowledge associated with their application. OSCEs should be integrated within a curriculum in conjunction with other relevant student evaluation methods.

Boosting morale and improving performance in the nursing setting.

AIM AND BACKGROUND: While the importance of morale is well researched in the nursing literature, strategies and interventions are not so prolific. The complexities of interpersonal relationships within the clinical domain, and the critical issues faced by nurses on a daily basis, indicate that morale, job satisfaction and motivation are essential components in improving workplace efficiency, output and communication amongst staff. Drawing on educational, organizational and psychological literature, this paper argues that the ability to inspire morale in staff is a fundamental indicator of sound leadership and managerial characteristics. EVALUATION AND KEY ISSUES: Four practical concepts that could be implemented in the clinical setting are proposed. These include: role preparation for managers, understanding internal and external motivation, fostering internal motivation in nursing staff, and the importance of attitude when investing in relationships.