Shaping the right conditions in programmatic assessment: how quality of narrative information affects the quality of high-stakes decision-making.

BACKGROUND: Programmatic assessment is increasingly being implemented within competency-based health professions education. In this approach a multitude of low-stakes assessment activities are aggregated into a holistic high-stakes decision on the student's performance. High-stakes decisions need to be of high quality. Part of this quality is whether an examiner perceives saturation of information when making a holistic decision. The purpose of this study was to explore the influence of narrative information in perceiving saturation of information during the interpretative process of high-stakes decision-making. METHODS: In this mixed-method intervention study the quality of the recorded narrative information was manipulated within multiple portfolios (i.e., feedback and reflection) to investigate its influence on 1) the perception of saturation of information and 2) the examiner's interpretative approach in making a high-stakes decision. Data were collected through surveys, screen recordings of the portfolio assessments, and semi-structured interviews. Descriptive statistics and template analysis were applied to analyze the data. RESULTS: The examiners perceived less frequently saturation of information in the portfolios with low quality of narrative feedback. Additionally, they mentioned consistency of information as a factor that influenced their perception of saturation of information. Even though in general they had their idiosyncratic approach to assessing a portfolio, variations were present caused by certain triggers, such as noticeable deviations in the student's performance and quality of narrative feedback. CONCLUSION: The perception of saturation of information seemed to be influenced by the quality of the narrative feedback and, to a lesser extent, by the quality of reflection. These results emphasize the importance of high-quality narrative feedback in making robust decisions within portfolios that are expected to be more difficult to assess. Furthermore, within these "difficult" portfolios, examiners adapted their interpretative process reacting on the intervention and other triggers by means of an iterative and responsive approach.

Electromyographic Analysis of Muscle Activation Patterns During Bovine Transrectal Palpation and the Development of the Bovine Pregnancy Diagnosis Improvement Exercise Program.

The unusual and tiring physical activity of bovine transrectal palpation (TRP) requires a novel approach to improve students' TRP and pregnancy diagnosis (PD) skills. It has been shown that students who participated in an exercise program and students who had a grip strength (GS) of more than 30 kilograms performed better in bovine PDs. Participation in the exercise program increased students' sensitivity (ability to identify pregnant cows) but did not increase total arm muscle strength. To identify which muscles are used during TRPs and to improve the exercise program, an electromyographic (EMG) analysis was used to identify muscle activation patterns and muscle activity levels during bovine TRPs. Eight subject matter experts (SMEs) each palpated two live cows and one Breed'n Betsy® rectal examination simulator while an EMG Triggered Stimulator recorded muscle activity. Muscle activation was higher for forearm muscles compared with all other examined muscle groups (p < .001); was higher during retraction of the uterus and palpation of left and right uterine horn, compared with palpation of cervix, uterine body, left ovary, and right ovary (p < .001); and showed an endurance pattern. Findings have been used to modify the previously developed exercise program in effort to improve students' TRP and PD skills. The Bovine PD Improvement Exercise Program is available to students through an online application (http://icarus.up.ac.za/vetmlp/) and aims to not only improve GS and TRP accuracy but also stamina and well-being while adding fun to busy study schedules.

Influence of an Exercise Program, Muscle Strength, Proprioception, and Arm Length on Veterinary Students' Bovine Pregnancy Diagnosis Accuracy.

Bovine pregnancy diagnosis (PD) by transrectal palpation (TRP) is an important skill for veterinary graduates. Factors influencing students' PD accuracy were investigated to optimize bovine PD by TRP training without increasing live animal exposure. The objective was to determine whether arm length and strength, proprioception, and exposure to a 6-week exercise training program were significantly associated with students' PD accuracy. Veterinary students (n = 128) who had previously received formal theoretical and practical training in bovine TRP and PD (live cows and TRP simulators) were assessed for PD accuracy on live cows. Prior to assessment, arm muscle strength measurement, an exercise program, and additional TRP sessions on Breed'n Betsy® simulators and live cows were offered to the students. Seventy-eight students volunteered to participate in the arm length measurement, muscle strength, and proprioception testing. Of these, 35 randomly allocated students completed a 6-week exercise program, after which muscle strength was reassessed. Each student performed PDs on six cows of which the pregnancy status, ranging from 6 weeks to 9 months pregnant or not pregnant, was predetermined by an experienced veterinarian. PD accuracy was measured as sensitivity and specificity, being defined as the proportion of pregnant or nonpregnant cows, respectively, correctly identified by the student. It is concluded that hand grip strength and participation in an exercise program are significant predictors of veterinary students' PD accuracy. Implementation of an exercise program aimed at improving grip strength in the veterinary curriculum is a novel approach to improve bovine TRP and PD training.

Effect of a high-intensity one-week training programme and student-level variables on the bovine transrectal palpation and pregnancy diagnosis skills of final-year veterinary students.

BACKGROUND: To improve bovine transrectal palpation (TRP) and pregnancy diagnosis (PD) training, the effect of a high-intensity one-week training programme for veterinary elective students (N=59) with an interest in production animal practice was evaluated. METHODS: Training consisted of exposure to rectal examination simulators, abattoir organs, theory materials and live cow PDs supervised by experienced large animal practitioners. Palpation skills were assessed before and after training using a validated TRP Objective Structured Clinical Examination (OSCE) in non-pregnant cows. Each student then performed PDs (n=12) on cows of known pregnancy status. Students' PD accuracy was measured as sensitivity and specificity, being respectively defined as the proportion of pregnant and non-pregnant cows correctly identified. RESULTS: Students' scores improved from the first to the second OSCE (P=0.03), mostly as a result of improved ability to identify uterine symmetry/asymmetry and the presence/absence of a corpus luteum on the right ovary (P<0.01 and P=0.03, respectively). Overall student sensitivity and specificity of PD were 89.1 per cent (95 per cent CI 78.1-92.2 per cent) and 67.7 per cent (95 per cent CI 60.1-74.5 per cent), respectively. CONCLUSION: This prospective cohort study describes a strategy to improve students' TRP skills with the potential to reduce training time and animal use at teaching institutions by outsourcing student training to private practitioners.

Qualified but not yet fully competent: perceptions of recent veterinary graduates on their day-one skills.

BACKGROUND: The goal of veterinary education is to prepare learners to successfully enter the profession. However, the transition from learner to professional can be an intense and stressful phase. In this study, recently graduated veterinarians' perceptions of readiness to work independently and to successfully cope with early career challenges are addressed. METHODS: A survey based on five commonly occurring entrustable professional activities (EPAs) in primary care was sent to newly qualified veterinarians (graduated between six months and three-and-a-half years ago and working in primary veterinary clinics). The survey was a combination of open and Likert scale-type questions and contained items on the self-reported need for supervision for these EPAs. One hundred and fifty-six participants (response rate 41.2 per cent) answered the survey. Descriptive statistics were used to analyse and present the quantitative data. RESULTS: The day-one after graduation levels varied per EPA between 'with direct, proactive supervision' and 'supervision at a distance'. On average after 6.8 months participants felt ready to execute all five tasks with distant supervision. After almost 10 months, participants had the feeling of being fully competent to execute the EPAs unsupervised. CONCLUSION: This study provides insight into early career challenges faced by recently graduated veterinarians. The results emphasise the importance of adequate preparation of veterinarians during education and the importance of guidance during early career to foster a successful transition from veterinary school to clinical practice.

Programmatic assessment: Can we provide evidence for saturation of information?

Purpose: According to the principles of programmatic assessment, a valid high-stakes assessment of the students' performance should amongst others, be based on a multiple data points, supposedly leading to saturation of information. Saturation of information is generated when a data point does not add important information to the assessor. In establishing saturation of information, institutions often set minimum requirements for the number of assessment data points to be included in the portfolio. Methods: In this study, we aimed to provide validity evidence for saturation of information by investigating the relationship between the number of data points exceeding the minimum requirements in a portfolio and the consensus between two independent assessors. Data were analyzed using a multiple logistic regression model. Results: The results showed no relation between the number of data points and the consensus. This suggests that either the consensus is predicted by other factors only, or, more likely, that assessors already reached saturation of information. This study took the first step in investigating saturation of information, further research is necessary to gain in-depth insights of this matter in relation to the complex process of decision-making.

The Development of Entrustable Professional Activities for Competency-Based Veterinary Education in Farm Animal Health.

Entrustable professional activities (EPAs) are professional tasks that can be entrusted to a student under a given level of supervision once he or she has demonstrated competence in these tasks. The EPA construct was conceived to increase transparency in objectives for clinical workplace learning and to help ensure patient safety and the quality of care. A first step in implementing EPAs in a veterinary curriculum is to identify the core EPAs of the profession. The aim of this study was to develop EPAs for farm animal health. An initial set of 36 EPAs for farm animal health was prepared by a team of six veterinarians and curriculum developers and used in a modified Delphi study. In this iterative process, the EPAs were evaluated until higher than 80% agreement was reached. Of 83 veterinarians who participated, 39 (47%) completed the Delphi procedure. After two rounds, the panel reached consensus. A small expert group further refined and reorganized the EPAs for educational purposes into seven core EPAs for farm animal health and 29 sub-EPAs. This study is an important step in optimizing competency-based training in veterinary medicine. Future steps are to implement EPAs in the curriculum and train supervisors to assess students' ability to perform EPAs with increasing levels of independence.

Predictive Value of Three Different Selection Methods for Admission of Motivated and Well-Performing Veterinary Medical Students.

In search for valid and reliable selection methods that predict applicants' study motivation and academic performance during the 3-year bachelor's program at the Dutch Faculty of Veterinary Medicine (FVM), this study aimed to (1) examine the predictive value of the three FVM selection methods for study motivation and academic performance (i.e., direct admission and weighted lottery based on secondary school grade point average [GPA], and selection based on non-cognitive criteria), and (2) examine whether type and level of study motivation could be of value regarding selection of well-performing students. Data from two cohorts at the FVM (n = 186) were obtained, including mean summed scores on study motivation (using the Academic Motivation Scale [AMS] and additional items) and several academic outcome measures; among others, analyses of covariance (ANCOVA) were performed to examine differences between the three admission groups. Spearman's correlations and linear regression were applied to examine the relationship between study motivation and academic performance. Lottery-admitted students demonstrated a stronger extrinsic motivation than selected students (p < .05). Directly admitted students outperformed students from the other two admission groups on several academic outcome measures (p < .05). Only the level of motivation was related to academic performance (p < .05). According to the results, direct admission based on a high secondary school GPA in particular has predictive value for good academic performance during the 3-year bachelor's program of the veterinary course. The type of motivation seems to be of no value regarding selection of well-performing students, whereas level of motivation might be a useful criterion for this purpose.

Am I ready for it? Students' perceptions of meaningful feedback on entrustable professional activities.

BACKGROUND: Receiving feedback while in the clinical workplace is probably the most frequently voiced desire of students. In clinical learning environments, providing and seeking performance-relevant information is often difficult for both supervisors and students. The use of entrustable professional activities (EPAs) can help to improve student assessment within competency-based education. This study aimed to illustrate what students' perceptions are of meaningful feedback viewed as conducive in preparing for performing EPA unsupervised. METHODS: In a qualitative multicentre study we explored students' perceptions on meaningful feedback related to EPAs in the clinical workplace. Focus groups were conducted in three different healthcare institutes. Based on concepts from the literature, the transcripts were coded, iteratively reduced and displayed. RESULTS: Participants' preferences regarding meaningful feedback on EPAs were quite similar, irrespective of their institution or type of clerkship. Participants explicitly mentioned that feedback on EPAs could come from a variety of sources. Feedback must come from a credible, trustworthy supervisor who knows the student well, be delivered in a safe environment and stress both strengths and points for improvement. The feedback should be provided immediately after the observed activity and include instructions for follow-up. Students would appreciate feedback that refers to their ability to act unsupervised. CONCLUSION: There is abundant literature on how feedback should be provided, and what factors influence how feedback is sought by students. This study showed that students who are training to perform an EPA unsupervised have clear ideas about how, when and from whom feedback should be delivered.

Veterinarian awareness of farmer goals and attitudes to herd health management in The Netherlands.

In providing advice on herd health, veterinarians need to be aware of farmers' goals and priorities. To determine the level of awareness, 29 veterinarians from 15 practices completed questionnaires during visits to dairy farms within the scope of veterinary herd health management (VHHM) programmes. The farmers (n=30) were asked to complete a questionnaire and their discussions with the veterinarian were recorded using a voice recorder. Herd performance goals were set by the farmer and veterinarian in 24% of cases. Veterinarians who did not set goals indicated that they and the farmer 'intuitively knew' what each wanted to achieve, and that the setting of performance goals was considered 'too formal'. Veterinarians often could not identify a farmer's main goal, and typically found milk production and nutrition significantly more important (P<0.01, and P<0.02, respectively), and fertility significantly less important (P<0.01) than the farmers. During on-farm conversations, veterinarians did not actively seek to identify farmers' goals or problems, suggest a co-operative strategy or summarise any advice given. The findings of this survey suggest that veterinarians need to focus more on goal setting, since awareness of goals and priorities is important for both communication and compliance with advice given. The needs of farmers with respect to herd health should also be more actively sought by veterinarians as the findings indicate that most farmers do not readily volunteer such information.

Effectiveness of a programme design for the development of competence in solving clinical problems.

BACKGROUND: To apply what has been learned theoretically in a clinical context is for many students a major challenge. In order to ease their transition into practice, a training programme was developed, focusing on learning to solve clinical problems. AIMS: The programme is designed for veterinary medicine students in the preclinical phase with already a sound theoretical base in biomedical and clinical sciences. The design is based on the engagement in learning and work processes derived from clinical practice and exposure to a large variety of real and paper-based cases. This article addresses the effectiveness of this programme design. METHOD: Programme effectiveness was defined in terms of the progress made by the students. This progress was established using methodological triangulation of the results from student questionnaires, performance observations and (pre and post) assessment. RESULTS: On all dimensions of effectiveness, the results consistently showed that the programme was perceived as effective and led to improved performances in solving clinical problems and better solutions in the assessment. The students' progress was substantial. CONCLUSIONS: At preclinical level, a course design based on the work processes in clinical practice and a mixture of real and paper-based patients is effective in enhancing problem-solving competence.

The perception of veterinary herd health management by Dutch dairy farmers and its current status in the Netherlands: a survey.

The importance of veterinary herd health management (VHHM) is increasing in both dairy farming and veterinary practice. Little is known, however, about how VHHM is perceived by farmers in terms of structure, content and satisfaction. In 2007 a questionnaire, containing questions about these three items was therefore sent to 800 Dutch dairy farmers. Farmers received two questionnaires, one for participants in VHHM and one for non-participants, allowing them to choose the appropriate one. Results were summarized and statistically analyzed. Farmers who were participating in VHHM had better farm performance. They were satisfied with the way VHHM was executed on their farm. However, there were some pressure points. Goal setting and evaluation was still not a regular part of VHHM, even though it is said to be effective in literature. Time spent on VHHM not visible to the farmer was often not charged or not clearly specified on the bill. The differences in opinions between participants and non-participants of VHHM indicated a lack of communication and/or product differentiation. Satisfaction with the way VHHM was executed on the farm had no significant influence on 305-day production. There was, however, some influence on calving interval and bulk milk somatic cell count (BMSCC).

An instructional model for training competence in solving clinical problems.

We examined the design of a course that aims to ease the transition from pre-clinical learning into clinical work. This course is based on the premise that many of the difficulties with which students are confronted in this transition result from a lack of experience in applying knowledge in real practice situations. It is focused on the development of competence in solving clinical problems; uses an instructional model with alternating clinical practicals, demonstrations, and tutorials; and extends throughout the last pre-clinical year. We used a "proof-of-concept" approach to establish whether the core principles of the course design are feasible with regard to achieving the intended results. With the learning functions and processes as a frame of reference, retrospective analysis of the course's design features shows that this design matches the conditions from theories of the development of competence in solving clinical problems and instructional design. Three areas of uncertainty in the design are identified: the quality of the cases (information, openness), effective teaching (student and teacher roles), and adjustment to the development of competence (progress, coherence).

Reliability of the bulk milk somatic cell count as an indication of average herd somatic cell count.

Bulk milk somatic cell count (BMSCC) is a frequently used parameter to estimate the subclinical mastitis prevalence in a dairy herd, but it often differs considerably from the average SCC of all individual cows in milk. In this study, first the sampling variation was determined on 53 dairy farms with a BMSCC ranging from 56 000 to 441 000 cells/ml by collecting five samples on each farm of the same bulk tank. The average absolute sampling variation ranged from 1800 to 19 800 cells/ml. To what extent BMSCC represents all lactating cows was evaluated in another 246 farms by comparing BMSCC to the average herd SCC corrected for milk yield (CHSCC), after the difference was corrected for the sampling variation of BMSCC. On average BMSCC was 49 000 cells/ml lower than CHSCC, ranging from -10 000 cells/ml to 182 000 cells/ml, while the difference increased with an increasing BMSCC. Subsequently, management practices associated with existing differences were identified. Farms with a small (<20%) difference between BMSCC and CHSCC administered intramuscular antibiotics for the treatment of clinical mastitis more often, used the high SCC history when cows were dried off more frequently and had a higher number of treatments per clinical mastitis case compared with farms with a large (20%) difference. Farms feeding high-SCC milk or milk with antibiotic residues to calves were 2.4-times more likely to have a large difference. Although sampling variation influences the differences between BMSCC and CHSCC, the remaining difference is still important and should be considered when BMSCC is used to review the average herd SCC and the subclinical mastitis prevalence.

Effect of penethamate hydriodide treatment on bacteriological cure, somatic cell count and milk production of cows and quarters with chronic subclinical Streptococcus uberis or Streptococcus dysgalactiae infection.

A randomized, controlled field trial was performed in The Netherlands to determine the therapeutic efficacy of parenteral penethamate hydriodide (Leocillin) against naturally occurring, chronic, streptococcal mastitis during lactation. Quarter milk samples were collected from subclinical cases of Streptococcus uberis or Streptococcus dysgalactiae mastitis to determine the effect of treatment on bacteriological cure and somatic cell count (SCC) at quarter level. A quarter was considered to be cured when the bacterial species, isolated prior to treatment, was not isolated from the quarter milk samples taken on days 10 and 20 post-treatment (bacteriological cure), or when a quarter milk SCC (QMSCC) was <250000 cells/ml on days 10 and 20 post-treatment (SCC cure). Longitudinal data analysis was performed to determine the effect of antibiotic therapy on SCC and milk yield at cow level. Bacteriological cure occurred in 59% of 29 treated quarters, while no cure was observed in any of the 21 untreated control quarters. Treatment resulted in a significant decrease in SCC at cow and quarter level in comparison with untreated controls. There was no significant effect of treatment on milk production. Antibacterial treatment of subclinical streptococcal infections during lactation also prevented clinical mastitis. Furthermore, the treatment may contribute to reduction of bulk milk SCC and to prevention of pathogen spread in dairy herds.