Perspectives on the production, and use, of rapid evidence in decision making during the COVID-19 pandemic: a qualitative study.

OBJECTIVES: To describe perceptions of providing, and using rapid evidence, to support decision making by two national bodies (one public health policy and one front-line clinical practice) during the COVID-19 pandemic. DESIGN: Descriptive qualitative study (March-August 2020): 25 semistructured interviews were conducted, transcribed verbatim and thematically analysed. SETTING: Data were obtained as part of an evaluation of two Irish national projects; the Irish COVID-19 Evidence for General Practitioners project (General Practice (GP) project) which provided relevant evidence to address clinical questions posed by GPs; and the COVID-19 Evidence Synthesis Team (Health Policy project) which produced rapid evidence products at the request of the National Public Health Emergency Team. PARTICIPANTS: Purposive sample of 14 evidence providers (EPs: generated and disseminated rapid evidence) and 11 service ssers (SUs: GPs and policy-makers, who used the evidence). MAIN OUTCOME MEASURES: Participant perceptions. RESULTS: The Policy Project comprised 27 EPs, producing 30 reports across 1432 person-work-days. The GP project comprised 10 members from 3 organisations, meeting 49 times and posting evidence-based answers to 126 questions. Four unique themes were generated. 'The Work' highlighted that a structured but flexible organisational approach to producing evidence was essential. Ensuring quality of evidence products was challenging, particularly in the context of absent or poor-quality evidence. 'The Use' highlighted that rapid evidence products were considered invaluable to decision making. Trust and credibility of EPs were key, however, communication difficulties were highlighted by SUs (eg, website functionality). 'The Team' emphasised that a highly skilled team, working collaboratively, is essential to meeting the substantial workload demands and tight turnaround time. 'The Future' highlighted that investing in resources, planning and embedding evidence synthesis support, is crucial to national emergency preparedness. CONCLUSIONS: Rapid evidence products were considered invaluable to decision making. The credibility of EPs, a close relationship with SUs and having a highly skilled and adaptable team to meet the workload demands were identified as key strengths that optimised the utilisation of rapid evidence. ETHICS APPROVAL: Ethical approval was obtained from the National Research Ethics Committee for COVID-19-related Research, Ireland.

Rapid response and learning for later: establishing high quality information networks and evaluation frameworks for the National Ambulance Service response to COVID-19 - the ENCORE COVID Project Protocol.

Background: The National Ambulance Service (NAS) is at the forefront of Ireland's response to the COVID-19 pandemic. As directed in Ireland's National Action Plan, NAS significantly expanded prehospital services, including provision of a novel COVID-19 testing service. Additionally, other health services rely on NAS's capacity to assess, transport and/or treat COVID-19 patients. In a climate of innovation and adaptation, NAS needs to learn from international ambulance services and share experience. Evaluation of the NAS response to COVID-19 is required to facilitate evidence-based planning for subsequent waves or future pandemics, and to identify innovative practice for mainstreaming into routine service provision. Aims: This project aims to test the utility of novel information networks and develop a tool that is tailored to evaluating pandemic-imposed change in an emergency medical service. Methods: The first aim will be to introduce and measure the impact of ambulance-specific research and information updates for NAS. Secondly, the usefulness to members of an international network of senior ambulance and research personnel ('AMBULANCE+COVID19' network), and the clarity and feasibility of a short-survey instrument, the Emergency Medical Services Five Question Survey (EMS-5QS), will be assessed. Finally, an evaluation framework for assessing pandemic-imposed change will be developed to enable NAS determine innovations: (1) for reactivation in another wave or new pandemic; (2) to be sustained as part of routine service. The framework will be developed in collaboration with NAS and the National Quality Improvement Team. The Research Team includes expertise from academia, ambulance services and the National Public Health Emergency Team. Conclusions: This project will facilitate the prompt introduction of information sharing processes to an emergency medical service and assess the impact of those processes. By developing a process for evaluating pandemic-imposed change in NAS, this project will add to the toolbox for future pandemic planning in emergency medical services internationally.

An analysis of general practitioners' perspectives on patient safety incidents using critical incident technique interviews.

BACKGROUND: General practitioners report difficulty in knowing how to improve patient safety. OBJECTIVES: To analyse general practitioners' perspectives of contributing factors to patient safety incidents by collecting accounts of incidents, identifying the contributory factors to these incidents, assessing the impact and likelihood of occurrence of these incidents and examining whether certain categories of contributory factors were associated with the occurrence of high-risk incidents. METHODS: Critical incident technique interviews were carried out with 30 general practitioners in Ireland about a patient safety incident they had experienced. The Yorkshire Contributory Factors Framework was used to classify the contributory factors to incidents. Seven subject matter experts rated the impact and likelihood of occurrence of each incident. RESULTS: A total of 26 interviews were analysed. Almost two-thirds of the patient safety incidents were rated as having a major-to-extreme impact on the patient, and over a third were judged as having at least a bimonthly likelihood of occurrence. The most commonly described active failures were 'Medication Error' (34.6%) and 'Diagnostic Error' (30.8%). 'Situational Domain' was identified as a contributory domain in all patient safety incidents. 'Communication' breakdown at both practice and other healthcare-provider interfaces (69.2%) was also a commonly cited contributory factor. There were no significant differences in the levels of risk associated with the contributory factors. CONCLUSIONS: Critical incident technique interviews support the identification of contributory factors to patient safety incidents. There is a need to explore the use of the resulting data for quality and safety improvement in general practice.

Perceived safety climate in Irish primary care settings-a comparison with Scotland and England.

BACKGROUND: Safety climate (SC) measurement is a key component of quality and safety initiatives in primary healthcare. OBJECTIVES: To (1) measure perceived SC in Irish primary care; (2) examine whether perceptions of safety varied according to respondent characteristics; and (3) compare responses from our Irish sample to data from England and Scotland. METHODS: PC-SafeQuest Survey was administered to all practice staff in Irish general practices between March and May 2016. This survey consists of 30 items across five safety domains (workload, communication, leadership, teamwork, and safety systems). Multiple regression analysis was used to identify predictor variables of perceived safety. The effect size of the difference between the Irish sample's scores and published English and Scottish data were calculated. RESULTS: A total of 231 questionnaires (38.5%) were returned. Generally, positive perceptions of perceived safety were identified among Irish respondents, but workload had the lowest overall mean score (M = 4.3, SD = 1.2) of the five domains. Comparisons across the Irish, English and Scottish samples identified a medium size effect difference in workload; Scottish respondents perceived workload to have less of a negative impact on safety than Irish or English counterparts (Cohen's d = 0.602, 0.67 respectively). Analyses indicated that Irish GP principals perceived a more negative impact of workload on safety than administrative staff (ß = 0.28, P = 0.03). CONCLUSION: Irish SC data are largely similar to those of England and Scotland. The perceived potential for workload to negatively impact upon safety emerged within each country. In Ireland, GP principals perceive this as a greater threat than practice administrators do.

The construct validity of HPAT-Ireland for the selection of medical students: unresolved issues and future research implications.

Aptitude tests are widely used in selection. However, despite certain advantages their use remains controversial. This paper aims to critically appraise five sources of evidence for the construct validity of the Health Professions Admission Test (HPAT)-Ireland, an aptitude test used for selecting undergraduate medical students. The objectives are to identify gaps in the evidence, draw comparisons with other aptitude tests and outline future research directions. Our appraisal of the literature found that stakeholder feedback indicates that there is reasonable evidence for test content validity for two of the three sections of HPAT-Ireland. By contrast the Non-Verbal Reasoning section is widely criticised as having limited relevance to medical school performance and future clinical practice. In terms of concurrent validity there is a significant small to medium, negative correlation with school exit examinations, but not consistently so across all studies (r = -0.18, -0.28, 0.017). Likewise predictive validity studies vary, from negative to moderate strength correlations with examination performance during early years at medical school. Five studies indicate that HPAT-Ireland is supported in principle by the majority of stakeholders. While one consequence of its introduction is that successful applicants are now coming from more diverse academic backgrounds, there is no evidence that the socio-economic background of medical school entrants has been altered significantly. Negative perceptions of unfairness relating to gender, coaching and socio-economics remain. The evidence to date suggests that while there are slight gender differences, initially favouring males, these vary year on year. In conclusion, the attitudes towards, and performance of, HPAT-Ireland is not unlike that of other aptitude tests widely used internationally. The main justifications for its introduction have been achieved, in that Ireland no longer relies exclusively on a single measure of academic record for selection to medical school. However a number of areas require further research and exploration.

Practise what you preach: health behaviours and stress among non-consultant hospital doctors.

High rates of psychological distress, depression and suicide have been reported among doctors. Furthermore, many doctors do not access healthcare by conventional means. This study aimed to increase understanding regarding non-consultant hospital doctors' (NCHDs') response to stress and barriers to accessing supports, and identify possible solutions. Medical manpower departments in 58 hospitals distributed a 25-item questionnaire to 4,074 NCHDs; we received 707 responses (response rate, 17.4%). 60% of NCHDs were unable to take time off work when unwell; 'letting teammates down' (90.8%) and 'difficulty covering call' (85.9%) were the leading reasons. 'Being too busy' (85%), 'self-prescription' (66.6%) and 'self-management' (53.1%) were ranked highest in deterring NCHDs from visiting a general practitioner (GP). 22.9% of NCHDs would not attend a GP with anxiety or depression until they began to feel hopeless, helpless or suicidal. 12.2% would not seek help at all. 55% of respondents (n = 330) had to move away from partners or dependants due to work, negatively affecting the social supports of 82.9%. Possible practical solutions were explored. NCHDS are a vulnerable population and have a particularly challenging lifestyle. Key recommendations include improved GP and counselling access for NCHDs, and addressing the culture of self-treatment and poor health behaviours through undergraduate and postgraduate education.

The fairness, predictive validity and acceptability of multiple mini interview in an internationally diverse student population--a mixed methods study.

BACKGROUND: International medical students, those attending medical school outside of their country of citizenship, account for a growing proportion of medical undergraduates worldwide. This study aimed to establish the fairness, predictive validity and acceptability of Multiple Mini Interview (MMI) in an internationally diverse student population. METHODS: This was an explanatory sequential, mixed methods study. All students in First Year Medicine, National University of Ireland Galway 2012 were eligible to sit a previously validated 10 station MMI. Quantitative data comprised: demographics, selection tool scores and First Year Assessment scores. Qualitative data comprised separate focus groups with MMI Assessors, EU and Non-EU students. RESULTS: 109 students participated (45% of class). Of this 41.3% (n = 45) were Non-EU and 35.8% (n = 39) did not have English as first language. Age, gender and socioeconomic class did not impact on MMI scores. Non-EU students and those for whom English was not a first language achieved significantly lower scores on MMI than their EU and English speaking counterparts (difference in mean 11.9% and 12.2% respectively, P<0.001). MMI score was associated with English language proficiency (IELTS) (r = 0.5, P<0.01). Correlations emerged between First Year results and IELTS (r = 0.44; p = 0.006; n = 38) and EU school exit exam (r = 0.52; p<0.001; n = 56). MMI predicted EU student OSCE performance (r = 0.27; p = 0.03; n = 64). In the analysis of focus group data two overarching themes emerged: Authenticity and Cultural Awareness. MMI was considered a highly authentic assessment that offered a deeper understanding of the applicant than traditional tools, with an immediate relevance to clinical practice. Cultural specificity of some stations and English language proficiency were seen to disadvantage international students. Recommendations included cultural awareness training for MMI assessors, designing and piloting culturally neutral stations, lengthening station duration and providing high quality advance information to candidates. CONCLUSION: MMI is a welcome addition to assessment armamentarium for selection, particularly with regard to stakeholder acceptability. Understanding the mediating and moderating influences for differences in performance of international candidates is essential to ensure that MMI complies with the metrics of good assessment practice and principles of both distributive and procedural justice for all applicants, irrespective of nationality and cultural background.

Views of doctors of varying disciplines on HPAT-Ireland as a selection tool for medicine.

BACKGROUND: Selection tools for medicine must achieve political validity and enjoy stakeholder acceptability. This qualitative study aimed to establish the perspectives of doctors, from various clinical specialities, on HPAT-Ireland, a new selection tool for undergraduate medical students. METHODS: Fifteen doctors participated over three iterative cycles of recruitment, interviewing and analysis. Prior to interview, participants sat a practice HPAT-Ireland test. HPAT-Ireland has three sections: (1) Logical reasoning/problem solving; (2) Interpersonal understanding and (3): Non-verbal reasoning. SUMMARY OF RESULTS: Three themes emerged: job relatedness; utility of HPAT-Ireland and diversity. Sections 1 and 2 were considered very job related however Section 3 was widely criticised for lacking clinical relevance. Doctors did not think that the test would reliably predict future performance. However, one-third felt it was acceptable as a selection tool in conjunction with academic record. Those who found it unacceptable were influenced by its perceived narrow focus, limited job relatedness, potential for socioeconomic bias, impact on gender and potential for negative influence on student diversity. CONCLUSIONS: A selection tool that does not enjoy the confidence of the medical profession is unlikely to achieve political validity and may ultimately fail, regardless of other objective measures of its effectiveness such as predictive validity.

To what extent does the Health Professions Admission Test-Ireland predict performance in early undergraduate tests of communication and clinical skills? An observational cohort study.

BACKGROUND: Internationally, tests of general mental ability are used in the selection of medical students. Examples include the Medical College Admission Test, Undergraduate Medicine and Health Sciences Admission Test and the UK Clinical Aptitude Test. The most widely used measure of their efficacy is predictive validity.A new tool, the Health Professions Admission Test- Ireland (HPAT-Ireland), was introduced in 2009. Traditionally, selection to Irish undergraduate medical schools relied on academic achievement. Since 2009, Irish and EU applicants are selected on a combination of their secondary school academic record (measured predominately by the Leaving Certificate Examination) and HPAT-Ireland score. This is the first study to report on the predictive validity of the HPAT-Ireland for early undergraduate assessments of communication and clinical skills. METHOD: Students enrolled at two Irish medical schools in 2009 were followed up for two years. Data collected were gender, HPAT-Ireland total and subsection scores; Leaving Certificate Examination plus HPAT-Ireland combined score, Year 1 Objective Structured Clinical Examination (OSCE) scores (Total score, communication and clinical subtest scores), Year 1 Multiple Choice Questions and Year 2 OSCE and subset scores. We report descriptive statistics, Pearson correlation coefficients and Multiple linear regression models. RESULTS: Data were available for 312 students. In Year 1 none of the selection criteria were significantly related to student OSCE performance. The Leaving Certificate Examination and Leaving Certificate plus HPAT-Ireland combined scores correlated with MCQ marks.In Year 2 a series of significant correlations emerged between the HPAT-Ireland and subsections thereof with OSCE Communication Z-scores; OSCE Clinical Z-scores; and Total OSCE Z-scores. However on multiple regression only the relationship between Total OSCE Score and the Total HPAT-Ireland score remained significant; albeit the predictive power was modest. CONCLUSION: We found that none of our selection criteria strongly predict clinical and communication skills. The HPAT- Ireland appears to measures ability in domains different to those assessed by the Leaving Certificate Examination. While some significant associations did emerge in Year 2 between HPAT Ireland and total OSCE scores further evaluation is required to establish if this pattern continues during the senior years of the medical course.