Evaluation of the Online Component of a Blended Learning Electroconvulsive Therapy Curriculum for Psychiatry Residents to Treat Depression in Older Adults.

OBJECTIVE: The objective of this study was to evaluate the online component of a blended curriculum for psychiatry residents on the use of electroconvulsive therapy (ECT) to treat depression in older adults. METHODS: Second- and third-year general psychiatry residents completed a blended learning curriculum during their core geriatric psychiatry rotation. The curriculum consisted of didactic seminars, hands-on clinical management, and two online clinical cases focused on the management of late-life depression with ECT. Knowledge acquisition following module completion was measured using a nine-question multiple-choice test. The authors adapted the Medical E-Learning Evaluation Survey (MEES) to measure resident satisfaction, clinical relevance, and instructional design. RESULTS: A total of 37 residents completed both online modules. Of these, 35 residents completed the knowledge test and 23 completed the adapted MEES. Almost all participants (96%) agreed or strongly agreed that the modules were relevant to their clinical work, evidence-based, able to be completed in a reasonable amount of time, and a valuable learning experience. The average score on the knowledge test, after removing one outlier, was 83%. CONCLUSION: Psychiatry residents are very satisfied with the content and delivery of the online component of a blended curriculum for understanding the use of ECT for late-life depression. Future work should examine satisfaction with the remainder of the curricula as well as the impact on longer-term knowledge acquisition and patient care.

Examining the Threat of ChatGPT to the Validity of Short Answer Assessments in an Undergraduate Medical Program.

OBJECTIVES: ChatGPT is an artificial intelligence model that can interpret free-text prompts and return detailed, human-like responses across a wide domain of subjects. This study evaluated the extent of the threat posed by ChatGPT to the validity of short-answer assessment problems used to examine pre-clerkship medical students in our undergraduate medical education program. METHODS: Forty problems used in prior student assessments were retrieved and stratified by levels of Bloom's Taxonomy. Thirty of these problems were submitted to ChatGPT-3.5. For the remaining 10 problems, we retrieved past minimally passing student responses. Six tutors graded each of the 40 responses. Comparison of performance between student-generated and ChatGPT-generated answers aggregated as a whole and grouped by Bloom's levels of cognitive reasoning, was done using t-tests, ANOVA, Cronbach's alpha, and Cohen's d. Scores for ChatGPT-generated responses were also compared to historical class average performance. RESULTS: ChatGPT-generated responses received a mean score of 3.29 out of 5 (n = 30, 95% CI 2.93-3.65) compared to 2.38 for a group of students meeting minimum passing marks (n = 10, 95% CI 1.94-2.82), representing higher performance (P = .008, η2 = 0.169), but was outperformed by historical class average scores on the same 30 problems (mean 3.67, P = .018) when including all past responses regardless of student performance level. There was no statistically significant trend in performance across domains of Bloom's Taxonomy. CONCLUSION: While ChatGPT was able to pass short answer assessment problems spanning the pre-clerkship curriculum, it outperformed only underperforming students. We remark that tutors in several cases were convinced that ChatGPT-produced responses were produced by students. Risks to assessment validity include uncertainty in identifying struggling students and inability to intervene in a timely manner. The performance of ChatGPT on problems requiring increasing demands of cognitive reasoning warrants further research.

Impact and Perceived Value of iGeriCare e-Learning Among Dementia Care Partners and Others: Pilot Evaluation Using the IAM4all Questionnaire.

BACKGROUND: Care partners of people living with dementia may benefit from web-based education. We developed iGeriCare, an award-winning internet-based platform with 12 multimedia e-learning lessons about dementia. OBJECTIVE: Our objective was to evaluate users' perceptions of impact. METHODS: From March 17, 2021 to May 16, 2022, data were collected upon lesson completion. We used the content-validated Information Assessment Method for all (IAM4all) for patients and the public adapted for dementia care partners. The IAM4all questionnaire assesses outcomes of web-based consumer health information. Responses were collected using SurveyMonkey, and data were analyzed using IBM SPSS Statistics (version 28). RESULTS: A total of 409 responses were collected, with 389 (95.1%) survey respondents completing the survey. Of 409 respondents, 179 (43.8%) identified as a family or friend care partner, 84 (20.5%) identified as an individual concerned they may have mild cognitive impairment or dementia, 380 (92.9%) identified the lesson as relevant or very relevant, and 403 (98.5%) understood the lesson well or very well. Over half of respondents felt they were motivated to learn more, they were taught something new, or they felt validated in what they do, while some felt reassured or felt that the lesson refreshed their memory. Of 409 respondents, 401 (98%) said they would use the information, in particular, to better understand something, discuss the information with someone else, do things differently, or do something. CONCLUSIONS: Users identified iGeriCare as relevant and beneficial and said that they would use the information. To our knowledge, this is the first time the IAM4all questionnaire has been used to assess patient and caregiver feedback on internet-based dementia education resources. A randomized controlled trial to study feasibility and impact on caregiver knowledge, self-efficacy, and burden is in progress.

Software Architecture for Automated Assessment of Prescription Writing.

Prescribing skills are a crucial competency in medical practice considering the increasing numbers of medications available and the increasingly complex patients with multiple diseases faced in clinical practice. Medical students need to become proficient in these skills during training, as required by medical licensing colleges. Not only is teaching the fundamentals of safe and cost-effective prescribing to medical students challenging but evaluating their prescribing skills by faculty members is difficult and time consuming. The COVID-19 pandemic has accelerated the interest in clinically relevant online exams, including automated assessment of short answer style questions. The goal of this project was to design a software to automate the assessment of learners' prescriptions written during low stakes formative assessments. After establishing the components of a legal prescription with multiple medications, and identifying the sources of errors in prescribing and prescribing assessment, we designed and validated an architecture and developed a prototype for automated parsing of learner prescriptions.

Online Educational Tools for Caregivers of People with Dementia: A Scoping Literature Review.

BACKGROUND: Informal caregivers of people with dementia provide the majority of health-based care to people with dementia. Providing this care requires knowledge and access to resources, which caregivers often do not receive. We set out to evaluate the effect of online educational tools on informal caregiver self-efficacy, quality of life, burden/stress, depression, and anxiety, and to identify effective processes for online educational tool development. METHODS: We conducted a scoping review of articles on online educational interventions for informal caregivers of people with dementia searching CINAHL, MEDLINE, EMBASE, and PubMed from 1990 to March 2018, with an updated search conducted in 2020. The identified articles were screened and the data were charted. RESULTS: 33 articles that reported on 24 interventions were included. There is some evidence that online interventions improve caregiver-related outcomes such as self-efficacy, depression, dementia knowledge, and quality of life; and decrease caregiver burden. Common findings across the studies included the need for tailored, stage-specific information applicable to the caregiver's situation and the use of psychosocial techniques to develop the knowledge components of the interventions. CONCLUSION: We demonstrate the importance of having caregivers and health-care professionals involved at all stages of tool conceptualization and development. Online tools should be evaluated with robust trials that focus on how increased knowledge and development approaches affect caregiver-related outcomes.

Spiralling Pre-clerkship Concepts into the Clinical Phase: Augmenting Knowledge Transfer Using Innovative Technology-Enhanced Curriculum Activities.

Background: Clerkship is a challenging transition for medical students where they learn to apply functional knowledge and diagnostic reasoning skills learned in the pre-clinical phase into the clinical environment. Rather than a smooth continuum to facilitate application of knowledge, clerkship blocks are discrete, fragmented structures with little integration. Developments in cognitive psychology and increasing attention to the student learning environment are driving more purposeful integration in medical education. We sought to enhance knowledge transfer in the Family Medicine clerkship by developing an e-learning pathway with both asynchronous and synchronous components to integrate pre-clerkship problem-based learning (PBL) cases into more complex clinical scenarios. Methods: A parallel-convergent mixed methods evaluation was conducted, which included comparison of learning outcomes (exit exams) relative to the prior class, knowledge check quiz performance, and qualitative analysis of student and faculty perceptions. Results: Analyses revealed no significant difference between exit exam scores of the intervention and pre-intervention clerks (p = 0.30). There were statistically significant differences in mean quiz scores over the rotation (p = 0.0001). Moreover, learners and faculty each perceived the integration components as facilitating the transfer of pre-clinical learning into clerkship activities. Conclusion: The novel e-learning pathway firmly anchored FM clerkship learning and will continue to ensure learners are ideally primed to optimize their direct clinical learning opportunities.

A Novel Educational Prescription Web-Based Application to Support Education for Caregivers of People Living With Dementia: Development and Usability Study With Clinicians.

BACKGROUND: It is estimated that 564,000 Canadians are currently living with dementia and there are approximately 486,000 to 1.1 million informal family/friend caregivers. Family/friend caregivers often receive little to no education or training about dementia but are expected to provide ongoing support for a complex condition. Web-based family/friend caregiver interventions may be helpful, but little is known about how best to implement them. OBJECTIVE: The objectives of this study were to 1) design and develop a novel education prescription application to help scale and spread web-based dementia education to family/friend caregivers, 2) conduct user testing, and 3) conduct a larger-scale field trial. METHODS: A novel education prescription web-based application was designed and developed. Initial user testing used task completion and the "think aloud" technique with a small sample of representative clinicians who work with people living with dementia and family/friend caregivers. Following iterative incorporation of feedback, a larger field trial was conducted with a convenience sample of clinicians. Account invitations were sent to 55 clinicians and, following a 2-month trial period, surveys were administered to participants including the System Usability Scale and the Net Promoter Score. RESULTS: During the initial user testing phase, participants (N=7) from representative disciplines easily completed associated tasks, and had very positive feedback with respect to the usability of the application. The System Usability Scale score during this phase was 91.4. Suggestions from feedback were incorporated into the application. During the larger field trial phase, participants (total N=55; activated account n=17; did not activate account n=38) were given access to the iGeriCare education prescription application. During this period, 2 participants created educational prescriptions; a total of 3 educational prescriptions were sent. Survey completers who did not activate their account (n=5) identified that their lack of use was due to time constraints, competing priorities, or forgetting to use the application. Survey completers who activated their account (n=5) identified their lower use was due to lack of time, lack of eligible family/friend caregivers during trial period, and competing priorities due to the COVID-19 pandemic. The System Usability Scale score during this phase was 78.75, and the Net Promoter Score was 50. CONCLUSIONS: Study findings indicate a generally positive response for the usability of a web-based application for clinicians to prescribe dementia education to family/friend caregivers. The dissonance between the promising data and widespread enthusiasm for the design and purpose of the education prescription application found in the initial user testing phase and subsequent lack of significant adoption in the field trial represents both an important lesson for other novel health technologies and a potential area for further investigation. Further research is required to better understand factors associated with implementation of this type of intervention and impact on dissemination of education to family/friend caregivers.

Barriers and Facilitators to Implementing Web-Based Dementia Caregiver Education From the Clinician's Perspective: Qualitative Study.

BACKGROUND: Internet-based dementia caregiver interventions have been shown to be effective for a range of caregiver outcomes; however, little is known about how to best implement them. We developed iGeriCare, an evidence-based, multimedia, web-based educational resource for family caregivers of people living with dementia. OBJECTIVE: This study aims to obtain feedback and opinions from experts and clinicians involved in dementia care and caregiver education about 1 iGeriCare and 2 barriers and facilitators to implementing a web-based caregiver program. METHODS: We carried out semistructured interviews with individuals who had a role in dementia care and/or caregiver education in several key stakeholder settings in Southern Ontario, Canada. We queried participants' perceptions of iGeriCare, caregiver education, the implementation process, and their experience with facilitators and barriers. Transcripts were coded and analyzed using a grounded theory approach. The themes that emerged were organized using the Consolidated Framework for Implementation Research. RESULTS: A total of 12 participants from a range of disciplines described their perceptions of iGeriCare and identified barriers and facilitators to the implementation of the intervention. The intervention was generally perceived as a high-quality resource for caregiver education and support, with many stakeholders highlighting the relative advantage of a web-based format. The intervention was seen to meet dementia caregiver needs, partially because of its flexibility, accessibility, and compatibility within existing clinical workflows. In addition, the intervention helps to overcome time constraints for both caregivers and clinicians. CONCLUSIONS: Study findings indicate a generally positive response to the use of internet-based interventions for dementia caregiver education. Results suggest that iGeriCare may be a useful clinical resource to complement traditional face-to-face and print material-based caregiver education. More comprehensive studies are required to identify the effectiveness and longevity of web-based caregiver education interventions and to better understand barriers and facilitators with regard to the implementation of technology-enhanced caregiver educational interventions in various health care settings.

Evaluation of an online knowledge translation intervention to promote cancer risk reduction behaviours: findings from a randomized controlled trial.

BACKGROUND: Many cancers are preventable through lifestyle modification; however, few adults engage in behaviors that are in line with cancer prevention guidelines. This may be partly due to the mixed messages on effective cancer prevention strategies in popular media. The goal of the McMaster Optimal Aging Portal (the Portal) is to increase access to trustworthy health information. The purpose of this study was to explore if and how knowledge translation strategies to disseminate cancer prevention evidence using the Portal influence participants' knowledge, intentions and health behaviors related to cancer risk. METHODS: Adults ≥40 years old, with no cancer history were randomized to a 12-week intervention (weekly emails and social media posts) or control group. Quantitative data on knowledge, intentions and behaviors (physical activity, diet, alcohol consumption and use of tobacco products) were collected at baseline, end of study and 3 months later. Participant engagement was assessed using Google Analytics, and participant satisfaction through open-ended survey questions and semi-structured interviews. RESULTS: Participants (n = 557, mean age 64.9) were predominantly retired (72%) females (81%). Knowledge of cancer prevention guidelines was higher in the intervention group at end of study only (+ 0.3, p = 0.01). Intentions to follow cancer prevention guidelines increased in both groups, with no between-group differences. Intervention participants reported greater light-intensity physical activity at end of study (+ 0.7 vs. 0.1, p = 0.03), and reduced alcohol intake at follow u (- 0.2 vs. + 0.3, p < 0.05), but no other between-group differences were found. Overall satisfaction with the Portal and intervention materials was high. CONCLUSIONS: Dissemination of evidence-based cancer prevention information through the Portal results in small increases in knowledge of risk-reduction strategies and with little to no impact on self-reported health behaviours, except in particular groups. Further tailoring of knowledge translation strategies may be needed to see more meaningful change in knowledge and health behaviours. TRIAL REGISTRATION: ClinicalTrials.gov NCT03186703, June 14, 2017.

A Tool That Assesses the Evidence, Transparency, and Usability of Online Health Information: Development and Reliability Assessment.

BACKGROUND: The internet is commonly used by older adults to obtain health information and this trend has markedly increased in the past decade. However, studies illustrate that much of the available online health information is not informed by good quality evidence, developed in a transparent way, or easy to use. Furthermore, studies highlight that the general public lacks the skills necessary to distinguish between online products that are credible and trustworthy and those that are not. A number of tools have been developed to assess the evidence, transparency, and usability of online health information; however, many have not been assessed for reliability or ease of use. OBJECTIVE: The first objective of this study was to determine if a tool assessing the evidence, transparency, and usability of online health information exists that is easy and quick to use and has good reliability. No such tool was identified, so the second objective was to develop such a tool and assess it for reliability when used to assess online health information on topics of relevant to optimal aging. METHODS: An electronic database search was conducted between 2002 and 2012 to identify published papers describing tools that assessed the evidence, transparency, and usability of online health information. Papers were retained if the tool described was assessed for reliability, assessed the quality of evidence used to create online health information, and was quick and easy to use. When no one tool met expectations, a new instrument was developed and tested for reliability. Reliability between two raters was assessed using the intraclass correlation coefficient (ICC) for each item at two time points. SPSS Statistics 22 software was used for statistical analyses and a one-way random effects model was used to report the results. The overall ICC was assessed for the instrument as a whole in July 2015. The threshold for retaining items was ICC>0.60 (ie, "good" reliability). RESULTS: All tools identified that evaluated online health information were either too complex, took a long time to complete, had poor reliability, or had not undergone reliability assessment. A new instrument was developed and assessed for reliability in April 2014. Three items had an ICC<0.60 (ie, "good" reliability). One of these items was removed ("minimal scrolling") and two were retained but reworded for clarity. Four new items were added that assessed the level of research evidence that informed the online health information and the tool was retested in July 2015. The total ICC score showed excellent agreement with both single measures (ICC=0.988; CI 0.982-0.992) and average measures (ICC=0.994; CI 0.991-0.996). CONCLUSIONS: The results of this study suggest that this new tool is reliable for assessing the evidence, transparency, and usability of online health information that is relevant to optimal aging.

If You Build It, Who Will Come? A Description of User Characteristics and Experiences With the McMaster Optimal Aging Portal.

Objectives: The McMaster Optimal Aging Portal (the Portal) aims to increase access to evidence-based health information. We would now like to understand who uses the Portal, why, and for what, and elicit feedback and suggestions for future initiatives. Methods: An online survey of users collected data on demographics, eHealth literacy, Internet use, information-seeking behavior, site acceptability and perceived impact on health behaviors, participant satisfaction, and suggestions for improvements using mixed methods. Results: Participants (n = 163, age 69.8 ± 8.6 years) were predominantly female (76%), married (67%), retired (80%), and well-educated with very good/excellent health (55%). The Portal was easy to use (83%) and relevant (80%), with 68% intending to, and 48% having changed behavior after using the Portal. A number of suggestions for improvement were obtained. Discussion: A better understanding of users' characteristics, needs, and preferences will allow us to improve content, target groups who are not engaging with the Portal, and plan future directions.

McMaster Optimal Aging Portal: an evidence-based database for geriatrics-focused health professionals.

OBJECTIVE: The objective of this work was to provide easy access to reliable health information based on good quality research that will help health care professionals to learn what works best for seniors to stay as healthy as possible, manage health conditions and build supportive health systems. This will help meet the demands of our aging population that clinicians provide high quality care for older adults, that public health professionals deliver disease prevention and health promotion strategies across the life span, and that policymakers address the economic and social need to create a robust health system and a healthy society for all ages. RESULTS: The McMaster Optimal Aging Portal's (Portal) professional bibliographic database contains high quality scientific evidence about optimal aging specifically targeted to clinicians, public health professionals and policymakers. The database content comes from three information services: McMaster Premium LiteratUre Service (MacPLUS™), Health Evidence™ and Health Systems Evidence. The Portal is continually updated, freely accessible online, easily searchable, and provides email-based alerts when new records are added. The database is being continually assessed for value, usability and use. A number of improvements are planned, including French language translation of content, increased linkages between related records within the Portal database, and inclusion of additional types of content. While this article focuses on the professional database, the Portal also houses resources for patients, caregivers and the general public, which may also be of interest to geriatric practitioners and researchers.

User Experiences of the McMaster Optimal Aging Portal's Evidence Summaries and Blog Posts: Usability Study.

BACKGROUND: Evidence summaries and blogs can support evidence-informed healthy aging, by presenting high-quality health research evidence in plain language for a nonprofessional (citizen) audience. OBJECTIVE: Our objective was to explore citizens' perceptions about the usability of evidence summaries and blog posts on the Web-based McMaster Optimal Aging Portal. METHODS: Twenty-two citizens (aged 50 years and older) and informal caregivers participated in a qualitative study using a think-aloud method and semistructured interviews. Eleven interviews were conducted in person, 7 over the telephone, and 4 by Skype. RESULTS: We identified themes that fell under 4 user-experience categories: (1) desirability: personal relevance, (2) understandability: language comprehension, grasping the message, dealing with uncertainty, (3) usability: volume of information, use of numbers, and (4) usefulness: intention to use, facility for sharing. CONCLUSIONS: Participants recognized that high-quality evidence on aging was valuable. Their intended use of the information was influenced by how much it applied to their own health circumstances or those of a loved one. Some specific formatting features that were preferred included consistent layout, content organized by subheadings, catchy titles, numerical information summarized in a table, and inclusion of a glossary.

The McMaster Optimal Aging Portal: Usability Evaluation of a Unique Evidence-Based Health Information Website.

BACKGROUND: Increasingly, older adults and their informal caregivers are using the Internet to search for health-related information. There is a proliferation of health information online, but the quality of this information varies, often based on exaggerated or dramatic findings, and not easily comprehended by consumers. The McMaster Optimal Aging Portal (Portal) was developed to provide Internet users with high-quality evidence about aging and address some of these current limitations of health information posted online. The Portal includes content for health professionals coming from three best-in-class resources (MacPLUS, Health Evidence, and Health Systems Evidence) and four types of content specifically prepared for the general public (Evidence Summaries, Web Resource Ratings, Blog Posts, and Twitter messages). OBJECTIVE: Our objectives were to share the findings of the usability evaluation of the Portal with particular focus on the content features for the general public and to inform designers of health information websites and online resources for older adults about key usability themes. METHODS: Data analysis included task performance during usability testing and qualitative content analyses of both the usability sessions and interviews to identify core themes. RESULTS: A total of 37 participants took part in 33 usability testing sessions and 21 focused interviews. Qualitative analysis revealed common themes regarding the Portal's strengths and challenges to usability. The strengths of the website were related to credibility, applicability, browsing function, design, and accessibility. The usability challenges included reluctance to register, process of registering, searching, terminology, and technical features. CONCLUSIONS: The study reinforced the importance of including end users during the development of this unique, dynamic, evidence-based health information website. The feedback was applied to iteratively improve website usability. Our findings can be applied by designers of health-related websites.

Development of a web-based, work-related asthma educational tool for patients with asthma.

BACKGROUND: Asthma is a common chronic condition. Work-related asthma (WRA) has a large socioeconomic impact and is increasing in prevalence but remains under-recognized. Although international guidelines recommend patient education, no widely available educational tool exists. OBJECTIVE: To develop a WRA educational website for adults with asthma. METHODS: An evidence-based database for website content was developed, which applied evidence-based website design principles to create a website prototype. This was subsequently tested and serially revised according to patient feedback in three moderated phases (one focus group and two interview phases), followed by face validation by asthma educators. RESULTS: Patients (n=10) were 20 to 28 years of age; seven (70%) were female, three (30%) were in university, two (20%) were in college and five (50%) were currently employed. Key format preferences included: well-spaced, bulleted text; movies (as opposed to animations); photos (as opposed to cartoons); an explicit listing of website aims on the home page; and an exploding tab structure. Participants disliked integrated games and knowledge quizzes. Desired informational content included a list of triggers, prevention/control methods, currently available tools and resources, a self-test for WRA, real-life scenario presentations, compensation information, information for colleagues on how to react during an asthma attack and a WRA discussion forum. CONCLUSIONS: The website met the perceived needs of young asthmatic patients. This resource could be disseminated widely and should be tested for its effects on patient behaviour, including job choice, workplace irritant/allergen avoidance and/or protective equipment, asthma medication use and physician prompting for management of WRA symptoms.

New directions in e-learning research in health professions education: Report of two symposia.

BACKGROUND: The use of Computer Assisted Instruction (CAI) is rising across health professions education. Research to date is of limited use in guiding the implementation and selection of CAI innovations. AIMS: In the context of two symposia, systemic reviews were discussed that evaluate literature in Internet-based learning, Virtual Patients, and animations. Each session included a debate with the goal of reaching consensus on best current practices and future research. METHODS: Thematic analysis of the discussions was performed to arrange the questions by theme, eliminate redundancy, and craft them into a cohesive narrative. RESULTS: The question analysis revealed that there are clear advantages to the use of CAI, and that established educational theories should certainly inform the future development and selection of CAI tools. Schools adopting CAI need to carefully consider the benefits, cost, available resources, and capacity for teachers and learners to accept change in their practice of education. Potential areas for future research should focus on the effectiveness of CAI instructional features, integration of e-learning into existing curricula and with other modalities like simulation, and the use of CAI in assessment of higher-level outcomes. CONCLUSIONS: There are numerous opportunities for future research and it will be important to achieve consensus on important themes.

E-learning interventions are comparable to user's manual in a randomized trial of training strategies for the AGREE II.

BACKGROUND: Practice guidelines (PGs) are systematically developed statements intended to assist in patient and practitioner decisions. The AGREE II is the revised tool for PG development, reporting, and evaluation, comprised of 23 items, two global rating scores, and a new User's Manual. In this study, we sought to develop, execute, and evaluate the impact of two internet interventions designed to accelerate the capacity of stakeholders to use the AGREE II. METHODS: Participants were randomized to one of three training conditions. 'Tutorial'--participants proceeded through the online tutorial with a virtual coach and reviewed a PDF copy of the AGREE II. 'Tutorial + Practice Exercise'--in addition to the Tutorial, participants also appraised a 'practice' PG. For the practice PG appraisal, participants received feedback on how their scores compared to expert norms and formative feedback if scores fell outside the predefined range. 'AGREE II User's Manual PDF (control condition)'--participants reviewed a PDF copy of the AGREE II only. All participants evaluated a test PG using the AGREE II. Outcomes of interest were learners' performance, satisfaction, self-efficacy, mental effort, time-on-task, and perceptions of AGREE II. RESULTS: No differences emerged between training conditions on any of the outcome measures. CONCLUSIONS: We believe these results can be explained by better than anticipated performance of the AGREE II PDF materials (control condition) or the participants' level of health methodology and PG experience rather than the failure of the online training interventions. Some data suggest the online tools may be useful for trainees new to this field; however, this requires further study.

Method and reporting quality in health professions education research: a systematic review.

CONTEXT: Studies evaluating reporting quality in health professions education (HPE) research have demonstrated deficiencies, but none have used comprehensive reporting standards. Additionally, the relationship between study methods and effect size (ES) in HPE research is unknown. OBJECTIVES: This review aimed to evaluate, in a sample of experimental studies of Internet-based instruction, the quality of reporting, the relationship between reporting and methodological quality, and associations between ES and study methods. METHODS: We conducted a systematic search of databases including MEDLINE, Scopus, CINAHL, EMBASE and ERIC, for articles published during 1990-2008. Studies (in any language) quantifying the effect of Internet-based instruction in HPE compared with no intervention or other instruction were included. Working independently and in duplicate, we coded reporting quality using the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement, and coded study methods using a modified Newcastle-Ottawa Scale (m-NOS), the Medical Education Research Study Quality Instrument (MERSQI), and the Best Evidence in Medical Education (BEME) global scale. RESULTS: For reporting quality, articles scored a mean±standard deviation (SD) of 51±25% of STROBE elements for the Introduction, 58±20% for the Methods, 50±18% for the Results and 41±26% for the Discussion sections. We found positive associations (all p<0.0001) between reporting quality and MERSQI (ρ=0.64), m-NOS (ρ=0.57) and BEME (ρ=0.58) scores. We explored associations between study methods and knowledge ES by subtracting each study's ES from the pooled ES for studies using that method and comparing these differences between subgroups. Effect sizes in single-group pretest/post-test studies differed from the pooled estimate more than ESs in two-group studies (p=0.013). No difference was found between other study methods (yes/no: representative sample, comparison group from same community, randomised, allocation concealed, participants blinded, assessor blinded, objective assessment, high follow-up). CONCLUSIONS: Information is missing from all sections of reports of HPE experiments. Single-group pre-/post-test studies may overestimate ES compared with two-group designs. Other methodological variations did not bias study results in this sample.

What do we mean by web-based learning? A systematic review of the variability of interventions.

OBJECTIVES: Educators often speak of web-based learning (WBL) as a single entity or a cluster of similar activities with homogeneous effects. Yet a recent systematic review demonstrated large heterogeneity among results from individual studies. Our purpose is to describe the variation in configurations, instructional methods and presentation formats in WBL. METHODS: We systematically searched MEDLINE, EMBASE, ERIC, CINAHL and other databases (last search November 2008) for studies comparing a WBL intervention with no intervention or another educational activity. From eligible studies we abstracted information on course participants, topic, configuration and instructional methods. We summarised this information and then purposively selected and described several WBL interventions that illustrate specific technologies and design features. RESULTS: We identified 266 eligible studies. Nearly all courses (89%) used written text and most (55%) used multimedia. A total of 32% used online communication via e-mail, threaded discussion, chat or videoconferencing, and 9% implemented synchronous components. Overall, 24% blended web-based and non-computer-based instruction. Most web-based courses (77%) employed specific instructional methods, other than text alone, to enhance the learning process. The most common instructional methods (each used in nearly 50% of courses) were patient cases, self-assessment questions and feedback. We describe several studies to illustrate the range of instructional designs. CONCLUSIONS: Educators and researchers cannot treat WBL as a single entity. Many different configurations and instructional methods are available for WBL instructors. Researchers should study when to use specific WBL designs and how to use them effectively.