Responsiveness of quality of life measures in children with peripheral vascular malformations: The OVAMA project.

BACKGROUND: The OVAMA (Outcome Measures for VAscular MAlformations) project determined quality of life (QoL) as a core outcome domain for evaluating treatment effect in vascular malformations. To correctly evaluate treatment effect on QoL, patient-reported outcome measures (PROMs) are needed that are responsive to changes. In children with vascular malformations, we explored if two widely used PROMs were responsive to changes: the Pediatric Quality of Life Inventory (PedsQL) and the Children's Dermatology Life Quality Index (CDLQI). METHODS: In an international multicenter prospective study, conservatively and invasively treated children completed the PedsQL and CDLQI at baseline and after follow-up of 6-8 weeks. At follow-up, change in health was assessed by a global rating of change (GRC) scale. Responsiveness was assessed by testing hypotheses on expected correlation strength between change scores of the PROMs and the GRC scale, and by calculating the area under the receiver operating characteristics curve (AUC). The PROMs were considered responsive if ≥75% of the hypotheses were confirmed or if the AUC was ≥0.7. RESULTS: Twenty-nine children were recruited in three centers in the Netherlands and United States, of which 25 completed all baseline and follow-up measurements. For both the PedsQL and CDLQI, less than 75% of the hypotheses were confirmed and the AUC was <0.7. DISCUSSION: The results suggest that these PROMs are not sufficiently responsive for evaluating treatment effect in peripheral vascular malformations. Our study emphasizes the need for assessing responsiveness before using a PROM in evaluating treatment effect.

COSMIN Risk of Bias tool to assess the quality of studies on reliability or measurement error of outcome measurement instruments: a Delphi study.

BACKGROUND: Scores on an outcome measurement instrument depend on the type and settings of the instrument used, how instructions are given to patients, how professionals administer and score the instrument, etc. The impact of all these sources of variation on scores can be assessed in studies on reliability and measurement error, if properly designed and analyzed. The aim of this study was to develop standards to assess the quality of studies on reliability and measurement error of clinician-reported outcome measurement instruments, performance-based outcome measurement instrument, and laboratory values. METHODS: We conducted a 3-round Delphi study involving 52 panelists. RESULTS: Consensus was reached on how a comprehensive research question can be deduced from the design of a reliability study to determine how the results of a study inform us about the quality of the outcome measurement instrument at issue. Consensus was reached on components of outcome measurement instruments, i.e. the potential sources of variation. Next, we reached consensus on standards on design requirements (n = 5), standards on preferred statistical methods for reliability (n = 3) and measurement error (n = 2), and their ratings on a four-point scale. There was one term for a component and one rating of one standard on which no consensus was reached, and therefore required a decision by the steering committee. CONCLUSION: We developed a tool that enables researchers with and without thorough knowledge on measurement properties to assess the quality of a study on reliability and measurement error of outcome measurement instruments.

Theoretical considerations on programmatic assessment.

Introduction: Programmatic assessment (PA) is an approach to assessment aimed at optimizing learning which continues to gain educational momentum. However, the theoretical underpinnings of PA have not been clearly described. An explanation of the theoretical underpinnings of PA will allow educators to gain a better understanding of this approach and, perhaps, facilitate its use and effective implementation. The purpose of this article is twofold: first, to describe salient theoretical perspectives on PA; second to examine how theory may help educators to develop effective PA programs, helping to overcome challenges around PA.Results: We outline a number of learning theories that underpin key educational principles of PA: constructivist and social constructivist theory supporting meaning making, and longitudinality; cognitivist and cognitive development orientation scaffolding the practice of a continuous feedback process; theory of instructional design underpinning assessment as learning; self-determination theory (SDT), self-regulation learning theory (SRL), and principles of deliberate practice providing theoretical tenets for student agency and accountability.Conclusion: The construction of a plausible and coherent link between key educational principles of PA and learning theories should enable educators to pose new and important inquiries, reflect on their assessment practices and help overcome future challenges in the development and implementation of PA in their programs.

Responsiveness of quality-of-life measures in patients with peripheral vascular malformations: the OVAMA project.

BACKGROUND: The OVAMA (Outcome Measures for Vascular Malformations) project determined quality of life (QoL) as a core outcome domain for patients with vascular malformations. In order to measure how current therapeutic strategies alter QoL in these patients, a patient-reported outcome measurement (PROM) responsive to changes in QoL is required. OBJECTIVES: To assess the responsiveness of two widely used generic QoL PROMs, the Medical Outcomes Study Short Form 36 (SF-36) and Skindex-29, in adult patients with vascular malformations. METHODS: In an international multicentre prospective study, treated and untreated patients completed the SF-36 and Skindex-29 at baseline and after a follow-up period of 6-8 weeks. Global rating of change (GRC) scales assessing various QoL-related outcome domains were additionally completed. Per subscale, responsiveness was assessed using two methods: by testing hypotheses on expected correlation strength between change scores of the questionnaires and the GRC scales, and by calculating the area under the receiver operating characteristics curve (AUC). The questionnaires were considered responsive if ≥ 75% of the hypotheses were confirmed or if the AUC was ≥ 0·7. RESULTS: Eighty-nine participants were recruited in three centres in the Netherlands and the U.S.A., of whom 67 completed all baseline and follow-up questionnaires. For all subscales of the SF-36 and Skindex-29, < 75% of the hypotheses were confirmed and the AUC was < 0·7. CONCLUSIONS: Our findings suggest that the SF-36 and Skindex-29 seemed unresponsive to change in QoL. This suggests that alternative PROMs are needed to measure - and ultimately improve - QoL in patients with vascular malformations. What's already known about this topic? Quality of life is often impaired in patients with vascular malformations. Quality of life is considered a core outcome domain for evaluating treatment of vascular malformations. To measure the effect of treatment on quality of life, a patient-reported outcome measure is required that is responsive to changes in quality of life. What does this study add? This is the first study assessing the responsiveness of quality-of-life measures in patients with vascular malformations. The results seem to indicate that the Medical Outcomes Study Short Form 36 (SF-36) and Skindex-29 are not responsive to changes in quality of life in patients with vascular malformations. What are the clinical implications of this work? Medical Outcomes Study Short Form 36 (SF-36) and Skindex-29 are not ideal to assess the effect on quality of life over time, of treatment strategies for peripheral vascular malformations.

The effect of adalimumab on the vasculature in psoriatic skin lesions.

BACKGROUND: Vascular proliferation is considered an important feature in psoriasis. Early psoriatic changes are characterized by vascular network expansion; healing of the lesions ultimately results in a decreasing size of the vascular network. Currently, the response of the vasculature in psoriatic lesions during treatment of adalimumab is still obscure. OBJECTIVE: To investigate the response of the vasculature in psoriatic lesions during adalimumab treatment, using parameters such as endothelial cell proliferation, vascular network size and alternations in vessel diameter. METHODS: The endothelial cell response (endothelial cell proliferation rate, vascular network size and vessel diameter) was studied, using an immunohistochemical double-staining of Ki67/CD31, in subsequent biopsies of psoriatic lesions from 10 patients treated with adalimumab at baseline, 10 days and 16 weeks after initiation of treatment. RESULTS: In active psoriatic skin the endothelial cell proliferation ratio is high and an increased vascular network size and vessel diameter is found. An evident decrease in all these parameters is found during 16 weeks treatment with adalimumab. Keratinocyte proliferation already decreased significantly and substantially after 10 days treatment. DISCUSSION: Adalimumab treatment of psoriasis causes an evident reduction in endothelial cell proliferation, vascular network size and vessel diameter, parallel to the clinical improvement. Vascular parameters do not capture early improvement to adalimumab treatment, however (non-)invasive measurement of vascular function parallels, the clinical improvement and may be a valuable marker for disease activity.

Assessment in a global context: An international perspective on dental education.

Assessments are widely used in dental education to record the academic progress of students and ultimately determine whether they are ready to begin independent dental practice. Whilst some would consider this a "rite-of-passage" of learning, the concept of assessments in education is being challenged to allow the evolution of "assessment for learning." This serves as an economical use of learning resources whilst allowing our learners to prove their knowledge and skills and demonstrating competence. The Association for Dental Education in Europe and the American Dental Education Association held a joint international meeting in London in May 2017 allowing experts in dental education to come together for the purposes of Shaping the Future of Dental Education. Assessment in a Global Context was one topic in which international leaders could discuss different methods of assessment, identifying the positives, the pitfalls and critiquing the method of implementation to determine the optimum assessment for a learner studying to be a healthcare professional. A post-workshop survey identified that educators were thinking differently about assessment, instead of working as individuals providing isolated assessments; the general consensus was that a longitudinally orientated systematic and programmatic approach to assessment provide greater reliability and improved the ability to demonstrate learning.

Development of an international core outcome set for peripheral vascular malformations: the OVAMA project.

BACKGROUND: An important limitation in vascular malformation research is the heterogeneity in outcome measures used for the evaluation of treatment outcome. OBJECTIVES: To reach international consensus on a core outcome set (COS) for clinical research on peripheral vascular malformations: lymphatic (LM), venous (VM) and arteriovenous malformations (AVM). In this consensus study, we determined what domains should constitute the COS. METHODS: Thirty-six possibly relevant outcome domains were proposed to an international group of physicians, patients and the parents of patients. In a three-round e-Delphi process using online surveys, participants repeatedly rated the importance of these domains on a five-point Likert scale. Participants could also propose other relevant domains. This process was performed for LM, VM and AVM separately. Consensus was predefined as 80% agreement on the importance of a domain among both the physician group and the patient/parent group. Outcomes were then re-evaluated in an online consensus meeting. RESULTS: 167 physicians and 134 patients and parents of patients with LM (n = 50), VM (n = 71) and AVM (n = 29) participated in the study. After three rounds and a consensus meeting, consensus was reached for all three types of vascular malformations on the core domains of radiological assessment, physician-reported location-specific signs, patient-reported severity of symptoms, pain, quality of life, satisfaction and adverse events. Vascular malformation type-specific signs and symptoms were included for LM, VM and AVM, separately. CONCLUSIONS: Our recommendation is that therapeutic-efficacy studies on peripheral vascular malformations should measure at least these core outcome domains.

How characteristic routines of clinical departments influence students' self-regulated learning: A grounded theory study.

BACKGROUND: In clerkships, students are expected to self-regulate their learning. How clinical departments and their routine approach on clerkships influences students' self-regulated learning (SRL) is unknown. AIM: This study explores how characteristic routines of clinical departments influence medical students' SRL. METHODS: Six focus groups including 39 purposively sampled participants from one Dutch university were organized to study how characteristic routines of clinical departments influenced medical students' SRL from a constructivist paradigm, using grounded theory methodology. The focus groups were audio recorded, transcribed verbatim and were analyzed iteratively using constant comparison and open, axial and interpretive coding. RESULTS: Students described that clinical departments influenced their SRL through routines which affected the professional relationships they could engage in and affected their perception of a department's invested effort in them. Students' SRL in a clerkship can be supported by enabling them to engage others in their SRL and by having them feel that effort is invested in their learning. CONCLUSIONS: Our study gives a practical insight in how clinical departments influenced students' SRL. Clinical departments can affect students' motivation to engage in SRL, influence the variety of SRL strategies that students can use and how meaningful students perceive their SRL experiences to be.

Construct validation of judgement-based assessments of medical trainees' competency in the workplace using a "Kanesian" approach to validation.

BACKGROUND: Evaluations of clinical assessments that use judgement-based methods have frequently shown them to have sub-optimal reliability and internal validity evidence for their interpretation and intended use. The aim of this study was to enhance that validity evidence by an evaluation of the internal validity and reliability of competency constructs from supervisors' end-of-term summative assessments for prevocational medical trainees. METHODS: The populations were medical trainees preparing for full registration as a medical practitioner (74) and supervisors who undertook ≥2 end-of-term summative assessments (n = 349) from a single institution. Confirmatory Factor Analysis was used to evaluate assessment internal construct validity. The hypothesised competency construct model to be tested, identified by exploratory factor analysis, had a theoretical basis established in workplace-psychology literature. Comparisons were made with competing models of potential competency constructs including the competency construct model of the original assessment. The optimal model for the competency constructs was identified using model fit and measurement invariance analysis. Construct homogeneity was assessed by Cronbach's α. Reliability measures were variance components of individual competency items and the identified competency constructs, and the number of assessments needed to achieve adequate reliability of R > 0.80. RESULTS: The hypothesised competency constructs of "general professional job performance", "clinical skills" and "professional abilities" provides a good model-fit to the data, and a better fit than all alternative models. Model fit indices were χ2/df = 2.8; RMSEA = 0.073 (CI 0.057-0.088); CFI = 0.93; TLI = 0.95; SRMR = 0.039; WRMR = 0.93; AIC = 3879; and BIC = 4018). The optimal model had adequate measurement invariance with nested analysis of important population subgroups supporting the presence of full metric invariance. Reliability estimates for the competency construct "general professional job performance" indicated a resource efficient and reliable assessment for such a construct (6 assessments for an R > 0.80). Item homogeneity was good (Cronbach's alpha = 0.899). Other competency constructs are resource intensive requiring ≥11 assessments for a reliable assessment score. CONCLUSION: Internal validity and reliability of clinical competence assessments using judgement-based methods are acceptable when actual competency constructs used by assessors are adequately identified. Validation for interpretation and use of supervisors' assessment in local training schemes is feasible using standard methods for gathering validity evidence.

Enhancing students' learning in problem based learning: validation of a self-assessment scale for active learning and critical thinking.

BACKGROUND: Problem based learning (PBL) is a powerful learning activity but fidelity to intended models may slip and student engagement wane, negatively impacting learning processes, and outcomes. One potential solution to solve this degradation is by encouraging self-assessment in the PBL tutorial. Self-assessment is a central component of the self-regulation of student learning behaviours. There are few measures to investigate self-assessment relevant to PBL processes. We developed a Self-assessment Scale on Active Learning and Critical Thinking (SSACT) to address this gap. We wished to demonstrated evidence of its validity in the context of PBL by exploring its internal structure. METHODS: We used a mixed methods approach to scale development. We developed scale items from a qualitative investigation, literature review, and consideration of previous existing tools used for study of the PBL process. Expert review panels evaluated its content; a process of validation subsequently reduced the pool of items. We used structural equation modelling to undertake a confirmatory factor analysis (CFA) of the SSACT and coefficient alpha. RESULTS: The 14 item SSACT consisted of two domains "active learning" and "critical thinking." The factorial validity of SSACT was evidenced by all items loading significantly on their expected factors, a good model fit for the data, and good stability across two independent samples. Each subscale had good internal reliability (>0.8) and strongly correlated with each other. CONCLUSIONS: The SSACT has sufficient evidence of its validity to support its use in the PBL process to encourage students to self-assess. The implementation of the SSACT may assist students to improve the quality of their learning in achieving PBL goals such as critical thinking and self-directed learning.

Twelve Tips for programmatic assessment.

Programmatic assessment is an integral approach to the design of an assessment program with the intent to optimise its learning function, its decision-making function and its curriculum quality-assurance function. Individual methods of assessment, purposefully chosen for their alignment with the curriculum outcomes and their information value for the learner, the teacher and the organisation, are seen as individual data points. The information value of these individual data points is maximised by giving feedback to the learner. There is a decoupling of assessment moment and decision moment. Intermediate and high-stakes decisions are based on multiple data points after a meaningful aggregation of information and supported by rigorous organisational procedures to ensure their dependability. Self-regulation of learning, through analysis of the assessment information and the attainment of the ensuing learning goals, is scaffolded by a mentoring system. Programmatic assessment-for-learning can be applied to any part of the training continuum, provided that the underlying learning conception is constructivist. This paper provides concrete recommendations for implementation of programmatic assessment.

Development of knowledge tests for multi-disciplinary emergency training: a review and an example.

BACKGROUND: The literature is sparse on written test development in a post-graduate multi-disciplinary setting. Developing and evaluating knowledge tests for use in multi-disciplinary post-graduate training is challenging. The objective of this study was to describe the process of developing and evaluating a multiple-choice question (MCQ) test for use in a multi-disciplinary training program in obstetric-anesthesia emergencies. METHODS: A multi-disciplinary working committee with 12 members representing six professional healthcare groups and another 28 participants were involved. Recurrent revisions of the MCQ items were undertaken followed by a statistical analysis. The MCQ items were developed stepwise, including decisions on aims and content, followed by testing for face and content validity, construct validity, item-total correlation, and reliability. RESULTS: To obtain acceptable content validity, 40 out of originally 50 items were included in the final MCQ test. The MCQ test was able to distinguish between levels of competence, and good construct validity was indicated by a significant difference in the mean score between consultants and first-year trainees, as well as between first-year trainees and medical and midwifery students. Evaluation of the item-total correlation analysis in the 40 items set revealed that 11 items needed re-evaluation, four of which addressed content issues in local clinical guidelines. A Cronbach's alpha of 0.83 for reliability was found, which is acceptable. CONCLUSION: Content and construct validity and reliability were acceptable. The presented template for the development of this MCQ test could be useful to others when developing knowledge tests and may enhance the overall quality of test development.

Integrating learning assessment and supervision in a competency framework for clinical workplace education.

Although competency-based education is well established in health care education, research shows that the competencies do not always match the reality of clinical workplaces. Therefore, there is a need to design feasible and evidence-based competency frameworks that fit the workplace reality. This theoretical paper outlines a competency-based framework, designed to facilitate learning, assessment and supervision in clinical workplace education. Integration is the cornerstone of this holistic competency framework.

Relationship between reflection ability and clinical performance: a cross-sectional and retrospective-longitudinal correlational cohort study in midwifery.

BACKGROUND: increasingly, reflection is highlighted as integral to core practice competencies but empirical research into the relationship between reflection and performance in the clinical workplace is scarce. AIM: this study investigated the relationship between reflection ability and clinical performance. METHODS: we designed a cross-sectional and a retrospective-longitudinal cohort study. Data from first, second and third year midwifery students were collected to study the variables 'clinical performance' and 'reflection ability'. Data were analysed with SPSS for Windows, Release 20.0. Descriptive statistics, Pearson׳s Product Moment Correlation Coefficients (r) and r² values were computed to investigate associations between the research variables. FINDINGS: the results showed a moderate observed correlation between reflection ability and clinical performance scores. When adopting a cross-sectional perspective, all correlation values were significant (p<0.01) and above 0.4, with the exception of the third year correlations. Assuming perfect reliability in the measurement, the adjusted correlations, for year 2 and year 3 indicated a high association between reflection ability and clinical performance (>0.6). The results based on the retrospective-longitudinal data set explained a moderate proportion of the variance after correction for attenuation. Finally, the results indicate that 'reflection ability' scores of earlier years are significant related with 'clinical performance' scores of subsequent years. These results suggest that (1) reflection ability is linked to clinical performance; (2) that written reflections are an important, but not the sole way to assess professional competence and that (3) reflection is a contributor to clinical performance improvement. CONCLUSIONS: the data showed a moderate but significant relationship between 'reflection ability' and 'clinical performance' scores in clinical practice of midwifery students. Reflection therefore seems an important component of professional competence.

What would happen to education if we take education evidence seriously?

Educational practice and educational research are not aligned with each other. Current educational practice heavily relies on information transmission or content delivery to learners. Yet evidence shows that delivery is only a minor part of learning. To illustrate the directions we might take to find better educational strategies, six areas of educational evidence are briefly reviewed. The flipped classroom idea is proposed to shift our expenditure and focus in education. All information delivery could be web distributed, thus creating more time for other more expensive educational strategies to support the learner. In research our focus should shift from comparing one curriculum to the other, to research that explains why things work in education and under which conditions. This may generate ideas for creative designers to develop new educational strategies. These best practices should be shared and further researched. At the same time attention should be paid to implementation and the realization that teachers learn in a way very similar to the people they teach. If we take the evidence seriously, our educational practice will look quite different to the way it does now.

A framework to facilitate self-directed learning, assessment and supervision in midwifery practice: a qualitative study of supervisors' perceptions.

BACKGROUND: Self-directed learning is an educational concept that has received increasing attention. The recent workplace literature, however, reports problems with the facilitation of self-directed learning in clinical practice. We developed the Midwifery Assessment and Feedback Instrument (MAFI) as a framework to facilitate self-directed learning. In the present study, we sought clinical supervisors' perceptions of the usefulness of MAFI. METHODS: Interviews with fifteen clinical supervisors were audio taped, transcribed verbatim and analysed thematically using Atlas-Ti software for qualitative data analysis. RESULTS: Four themes emerged from the analysis. (1) The competency-based educational structure promotes the setting of realistic learning outcomes and a focus on competency development, (2) instructing students to write reflections facilitates student-centred supervision, (3) creating a feedback culture is necessary to achieve continuity in supervision and (4) integrating feedback and assessment might facilitate competency development under the condition that evidence is discussed during assessment meetings. Supervisors stressed the need for direct observation, and instruction how to facilitate a self-directed learning process. CONCLUSION: The MAFI appears to be a useful framework to promote self-directed learning in clinical practice. The effect can be advanced by creating a feedback and assessment culture where learners and supervisors share the responsibility for developing self-directed learning.

Scaffolding reflective learning in clinical practice: a comparison of two types of reflective activities.

BACKGROUND: The development of reflective learning skills is a continuous process that needs scaffolding. It can be described as a continuum, with the focus of reflection differing in granularity from recent, concrete activities to global competency development. AIM: To explore learners' perceptions regarding the effects of two reflective writing activities designed to stimulate reflection at different degrees of granularity during clinical training. METHODS: Totally 142 respondents (students and recent graduates) completed a questionnaire. Quantitative and qualitative data were triangulated. RESULTS: Immediate reflection-on-action was perceived to be more valuable than delayed reflection-on-competency-development because it facilitated day-to-day improvement. Delayed reflection was perceived to facilitate overall self-assessment, self-confidence and continuous improvement, but this perception was mainly found among graduates. Detailed reflection immediately after a challenging learning experience and broad reflection on progress appeared to serve different learning goals and consequently require different arrangements regarding feedback and timing. CONCLUSIONS: Granularity of focus has consequences for scaffolding reflective learning, with immediate reflection on concrete events and reflection on long-term progress requiring different approaches. Learners appeared to prefer immediate reflection-on-action.