The Mean Delta Method: Quantifying Assessor Stringency and Leniency and Identifying Outliers in Workplace-Based Assessments.

ABSTRACT: Assessor stringency and leniency (ASL)-an assessor's tendency to award low or high scores-has a significant effect on workplace-based assessments. Outliers on this spectrum have a disproportionate effect. However, no method has been published for quantifying ASL or identifying outlier stringent or lenient assessors using workplace-based assessment data. The authors propose the mean delta method, which compares the scores that an assessor awards to trainees with those trainees' mean scores. This novel, simple method can be used to quantify ASL and identify outlier assessors without requiring specialized statistical knowledge or software. As a worked example, the mean delta method was applied to a set of end-of-shift assessments completed in a large Canadian academic emergency department from July 1, 2017, to May 31, 2018, and used to examine the net effect of ASL on learners' assessment scores. A total of 3,908 assessments were completed by 99 assessors for 151 trainees, with a median (interquartile range) of 37 (12-39) completed assessments per trainee. Using cutoff values of 1.5 and 2 standard deviations, a total of 11 and 3 outlier assessors were identified, respectively. Moreover, ASL changed overall scores by more than the mean difference between years of training for nearly 1 in 4 learners. The mean delta method was able to quantify ASL and identify outlier lenient and stringent assessors. It was also used to quantify the net effect of ASL on individual trainees. This method could be used to further study outlier assessors, to identify assessors who may benefit most from targeted coaching and feedback, and to measure changes in assessors' tendencies over time or with specific intervention.

Pharmaceuticals and personal care products in the environment: what are the big questions?

BACKGROUND: Over the past 10-15 years, a substantial amount of work has been done by the scientific, regulatory, and business communities to elucidate the effects and risks of pharmaceuticals and personal care products (PPCPs) in the environment. OBJECTIVE: This review was undertaken to identify key outstanding issues regarding the effects of PPCPs on human and ecological health in order to ensure that future resources will be focused on the most important areas. DATA SOURCES: To better understand and manage the risks of PPCPs in the environment, we used the "key question" approach to identify the principle issues that need to be addressed. Initially, questions were solicited from academic, government, and business communities around the world. A list of 101 questions was then discussed at an international expert workshop, and a top-20 list was developed. Following the workshop, workshop attendees ranked the 20 questions by importance. DATA SYNTHESIS: The top 20 priority questions fell into seven categories: a) prioritization of substances for assessment, b) pathways of exposure, c) bioavailability and uptake, d) effects characterization, e) risk and relative risk, f ) antibiotic resistance, and g) risk management. CONCLUSIONS: A large body of information is now available on PPCPs in the environment. This exercise prioritized the most critical questions to aid in development of future research programs on the topic.

Generation of priority research questions to inform conservation policy and management at a national level.

Integrating knowledge from across the natural and social sciences is necessary to effectively address societal tradeoffs between human use of biological diversity and its preservation. Collaborative processes can change the ways decision makers think about scientific evidence, enhance levels of mutual trust and credibility, and advance the conservation policy discourse. Canada has responsibility for a large fraction of some major ecosystems, such as boreal forests, Arctic tundra, wetlands, and temperate and Arctic oceans. Stressors to biological diversity within these ecosystems arise from activities of the country's resource-based economy, as well as external drivers of environmental change. Effective management is complicated by incongruence between ecological and political boundaries and conflicting perspectives on social and economic goals. Many knowledge gaps about stressors and their management might be reduced through targeted, timely research. We identify 40 questions that, if addressed or answered, would advance research that has a high probability of supporting development of effective policies and management strategies for species, ecosystems, and ecological processes in Canada. A total of 396 candidate questions drawn from natural and social science disciplines were contributed by individuals with diverse organizational affiliations. These were collaboratively winnowed to 40 by our team of collaborators. The questions emphasize understanding ecosystems, the effects and mitigation of climate change, coordinating governance and management efforts across multiple jurisdictions, and examining relations between conservation policy and the social and economic well-being of Aboriginal peoples. The questions we identified provide potential links between evidence from the conservation sciences and formulation of policies for conservation and resource management. Our collaborative process of communication and engagement between scientists and decision makers for generating and prioritizing research questions at a national level could be a model for similar efforts beyond Canada.