Designing a questionnaire with retrospective pre-post items: Format matters.

Evaluators are frequently asked to evaluate educational interventions that are one-time events. The Retrospective Pretest (RPT) methodology is well suited for these circumstances and the evaluation literature is replete with discussions about the pros and cons of this approach. In RPT, program participants rate their attitudes or knowledge now, and also rate how they were before participation in the intervention. The difference between now and before ratings constitutes the measure of change. Little published literature exists about whether the layout of RPT items within the evaluation questionnaire yields different results. This study compared six different layouts using a sample of 1941 caregivers who participated in one of 96 training workshops. The layouts were compared on inattentiveness, unexpected decline in perceived knowledge, and the degree of before-now change. Findings show that design matters. The best performing layout is where items are placed in the center with before response options on the left of the page and the now response options on the right. Results point to the need for evaluators to pay attention to RPT layout, and for the field to establish criteria for assessing survey layout quality. The unexpectedly high rate of inattentiveness calls for evaluators to pay more attention identifying and addressing this in their survey data.

Fly-CURE, a multi-institutional CURE using Drosophila, increases students' confidence, sense of belonging, and persistence in research.

The Fly-CURE is a genetics-focused multi-institutional Course-Based Undergraduate Research Experience (CURE) that provides undergraduate students with hands-on research experiences within a course. Through the Fly-CURE, undergraduate students at diverse types of higher education institutions across the United States map and characterize novel mutants isolated from a genetic screen in Drosophila melanogaster. To date, more than 20 mutants have been studied across 20 institutions, and our scientific data have led to eleven publications with more than 500 students as authors. To evaluate the impact of the Fly-CURE experience on students, we developed and validated assessment tools to identify students' perceived research self-efficacy, sense of belonging in science, and intent to pursue additional research opportunities. Our data, collected over three academic years and involving 14 institutions and 480 students, show gains in these metrics after completion of the Fly-CURE across all student subgroups analyzed, including comparisons of gender, academic status, racial and ethnic groups, and parents' educational background. Importantly, our data also show differential gains in the areas of self-efficacy and interest in seeking additional research opportunities between Fly-CURE students with and without prior research experience, illustrating the positive impact of research exposure (dosage) on student outcomes. Altogether, our data indicate that the Fly-CURE experience has a significant impact on students' efficacy with research methods, sense of belonging to the scientific research community, and interest in pursuing additional research experiences.

Increasing faculty support, respect, and ability to help doctoral students explore non-academic research career opportunities.

We examined the role of the NIH-funded Broadening Experiences in Scientific Training (BEST) program at Wayne State University in increasing faculty (1) support for doctoral students exploring non-academic research opportunities, (2) respect for non-academic research, and (3) ability to help students with non-academic research career exploration. Ninety-seven faculty participated in one or more BEST activities over a five-year period. Fifty-three of those faculty (55%) completed an online survey about their participation in the program and their support, respect, and ability to help students explore non-academic research careers. Sixteen of these faculty were also interviewed in depth about their perspectives on the role professional development can play in enhancing faculty perspectives about non-academic research career options for their students. The survey and interview data reveal some changing perceptions of BEST faculty participants in their attitudes toward and respect for non-academic research careers, as well as in their ability to help students in career exploration. These faculty perceptions correlated with their level of participation in BEST activities. Importantly, this study also showed that some faculty believe they lack the experience and connections outside of academia to adequately support doctoral students' career exploration. The results of this NIH-funded BEST program on faculty attitudes underscore the influence of federally funded programs in changing institutional attitudes towards supporting student career choices that have broad societal impact.

Fly-CURE, a Multi-institutional CURE using Drosophila, Increases Students' Confidence, Sense of Belonging, and Persistence in Research.

The Fly-CURE is a genetics-focused multi-institutional Course-Based Undergraduate Research Experience (CURE) that provides undergraduate students with hands-on research experiences within a course. Through the Fly-CURE, undergraduate students at diverse types of higher education institutions across the United States map and characterize novel mutants isolated from a genetic screen in Drosophila melanogaster. To evaluate the impact of the Fly-CURE experience on students, we developed and validated assessment tools to identify students' perceived research self-efficacy, sense of belonging in science, and intent to pursue additional research opportunities. Our data show gains in these metrics after completion of the Fly-CURE across all student subgroups analyzed, including comparisons of gender, academic status, racial and ethnic groups, and parents' educational background. Importantly, our data also show differential gains in the areas of self-efficacy and interest in seeking additional research opportunities between Fly-CURE students with and without prior research experience, illustrating the positive impact of research exposure (dosage) on student outcomes. Altogether, our data indicate that the Fly-CURE experience has a significant impact on students' efficacy with research methods, sense of belonging to the scientific community, and interest in pursuing additional research experiences.

Advice from local/site evaluators: How to manage "up" within a large-scale initiative.

BUilding Infrastructure Leading to Diversity (BUILD), an initiative of the National Institutes of Health (NIH), provides grants to undergraduate institutions to implement and study innovative approaches to engaging and retaining students from diverse backgrounds in biomedical research. The NIH awarded BUILD grants to 10 higher education institutions in multiple states, including funding for local evaluations. This chapter presents findings from an online survey and interviews with 15 local evaluators from nine of the 10 BUILD sites. Participants shared their perspectives on the role of professional local evaluators in national evaluations, ideal national-local multisite evaluation partnerships, and the ways that funders can support these partnerships to maximize impact. They argued for customized technical assistance and other support for local evaluations; the importance of including local results in national evaluation findings; the value of local evaluators' subject-matter expertise; and the potential for funders to act as central organizing entities in national-local evaluation partnerships.

Developing Undergraduate Scientists by Scaffolding the Entry into Mentored Research.

For many college students, joining a research group is a critical step toward developing strong mentor-mentee relationships that help shape their science identities and research self-efficacy. ReBUILDetroit, a program that seeks to diversify the biomedical research workforce, uses a scaffolded process to help its scholars transition into research. The first-year curriculum includes a research methods course and a course-based undergraduate research experience that prepare ReBUILDetroit Scholars for entering a research group. Curricular and cocurricular elements prepare scholars for faculty interactions and diminish barriers that might otherwise prevent diverse students from obtaining these research experiences. The program facilitates research placements through student coaching and speed-pairing events. Quantitative and qualitative data on the scholars show strong perceived gains in science identity, enhanced research self-efficacy, and greater research preparedness.

Older workers: an opportunity to expand the long-term care/direct care labor force.

PURPOSE: This study examined issues related to using older workers in frontline jobs in long-term care from employers' and prospective employees' perspectives. DESIGN AND METHODS: Telephone surveys were conducted with employers representing 615 nursing homes and 410 home health agencies, and 1,091 low-income participants aged 40+ in Operation ABLE employment and training organizations. A total of 696 of these participants were 55 years or older. RESULTS: Low-income older workers were interested in paraprofessional careers in long-term care. More were interested in home health care jobs than working in nursing homes. Job titles that most interested these workers were infrequent in nursing homes. Many workers perceived their health status as sufficient for frontline work. The majority was interested in career advancement opportunities and went to senior centers and places of worship to seek employment. Industry employers had many positive perceptions of older direct care workers, but there were real and perceived deterrents to hiring older workers. The most prominent deterrents were employers' perceptions that older workers have higher health care costs and are less willing to use technology. IMPLICATIONS: Policy makers should target Title V funds toward training low-income older workers for long-term care jobs, particularly in using mechanical devices and long-term care technologies. Employment and training organizations should add coursework in long-term care technologies, learn about legal issues in targeting advertising to low-income older workers, and educate employers about ways to reach these prospective employees. Older workers can use information about employers' perceptions when seeking employment.