Don Lindberg's Outreach Legacy at the National Library of Medicine.

Friends and colleagues of Donald A.B. Lindberg M.D. came together to give tribute to his extraordinary contributions during his tenure (1984-2015) as Director of the U.S. National Library of Medicine (NLM). Dr. Lindberg died in 2019. The book, Transforming biomedical informatics and health information access: Don Lindberg and the U.S. National Library of Medicine. includes four sections. The ten edited chapters in section three (the Outreach section) are briefly summarized in this overview. As Associate Director for Health Information Programs Development, Elliot R. Siegel Ph.D. coordinated NLM's outreach programming under Dr. Lindberg's leadership from its inception in 1989 to his own retirement in 2010. Dr. Lindberg's legacy at NLM is one of new possibilities imagined, significant changes made in the mission and ethos of a venerable institution, and numerous successes achieved in a variety of settings and contexts. Like so much else Dr. Lindberg accomplished, these Outreach programs that profoundly changed the character of NLM would likely not have occurred without him. He made a difference.

Are We Making a Difference? Outreach Evaluation in Practice.

Donald A.B. Lindberg, M.D., brought with him when he joined NLM an inquisitive mind, tech savvy, and new ideas. He was an early advocate of both outreach and evaluation innovation at NLM. Dr. Lindberg initiated and supported multiple pilot test and implementation projects to strengthen NLM's health information outreach to healthcare providers, research scientists, health science and hospital librarians, and the general public, including minority and underserved populations. He helped steer NLM's transition to the Internet, and NLM's development of a robust framework for evaluating Internet and Web-based health information dissemination and outreach to its many audiences. Dr. Lindberg's leadership led to numerous landmark accomplishments, including the capacity-building "Measuring the Difference" outreach evaluation Guide, and a multidimensional approach to Internet and website evaluation that placed NLM at the forefront of federal agencies using these new and emerging technologies to support their missions.

Native Voices Exhibition: Stories of Health, Wellness, and Illness from American Indians, Alaska Natives, and Native Hawaiians.

The U.S. National Library of Medicine (NLM) exhibition known as Native Voices reflected Donald A.B. Lindberg M.D.'s keen and long-held desire to help improve public understanding of Native American health challenges and honor the culture, tradition, and healing ways of Native Peoples. A centerpiece of the exhibition was a large set of video interviews that Dr. Lindberg conducted with Native health and community leaders. Dr. Lindberg and his team engaged Native advisors in the exhibition development; sought Native input through Listening Circles, Tribal Consultations; and site visits, and made the video interviews accessible via interactive kiosks and iPads. For its time, this was state-of-the-art exhibition technology. The exhibition also included Native artifacts and art works to complement the videos, including a scale model of the iconic Hokule'a Native Hawaiian voyaging canoe, and a full-size Lummi Indian Healing Totem Pole. The totem journeyed across the U.S. prior to its installation next to the NLM herbal garden in Bethesda, MD. A traveling version of the exhibition visited more than 130 venues in 40 States across the U.S. The interview clips and other content are accessible on the exhibition website, and the full-length interviews are retained in the NLM permanent video collection.

Don Lindberg's outreach legacy at the National Library of Medicine.

Friends and colleagues of Donald A.B. Lindberg M.D. came together to give tribute to his extraordinary contributions during his tenure (1984-2015) as Director of the U.S. National Library of Medicine (NLM). Dr. Lindberg died in 2019. The book, Transforming biomedical informatics and health information access: Don Lindberg and the U.S. National Library of Medicine. includes four sections. The ten edited chapters in section three (the Outreach section) are briefly summarized in this overview. As Associate Director for Health Information Programs Development, Elliot R. Siegel Ph.D. coordinated NLM's outreach programming under Dr. Lindberg's leadership from its inception in 1989 to his own retirement in 2010. Dr. Lindberg's legacy at NLM is one of new possibilities imagined, significant changes made in the mission and ethos of a venerable institution, and numerous successes achieved in a variety of settings and contexts. Like so much else Dr. Lindberg accomplished, these Outreach programs that profoundly changed the character of NLM would likely not have occurred without him. He made a difference.

Native Voices Exhibition: Stories of health, wellness, and illness from American Indians, Alaska Natives, and Native Hawaiians.

The U.S. National Library of Medicine (NLM) exhibition known as Native Voices reflected Donald A.B. Lindberg M.D.'s keen and long-held desire to help improve public understanding of Native American health challenges and honor the culture, tradition, and healing ways of Native Peoples. A centerpiece of the exhibition was a large set of video interviews that Dr. Lindberg conducted with Native health and community leaders. Dr. Lindberg and his team engaged Native advisors in the exhibition development; sought Native input through Listening Circles, Tribal Consultations; and site visits, and made the video interviews accessible via interactive kiosks and iPads. For its time, this was state-of-the-art exhibition technology. The exhibition also included Native artifacts and art works to complement the videos, including a scale model of the iconic Hokule'a Native Hawaiian voyaging canoe, and a full-size Lummi Indian healing totem pole. The totem journeyed across the U.S. prior to its installation next to the NLM herbal garden in Bethesda, MD. A traveling version of the exhibition visited more than 130 venues in 40 States across the U.S. The interview clips and other content are accessible on the exhibition website, and the full-length interviews are retained in the NLM permanent video collection.

Are we making a difference? Outreach evaluation in practice.

Donald A.B. Lindberg, M.D., brought with him when he joined NLM an inquisitive mind, tech savvy, and new ideas. He was an early advocate of both outreach and evaluation innovation at NLM. Dr. Lindberg initiated and supported multiple pilot test and implementation projects to strengthen NLM's health information outreach to healthcare providers, research scientists, health science and hospital librarians, and the general public, including minority and underserved populations. He helped steer NLM's transition to the Internet, and NLM's development of a robust framework for evaluating Internet and Web-based health information dissemination and outreach to its many audiences. Dr. Lindberg's leadership led to numerous landmark accomplishments, including the capacity-building "Measuring the Difference" outreach evaluation Guide, and a multi-dimensional approach to Internet and website evaluation that placed NLM at the forefront of federal agencies using these new and emerging technologies to support their missions.

Enhancing the Capacity of Community Organizations to Evaluate HIV/AIDS Information Outreach: A Pilot Experiment in Expert Consultation.

The National Library of Medicine's AIDS Community Information Outreach Program (ACIOP) supports and enables access to health information on the Internet by community-based organizations. A technical assistance (TA) model was developed to enhance the capacity of ACIOP awardees to plan, evaluate, and report the results of their funded projects. This consisted of individual Consultation offered by an experienced evaluator to advise on the suitability of proposed project plans and objectives, improve measurement analytics, assist in problem resolution and outcomes reporting, and identify other improvement possibilities. Group webinars and a moderated blog for the exchange of project-specific information were also offered. Structured data collections in the form of reports, online surveys, and key informant telephone interviews provided qualitative feedback on project progress, satisfaction with the TA, and the perceived impact of the interventions on evaluation capacity building. The Model was implemented in the 2013 funding cycle with seven organizations, and the level of reported satisfaction was uniformly high. One-on-one TA was requested by four awardee organizations, and was determined to have made a meaningful difference with three. Participation in the webinars was mandatory and high overall; and was deemed to be a useful means for delivering evaluation information. In subsequent funding cycles, submission of a Logic Model will be required of awardees as a new model intervention in the expectation that it will produce stronger proposals, and enable the evaluation consultant to identify earlier intervention opportunities leading to project improvements and evaluation capacity enhancements.

After-School Program for urban youth: Evaluation of a health careers course in New York City high schools.

Mentoring in Medicine (MIM) addresses an urgent national need for minority health professionals and promotes careers in health care for urban youth. The MIM After School Program (ASP or The Course) has as its primary objectives to provide academic enrichment in human biology and motivate disadvantaged youth to pursue a career in the health professions. Secondary objectives of The Course, although not evaluated here, are to improve students' health literacy and knowledge of healthy living behaviors. Since 2009, over 1500 middle and high school students have completed the New York City based Course, which is offered once a week over a 10 week semester in an out-of-school venue. This study assesses the success of The Course in achieving its primary objectives with 84 students at five New York City high schools during the fall 2014 semester. The Course curriculum was created especially for MIM, comprises the body's 11 organ systems, and is presented in discrete modules (one each semester), along with complementary educational activities, including field trips and class projects. This study reports on a formal evaluation using quantitative and qualitative methods. The quantitative evaluation found that the students significantly increased their knowledge of the Gastrointestinal System. Students across the academic spectrum appeared to have learned the MIM ASP Course content - high school GPA was not a predictor of knowledge acquisition. The students also reported that The Course significantly increased their self-confidence in their ability to succeed (self-efficacy). The students expressed a significant increase in five health care related attitudes and an additional increase in their ability to overcome personal issues to succeed in their career and significantly improving their feeling toward, and likely pursuit of, a health career. The students stated that The Course significantly increased their interest and intent to seek out more information about health care, participate in health care activities, and take more health care courses in high school. The qualitative evaluation found that the students and their parents were pleased with the MIM ASP Course's composition, presentation, and effectiveness. With a large majority of the parents stating that their child got out of The Course what they had hoped for and that The Course made it more likely that they would recommend a health career for their child. The students and instructional staff also identified The Course elements that they felt were most and least effective. Best practices that were used in designing and conducting The Course were identified. The MIM ASP Course appears to have achieved its principal educational objectives of providing academic enrichment in human biology and improving attitudes towards a health career for a self-selected population of disadvantaged, underrepresented minority high school students in an urban setting.

Promoting careers in health care for urban youth: What students, parents and educators can teach us.

There are many obstacles that urban youth experience in pursuing health careers, but the benefits of diversifying the classroom and workforce are clear. This is especially true today as educators and policymakers seek to enhance underrepresented minority students' access to health careers, and also achieve the health workforce needed to support the Affordable Care Act. The creation of student pipeline programs began more than 40 years ago, but success has been equivocal. In 2008, Mentoring in Medicine (MIM) conducted a research project to identify how students learn about health careers; develop strategies for an integrated, experiential learning program that encourages underrepresented minority students to pursue careers in health; and translate these into best practices for supporting students through their entire preparatory journey. Six focus groups were conducted with educators, students, and their parents. The inclusion of parents was unusual in studies of this kind. The outcome yielded important and surprising differences between student and parent knowledge, attitudes and beliefs. They informed our understanding of the factors that motivate and deter underrepresented minority students to pursue careers in health care. Specific programmatic strategies emerged that found their place in the subsequent development of new MIM programming that falls into the following three categories: community-based, school-based and Internet based. Best practices derived from these MIM programs are summarized and offered for consideration by other health career education program developers targeting underrepresented minority students, particularly those located in urban settings.

Empowering Patients and Community Online: Evaluation of the AIDS Community Information Outreach Program.

The AIDS Community Information Outreach Program (ACIOP) was created in 1994 to assist the affected community in utilizing electronic HIV/AIDS information resources. Nearly 300 competitive awards have been made to mostly community-based organizations. A formal evaluation was undertaken to determine the performance and impact of the ACIOP. A mixed methods design combined quantitative abstractions and summarization of 47 awardee final reports from 44 organizations, and qualitative telephone interviews with 17 individuals representing 20 projects. Findings revealed that project objectives were mostly met; high-risk populations were reached; low resource organizations were funded; community partnerships were significant; projects built on existing efforts; information resources and training were tailored to local needs; and most projects overcame barriers experienced. Needed modifications centered on: 1) enhancing evaluation capacity at the individual project level and 2) revising project reporting requirements to increase the amount of information available to assess the ACIOP; both have been implemented.

NLM Workshop Marks 20 Years of Community Outreach and Capacity Building in HIV/AIDS.

The National Library of Medicine's (NLM) AIDS Community Information Outreach Program (ACIOP) was launched in 1994 to provide the HIV/AIDS affected community with access to vital health information resources increasingly becoming available on the Internet. Three hundred awards have been made mostly to community-based organizations. An evaluation in 2012 found that most program objectives are being met; a principal recommendation going forward was that NLM seek to enhance the capacity of community-based awardees to conduct evaluations of their own projects. This article reports on a workshop whose invitees were drawn from AIDS serving organizations, along with scientists, clinicians, and information technologists, to review the evaluation findings and recommendations. They considered alternatives for improving awardees' evaluation capabilities with the help of expert consultation, identified additional steps that could be taken to make individual project results more transparent and sharable, and looked at external influences ranging from mobile health devices to the latest HIV/AIDS scientific research findings that could be used to align future awards with unmet needs in the community. The paper identifies efforts subsequently made by ACIOP managers to prioritize and operationalize guidance from the evaluation and the workshop, and discusses the benefits of community engagement.

E-learning in a virtual science camp for urban youth.

The Virtual Science Camp (VSC) is a unique demonstration of synchronous e-learning developed by Mentoring in Medicine (MIM). This paper reports on a pilot offering during the summer of 2012 that taught advanced biological concepts, healthy living and health care career opportunities to medically underserved urban youth. Livestream's interactive video technology was used to engage a diverse audience of mostly high school students at remote sites in a new two week instructional program that provided custom course content free of charge over the internet. We describe the technical and program preparations undertaken, their implementation, the IT environment, a multi-faceted evaluation plan, the results of the experiment, and lessons learned.

Defining the next generation journal: The NLM-Elsevier interactive publications experiment.

OBJECTIVE: A unique collaborative project to identify interactive enhancements to conventional-print journal articles, and to evaluate their contribution to readers' learning and satisfaction. HYPOTHESIS: It was hypothesized that (a) the enhanced article would yield more knowledge acquisition than the original article; (b) the interactivity aspects of the enhanced article would measurably contribute to the acquisition of knowledge; and (c) the enhancements to the original article would increase reader acceptance. METHODS: Fifteen SNMA medical students, assumed to have a greater generational familiarity and comfort level with interactive electronic media, reviewed 12 articles published in three Elsevier clinical and basic science journals. They used the Student National Medical Association's asynchronous online discussion forum over a four month period to suggest desired enhancements to improve learning. "Prognostic Factors in Stage T1 Bladder Cancer", published in the journal Urology was selected by the investigators as presenting the best opportunity to incorporate many of the students' suggested interactive and presentational enhancements in the limited timeframe available prior to the established test date. Educational, statistical, and medical consultants assisted in designing a test protocol in which 51 second to fourth year medical students were randomly assigned to experimental and control conditions, and were administered either the original or enhanced interactive version of the article on individual computer workstations. Test subjects consisted of 23 participants in the control group (8 males, 15 females) and 28 participants in the experimental group (9 males, 19 females). All subjects completed pre- and post-test instruments which measured their knowledge gain on 30 true-false and multiple-choice questions, along with 7 Likert-type questions measuring acceptance of the articles' format. Time to completion was recorded with the experimental group taking 22 min on average compared to 18 min for the controls; pre- and post-test times were 6 and 7 min, respectively. Statistical comparisons were based on change scores using either the Student t-test or the Two Way Analysis of Variance or Covariance. Significance was set at α = 0.05 or better. RESULTS: on the dependent measure of knowledge acquisition showed no difference overall on the 30 questions, but learning gain was statistically significant for the subset of 10 questions that measured gain on content that was accessible by the user-invoked interactive features of the enhanced article. Further analyses revealed significant interactions by student year and gender. Second year students (11 in the control group, 8 in the experimental group) were the best performers in terms of knowledge acquisition from both articles. The female medical students received a larger learning gain from journal enhancements and interactivity components than their male counterparts. Acceptance overall was greater for the experimental group who rated the experience more favorably than the controls. CONCLUSIONS: Failure to consider human factors such as gender and learning style may obscure underlying differences and their impact on the interactive aspects of scientific publications. Preliminary findings suggest the need for further study to include a heavier focus on interactivity apart from presentational enhancements; a more rigorous treatment of time as a specific variable; and an expanded experimental design that evaluates acquisition, understanding, integration and acceptance as dependent measures.

Web evaluation at the US National Institutes of Health: use of the American Customer Satisfaction Index online customer survey.

BACKGROUND: The National Institutes of Health (NIH), US Department of Health and Human Services (HHS), realized the need to better understand its Web users in order to help assure that websites are user friendly and well designed for effective information dissemination. A trans-NIH group proposed a trans-NIH project to implement an online customer survey, known as the American Customer Satisfaction Index (ACSI) survey, on a large number of NIH websites-the first "enterprise-wide" ACSI application, and probably the largest enterprise Web evaluation of any kind, in the US government. The proposal was funded by the NIH Evaluation Set-Aside Program for two years at a cost of US $1.5 million (US $1.275 million for survey licenses for 60 websites at US $18000 per website; US $225,000 for a project evaluation contractor). OBJECTIVE: The overall project objectives were to assess the value added to the participating NIH websites of using the ACSI online survey, identify any NIH-wide benefits (and limitations) of the ACSI, ascertain any new understanding about the NIH Web presence based on ACSI survey results, and evaluate the effectiveness of a trans-NIH approach to Web evaluation. This was not an experimental study and was not intended to evaluate the ACSI survey methodology, per se, or the impacts of its use on customer satisfaction with NIH websites. METHODS: The evaluation methodology included baseline pre-project websites profiles; before and after email surveys of participating website teams; interviews with a representative cross-section of website staff; observations of debriefing meetings with website teams; observations at quarterly trans-NIH Web staff meetings and biweekly trans-NIH leadership team meetings; and review and analysis of secondary data. RESULTS: Of the original 60 NIH websites signed up, 55 implemented the ACSI survey, 42 generated sufficient data for formal reporting of survey results for their sites, and 51 completed the final project survey. A broad cross-section of websites participated, and a majority reported significant benefits and new knowledge gained from the ACSI survey results. NIH websites as a group scored consistently higher on overall customer satisfaction relative to US government-wide and private sector benchmarks. CONCLUSIONS: Overall, the enterprise-wide experiment was successful. On the level of individual websites, the project confirmed the value of online customer surveys as a Web evaluation method. The evaluation results indicated that successful use of the ACSI, whether site-by-site or enterprise-wide, depends in large part on strong staff and management support and adequate funding and time for the use of such evaluative methods. In the age of Web-based e-government, a broad commitment to Web evaluation may well be needed. This commitment would help assure that the potential of the Web and other information technologies to improve customer and citizen satisfaction is fully realized.

MedlinePlus and the challenge of low health literacy: findings from the Colonias project.

OBJECTIVE: To explore the potential of a community-based health information outreach project to overcome problems associated with health literacy in low-income Hispanic communities along the Texas-Mexico border. METHODS: Using a train-the-trainer approach, community outreach workers known as promotoras were trained by a health information outreach team to search English and Spanish versions of MedlinePlus. These 15 promotoras submitted written examples on a weekly basis of the topics they helped residents explore on MedlinePlus and the ways in which the residents used the information. These weekly reports, along with verbal interviews with promotoras and others in the communities, allowed development of a database of 161 incidents ("stories") demonstrating how community residents used MedlinePlus. These stories were thematically analyzed to explore how the program benefited participants. RESULTS: The database of stories included examples of community residents becoming better informed about their illnesses, resolving to visit doctors, making decisions about recommended treatments, reducing their anxiety about health conditions, committing to healthy or preventive behavior, and assisting family members. CONCLUSION: With the help of paraprofessionals like promotoras, community-based health information outreach projects may improve the ability of community residents to understand their health conditions and to participate actively in their health care.

Information Rx: Evaluation of a new informatics tool for physicians, patients, and libraries.

This paper reports selected results from two comprehensive evaluation studies of the Information Prescription (or "Information Rx") Program implementation conducted from 2002-05 by the American College of Physicians Foundation (ACPF) and the U.S. National Library of Medicine (NLM). In this Program physicians are provided with Information Prescription pads, analogous to pads used to prescribe medications, that are used to direct patients to the MedlinePlus web site and its contents that are applicable to a patient's health condition. The results describe the Program's potential to enhance patient education and interpersonal communication from physician and patient perspectives. The findings suggest once physicians adopt the use of an information prescription, they perceive they are providing an additional clinical service that enhances patient education and interpersonal communication. For physicians, participation in information prescription may improve patient communication, encourage information seeking, and lessen the number of poor quality Internet searches that patients frequently self-perform and bring to a doctor's office. Similarly, once patients receive a recommendation from a physician to seek health information on the web, patients may be more comfortable with health seeking on the Internet and discussing their findings with their doctor. The conclusions of the two evaluation studies imply an Information Prescription fosters a dialogue between providers and patients, helps patients use the Internet more effectively and seems to favorably impact patient education. As the medical community and patient advocacy groups continue to emphasize the importance of evidence-based information as the gold standard for accepted care, it can be expected that informatics tools such as Information Rx will come to play an increasingly important role as a vehicle to help identify and access high quality health information on the Internet.

Health information Hispanic outreach in the Texas Lower Rio Grande Valley.

PURPOSE: This paper provides an overview of the two-year Texas Lower Rio Grande Valley Health Information Hispanic Outreach (HI HO) project. The project included a needs assessment, four pilot projects, and focus groups on the use of MedlinePlus and MedlinePlus en español. The needs assessment included a survey of physicians' information usage and a review of the circuit librarian program that had been established in 1989. The pilot projects were located at a high school, a rural health clinic, an urban health clinic, and a community center. Diffusion of innovation theory provided a framework for interpreting the results of the pilot projects. METHODS: The survey of physicians' information usage partially replicated a similar 1990 survey. The review of the circuit librarian program included usage statistics, interviews of administrators, and a survey of participants. Pilot project methodology varied by site. At the high school, four students were trained to instruct their peers in the use of MedlinePlus. At the two clinics, a computer workstation was installed for patients to access MedlinePlus. At the community center, staff were trained to use MedlinePlus en español to train community residents. Project evaluation included surveys, focus groups, and interviews. Indicators of success included increased level of consumer use of MedlinePlus, reports by key informants and consumers of how MedlinePlus was used, reports about training, and development of self-sustaining activity. RESULTS: The physician survey documented usage of health information resources in 2002 compared to 1990. The review of the circuit librarian program documented the change in program usage between 1989 and 2003. The pilot project at the high school was the most successful of the four pilot projects in introducing MedlinePlus to a large number of people, followed by the community center project. In the high school and community center projects, the participating institutions had reinforcing educational missions and paid staff who were highly motivated to achieve the project goals. The computer workstations projects at the two clinics were less successful, due in part to limited staff commitment and conflicting priorities. CONCLUSIONS: The HI HO project tested methods of reaching the Hispanic community in the Lower Rio Grande Valley region of Texas. The four HI HO pilot projects varied in achieving their stated objectives. But taken as a whole, the HI HO project significantly contributed to a better understanding of health information outreach to the Hispanic community, knowledge that should be useful to others with similar outreach activities.