Automated Writing Assessments Measure Undergraduate Learning after Completion of a Computer-Based Cellular Respiration Tutorial.

The focus of biology education has shifted from memorization to conceptual understanding of core biological concepts such as matter and energy relationships. To examine undergraduate learning about matter and energy, we incorporated constructed-response (CR) questions into an interactive computer-based tutorial. The objective of this tutorial is to teach students about matter and energy and help dispel common misconceptions through the context of cellular respiration. We used a constructed-response classifier (CRC) tool to categorize ideas in responses to three CR questions and measure changes in student thinking about cellular respiration. Our data set includes 841 undergraduates from 19 geographically diverse institutions including two-year colleges, primarily undergraduate institutions, and research-intensive colleges and universities. We found students from all institution types included more scientific ideas in CRs post-tutorial. Students used an average of 2.1 ideas in CRs and frequently used both scientific and developing ideas. We found this mixed thinking persisted after the tutorial regardless of institution type. Students' multiple-choice (MC) selections were correlated with their CRs, but CRs revealed more mixed thinking than would be inferred from MC responses. Our study shows a CRC tool can measure student learning after a computer-based tutorial and provides more complete information than MC responses.

Introductory biology undergraduate students' mixed ideas about genetic information flow.

The core concept of genetic information flow was identified in recent calls to improve undergraduate biology education. Previous work shows that students have difficulty differentiating between the three processes of the Central Dogma (CD; replication, transcription, and translation). We built upon this work by developing and applying an analytic coding rubric to 1050 student written responses to a three-question item about the CD. Each response was previously coded only for correctness using a holistic rubric. Our rubric captures subtleties of student conceptual understanding of each process that previous work has not yet captured at a large scale. Regardless of holistic correctness scores, student responses included five or six distinct ideas. By analyzing common co-occurring rubric categories in student responses, we found a common pair representing two normative ideas about the molecules produced by each CD process. By applying analytic coding to student responses preinstruction and postinstruction, we found student thinking about the processes involved was most prone to change. The combined strengths of analytic and holistic rubrics allow us to reveal mixed ideas about the CD processes and provide a detailed picture of which conceptual ideas students draw upon when explaining each CD process.

Characterizing college science instruction: The Three-Dimensional Learning Observation Protocol.

The importance of improving STEM education is of perennial interest, and to this end, the education community needs ways to characterize transformation efforts. Three-dimensional learning (3DL) is one such approach to transformation, in which core ideas of the discipline, scientific practices, and crosscutting concepts are combined to support student development of disciplinary expertise. We have previously reported on an approach to the characterization of assessments, the Three-Dimensional Learning Assessment Protocol (3D-LAP), that can be used to identify whether assessments have the potential to engage students in 3DL. Here we present the development of a companion, the Three-Dimensional Learning Observation Protocol (3D-LOP), an observation protocol that can reliably distinguish between instruction that has potential for engagement with 3DL and instruction that does not. The 3D-LOP goes beyond other observation protocols, because it is intended not only to characterize the pedagogical approaches being used in the instructional environment, but also to identify whether students are being asked to engage with scientific practices, core ideas, and crosscutting concepts. We demonstrate herein that the 3D-LOP can be used reliably to code for the presence of 3DL; further, we present data that show the utility of the 3D-LOP in differentiating between instruction that has the potential to promote 3DL from instruction that does not. Our team plans to continue using this protocol to evaluate outcomes of instructional transformation projects. We also propose that the 3D-LOP can be used to support practitioners in developing curricular materials and selecting instructional strategies to promote engagement in three-dimensional instruction.

Through the Eyes of Faculty: Using Personas as a Tool for Learner-Centered Professional Development.

College science instructors need continuous professional development (PD) to meet the call to evidence-based practice. New PD efforts need to focus on the nuanced blend of factors that influence instructors' teaching practices. We used persona methodology to describe the diversity among instructors who were participating in a long-term PD initiative. Persona methodology originates from ethnography. It takes data from product users and compiles those data in the form of fictional characters. Personas facilitate user-centered design. We identified four personas among our participants: Emma the Expert views herself as the subject-matter expert in the classroom and values her hard-earned excellence in lecturing. Ray the Relater relates to students and focuses on their points of view about innovative pedagogies. Carmen the Coach coaches her students by setting goals for them and helping them develop skill in scientific practices. Beth the Burdened owns the responsibility for her students' learning and feels overwhelmed that students still struggle despite her use of evidence-based practice. Each persona needs unique PD. We suggest ways that PD facilitators can use our personas as a reflection tool to determine how to approach the learners in their PD. We also suggest further avenues of research on learner-centered PD.

Mixed Student Ideas about Mechanisms of Human Weight Loss.

Recent calls for college biology education reform have identified "pathways and transformations of matter and energy" as a big idea in biology crucial for students to learn. Previous work has been conducted on how college students think about such matter-transforming processes; however, little research has investigated how students connect these ideas. Here, we probe student thinking about matter transformations in the familiar context of human weight loss. Our analysis of 1192 student constructed responses revealed three scientific (which we label "Normative") and five less scientific (which we label "Developing") ideas that students use to explain weight loss. Additionally, students combine these ideas in their responses, with an average number of 2.19 ± 1.07 ideas per response, and 74.4% of responses containing two or more ideas. These results highlight the extent to which students hold multiple (both correct and incorrect) ideas about complex biological processes. We described student responses as conforming to either Scientific, Mixed, or Developing descriptive models, which had an average of 1.9 ± 0.6, 3.1 ± 0.9, and 1.7 ± 0.8 ideas per response, respectively. Such heterogeneous student thinking is characteristic of difficulties in both conceptual change and early expertise development and will require careful instructional intervention for lasting learning gains.

A Faculty Professional Development Model That Improves Student Learning, Encourages Active-Learning Instructional Practices, and Works for Faculty at Multiple Institutions.

Helping faculty develop high-quality instruction that positively affects student learning can be complicated by time limitations, a lack of resources, and inexperience using student data to make iterative improvements. We describe a community of 16 faculty from five institutions who overcame these challenges and collaboratively designed, taught, iteratively revised, and published an instructional unit about the potential effect of mutations on DNA replication, transcription, and translation. The unit was taught to more than 2000 students in 18 courses, and student performance improved from preassessment to postassessment in every classroom. This increase occurred even though faculty varied in their instructional practices when they were teaching identical materials. We present information on how this faculty group was organized and facilitated, how members used student data to positively affect learning, and how they increased their use of active-learning instructional practices in the classroom as a result of participation. We also interviewed faculty to learn more about the most useful components of the process. We suggest that this professional development model can be used for geographically separated faculty who are interested in working together on a known conceptual difficulty to improve student learning and explore active-learning instructional practices.

What Motivates Biology Instructors to Engage and Persist in Teaching Professional Development?

We conducted a study of 19 biology instructors participating in small, local groups at six research-intensive universities connected to the Automated Analysis of Constructed Response (AACR) project (www.msu.edu/∼aacr). Our aim was to uncover participants' motivation to persist in a long-term teaching professional development effort, a topic that is understudied in discipline-based educational research. We interviewed each participant twice over a 2-year period and conducted qualitative analyses on the data, using expectancy-value theory as a framework for considering motivation. Our analyses revealed that motivation among instructors was high due to their enjoyment of the AACR groups. The high level of motivation is further explained by the fact that AACR groups facilitated instructor involvement with the larger AACR project. We also found that group dynamics encouraged persistence; instructors thought they might never talk with colleagues about teaching in the absence of AACR groups; and groups were perceived to have a low-enough time requirement to warrant sustained involvement. We conclude that instructors have persisted in AACR groups because the groups provided great value with limited cost. The characterization of instructor experiences described here can contribute to a better understanding of faculty needs in teaching professional development.

Examining the impact of question surface features on students' answers to constructed-response questions on photosynthesis.

One challenge in science education assessment is that students often focus on surface features of questions rather than the underlying scientific principles. We investigated how student written responses to constructed-response questions about photosynthesis vary based on two surface features of the question: the species of plant and the order of two question prompts. We asked four versions of the question with different combinations of the two plant species and order of prompts in an introductory cell biology course. We found that there was not a significant difference in the content of student responses to versions of the question stem with different species or order of prompts, using both computerized lexical analysis and expert scoring. We conducted 20 face-to-face interviews with students to further probe the effects of question wording on student responses. During the interviews, we found that students thought that the plant species was neither relevant nor confusing when answering the question. Students identified the prompts as both relevant and confusing. However, this confusion was not specific to a single version.

What are they thinking? Automated analysis of student writing about acid-base chemistry in introductory biology.

Students' writing can provide better insight into their thinking than can multiple-choice questions. However, resource constraints often prevent faculty from using writing assessments in large undergraduate science courses. We investigated the use of computer software to analyze student writing and to uncover student ideas about chemistry in an introductory biology course. Students were asked to predict acid-base behavior of biological functional groups and to explain their answers. Student explanations were rated by two independent raters. Responses were also analyzed using SPSS Text Analysis for Surveys and a custom library of science-related terms and lexical categories relevant to the assessment item. These analyses revealed conceptual connections made by students, student difficulties explaining these topics, and the heterogeneity of student ideas. We validated the lexical analysis by correlating student interviews with the lexical analysis. We used discriminant analysis to create classification functions that identified seven key lexical categories that predict expert scoring (interrater reliability with experts = 0.899). This study suggests that computerized lexical analysis may be useful for automatically categorizing large numbers of student open-ended responses. Lexical analysis provides instructors unique insights into student thinking and a whole-class perspective that are difficult to obtain from multiple-choice questions or reading individual responses.

Exploring undergraduates' understanding of photosynthesis using diagnostic question clusters.

We present a diagnostic question cluster (DQC) that assesses undergraduates' thinking about photosynthesis. This assessment tool is not designed to identify individual misconceptions. Rather, it is focused on students' abilities to apply basic concepts about photosynthesis by reasoning with a coordinated set of practices based on a few scientific principles: conservation of matter, conservation of energy, and the hierarchical nature of biological systems. Data on students' responses to the cluster items and uses of some of the questions in multiple-choice, multiple-true/false, and essay formats are compared. A cross-over study indicates that the multiple-true/false format shows promise as a machine-gradable format that identifies students who have a mixture of accurate and inaccurate ideas. In addition, interviews with students about their choices on three multiple-choice questions reveal the fragility of students' understanding. Collectively, the data show that many undergraduates lack both a basic understanding of the role of photosynthesis in plant metabolism and the ability to reason with scientific principles when learning new content. Implications for instruction are discussed.

Applying computerized-scoring models of written biological explanations across courses and colleges: prospects and limitations.

Our study explored the prospects and limitations of using machine-learning software to score introductory biology students' written explanations of evolutionary change. We investigated three research questions: 1) Do scoring models built using student responses at one university function effectively at another university? 2) How many human-scored student responses are needed to build scoring models suitable for cross-institutional application? 3) What factors limit computer-scoring efficacy, and how can these factors be mitigated? To answer these questions, two biology experts scored a corpus of 2556 short-answer explanations (from biology majors and nonmajors) at two universities for the presence or absence of five key concepts of evolution. Human- and computer-generated scores were compared using kappa agreement statistics. We found that machine-learning software was capable in most cases of accurately evaluating the degree of scientific sophistication in undergraduate majors' and nonmajors' written explanations of evolutionary change. In cases in which the software did not perform at the benchmark of "near-perfect" agreement (kappa > 0.80), we located the causes of poor performance and identified a series of strategies for their mitigation. Machine-learning software holds promise as an assessment tool for use in undergraduate biology education, but like most assessment tools, it is also characterized by limitations.

Harnessing technology to improve formative assessment of student conceptions in STEM: forging a national network.

Concept inventories, consisting of multiple-choice questions designed around common student misconceptions, are designed to reveal student thinking. However, students often have complex, heterogeneous ideas about scientific concepts. Constructed-response assessments, in which students must create their own answer, may better reveal students' thinking, but are time- and resource-intensive to evaluate. This report describes the initial meeting of a National Science Foundation-funded cross-institutional collaboration of interdisciplinary science, technology, engineering, and mathematics (STEM) education researchers interested in exploring the use of automated text analysis to evaluate constructed-response assessments. Participants at the meeting shared existing work on lexical analysis and concept inventories, participated in technology demonstrations and workshops, and discussed research goals. We are seeking interested collaborators to join our research community.

The development of a conceptual framework and tools to assess undergraduates' principled use of models in cellular biology.

Recent science education reform has been marked by a shift away from a focus on facts toward deep, rich, conceptual understanding. This requires assessment that also focuses on conceptual understanding rather than recall of facts. This study outlines our development of a new assessment framework and tool-a taxonomy- which, unlike existing frameworks and tools, is grounded firmly in a framework that considers the critical role that models play in science. It also provides instructors a resource for assessing students' ability to reason about models that are central to the organization of key scientific concepts. We describe preliminary data arising from the application of our tool to exam questions used by instructors of a large-enrollment cell and molecular biology course over a 5-yr period during which time our framework and the assessment tool were increasingly used. Students were increasingly able to describe and manipulate models of the processes and systems being studied in this course as measured by assessment items. However, their ability to apply these models in new contexts did not improve. Finally, we discuss the implications of our results and the future directions for our research.