Designing and evaluating complex educational research to meet federal grant requirement: A demonstration of a funded large-scale protocol.

Federal funding agencies have invested significant resources supporting evidence-based, innovative approaches that address education problems and incorporate rigorous design and evaluation, particularly through randomized controlled trials (RCTs), the gold standard for yielding causal inference in scientific research. In this study, we introduced factors (i.e., evaluation design, attrition, outcome measures, analytic approach, and fidelity of implementation) that are often times required in the Federal Notice for application by the U.S. Department of Education, with an emphasis on What Works Clearinghouse (WWC) standards. We further presented a federally funded research protocol with a multi-year, clustered RCT design to determine the impact of an instructional intervention on students' academic performance in high-needs schools. In the protocol, we elaborated on how our research design, evaluation plan, power analysis, as well as confirmatory research questions and analytical approaches were aligned with the grant requirement and WWC standards. We intend to provide a road map to meeting WWC standards and to increase the likelihood of successful grant applications.

Inter-rater reliability data of classroom observation: Fidelity in large-scale randomized research in education.

This dataset belongs to a large-scale randomized controlled trial (RCT) in educational research targeting English learning students and their teachers' instructional capacity. The dataset includes ratings conducted through classroom observations of 45-minute English as a Second language (ESL) blocks. Each coder rated 60 recorded video segments collected from each teacher. During the 20-second segment, ratings of six domains of teachers' instruction (i.e., ESL Strategies, Group, Activity Structure, Mode, Language Content, Language of Teacher, Language of Student) were collected. The dataset is organized by teacher, by coder, and by domain, for researchers to analyze inter-rater reliability among coders by domain and/or cross-domain. This data article is related to the research article Tong et al. [3] on "The determination of appropriate coefficient indices for inter-rater reliability: using classroom observation instruments as fidelity measures in large-scale randomized research".

Analyzing Complex Longitudinal Data in Educational Research: A Demonstration With Project English Language and Literacy Acquisition (ELLA) Data Using xxM.

When analyzing complex longitudinal data, especially data from different educational settings, researchers generally focus only on the mean part (i.e., the regression coefficients), ignoring the equally important random part (i.e., the random effect variances) of the model. By using Project English Language and Literacy Acquisition (ELLA) data, we demonstrated the importance of taking the complex data structure into account by carefully specifying the random part of the model, showing that not only can it affect the variance estimates, the standard errors, and the tests of significance of the regression coefficients, it also can offer different perspectives of the data, such as information related to the developmental process. We used xxM (Mehta, 2013), which can flexibly estimate different grade-level variances separately and the potential carryover effect from each grade factor to the later time measures. Implications of the findings and limitations of the study are discussed.

A primer on quality indicators of distance education.

In the past decade, there has been an enormous growth of distance education courses and programs in higher education. The growth of distance education is particularly evident in the field of health education. However, the enormous potential of distance education is tempered by one overriding question: How does one ensure that distance education coursework and degrees are of high quality? To this end, the purpose of this study is twofold: to identify quality indicators of distance education and to provide implications of the identified quality indicators for health education researchers and practitioners. The results of the study reveal common benchmarks and quality indicators that all parties deem important in designing, implementing, and evaluating distance education courses and programs.