Educational Evaluation and Policy Analysis

Includes bibliographical references.

Effectiveness of Cognitive Tutor Algebra I at Scale - Reversing the Logic: An Outcomes-Based Student Typology for Determining "What Works" in Promoting an Array of Engineering-Related Student Learning Outcomes -- Evaluating the Effectiveness of Developmental Mathematics by Embedding a Randomized Experiment Within a Regression Discontinuity Design -- Contexts Matter: Selection in Means-Tested School Voucher Programs -- Staffing for Success: Linking Teacher Evaluation and School Personnel Management in Practice -- Changes in Levels of Affirmative Action in College Admissions in Response to Statewide Bans and Judicial Rulings -- Corrigendum.

[Article Title : Effectiveness of Cognitive Tutor Algebra I at Scale / John F. Pane, Beth Ann Griffin, Daniel F. McCaffrey, and Rita Karam, p. 127-144] Abstract : This article examines the effectiveness of a technology-based algebra curriculum in a wide variety of middle schools and high schools in seven states. Participating schools were matched into similar pairs and randomly assigned to either continue with the current algebra curriculum for 2 years or to adopt Cognitive Tutor Algebra I (CTAI), which uses a personalized, mastery-learning, blended-learning approach. Schools assigned to implement CTAI did so under conditions similar to schools that independently adopt it. Analysis of posttest outcomes on an algebra proficiency exam finds no effects in the first year of implementation, but finds evidence in support of positive effects in the second year. The estimated effect is statistically significant for high schools but not for middle schools; in both cases, the magnitude is sufficient to improve the median student's performance by approximately eight percentile points.;[Article Title : Reversing the Logic: An Outcomes-Based Student Typology for Determining "What Works" in Promoting an Array of Engineering-Related Student Learning Outcomes / David B. Knight, p 145-169] Abstract : Colleges and universities are being pressed to seek innovative ways to measure student learning outcomes and identify the conditions that lead to their development. Understanding how students group according to a multidimensional set of learning outcomes provides information on the extent to which institutions are meeting goals. This study develops a typology based on engineering undergraduates' array of outcomes. The study also demonstrates variation in personal and educational experiences across outcomes-based student groupings, thus providing insight into "what works" for programs who seek to graduate students who have developed an array of engineering-related outcomes. This outcomes-based approach is readily applicable to any set of student learning outcomes that programs or institutions seek to cultivate.;[Article Title : Evaluating the Effectiveness of Developmental Mathematics by Embedding a Randomized Experiment Within a Regression Discontinuity Design / Brian G. Moss, William H. Yeaton, and Jane E. LIoyd, p. 170-185] Abstract : Using a novel design approach, a randomized experiment (RE) was embedded within a regression discontinuity (RD) design (R-RE-D) to evaluate the impact of developmental mathematics at a large midwestern college (n = 2,122). Within a region of uncertainty near the cut-score, estimates of benefit from a prospective RE were closely comparable with those found using an RD design, based on retrospective data. Parametric and nonparametric analyses were conducted, yielding increases that generally ranged from one quarter to one third of a grade point. The within-study findings from the RE further established the credibility of RD to produce unbiased estimates. Qualifications of this embedded design strategy were discussed along with its numerous strengths, including elimination of additional validity threats.;[Article Title : Contexts Matter: Selection in Means-Tested School Voucher Programs / Cassandra M. D. Hart, p. 186-206] Abstract : This study examines public school characteristics, and public and private school market characteristics, associated with participation among elementary-aged students in a means-tested school voucher program in Florida. Participants are more likely than eligible nonparticipants to come from disadvantaged public schools on multiple dimensions. On average, participants' public schools have lower aggregate student achievement, and higher rates of disciplinary incidents. Participants' schools receive less positive ratings on various measures from principals and teachers. Participants face more competitive private school markets, and less competitive public school markets, than do nonparticipants. When these factors are considered together, the mean achievement and disciplinary incident rate of students' own public schools, as well as public and private school market variables, independently predict voucher use.;[Article Title : Staffing for Success: Linking Teacher Evaluation and School Personnel Management in Practice / Benjamin Master, p. 207-227] Abstract : Teacher evaluation is at the center of current education policy reform. Most evaluation systems rely at least in part on principals' assessments of teachers, and their discretionary judgments carry substantial weight. However, we know relatively little about what they value when determining evaluations and high stakes personnel decisions. Using unique data from an independently managed public charter school district, I explore the extent to which autonomous school administrators' formative evaluations of teachers predict a variety of future personnel decisions. I also assess the extent to which their evaluations predict alternative measures of teacher performance, including student and parent evaluations of individual teachers in the same and future school years. I find that formative midyear ratings-shared by administrators with teachers-clearly differentiate between teachers and are strongly associated with end-of-year dismissal and promotion decisions. I use an exploratory factor analysis to identify four distinct components of administrators' feedback to teachers and show that different components predict different types of personnel decisions in schools. In addition, different components predict different teacher performance measures. The results suggest the importance of accounting for multiple aspects of teachers' work in evaluation systems that are meant to inform multiple types of personnel decisions.;[Article Title : Changes in Levels of Affirmative Action in College Admissions in Response to Statewide Bans and Judicial Rulings / Grant H. Blume and Mark C. Long, p. 228-252] Abstract : Affirmative action in college admissions was effectively banned in Texas by the Hopwood ruling in 1997, by voter referenda in California and Washington in 1996 and 1998, and by administrative decisions in Florida in 1999. The Hopwood and Johnson rulings also had possible applicability to public colleges throughout Alabama, Georgia, Louisiana, and Mississippi. The Supreme Court's 2003 decision in the Grutter and Gratz cases reaffirmed but limited the legal basis for affirmative action in colleges. This article uses nationally representative data on the admissions decisions of high school students in 1992 and 2004 to estimate the magnitude of the change in affirmative action in college admission decisions (i.e., how these policy changes affected the relative likelihood of admission of minority and nonminority applicants). We find substantial declines in levels of affirmative action practiced by highly selective colleges in the states affected by bans and the Hopwood and Johnson rulings, and no evidence of declines outside these states (and thus modest and generally insignificant declines nationwide). We show how the decline in affirmative action in these particular states affects not only students in these states but also those students who live in adjacent states, particularly when the adjacent states lack highly selective colleges.;[Article Title : Corrigendum, p. 253] Abstract : Corrigendum to the article "Keeping Sight of the Forest Through the Trees: Response to 'Collective Bargaining, Transfer Rights, and Disadvantaged Schools,'" published in Volume 36, March 2014 of Educational Evaluation and Policy Analysis.