Stanford University, Stanford California 94305-3084

Henry M. Levin
David Jacks Professor of Higher Education
School of Education
(415) 723-0840
September 22, 1998

Becki Herman
Project Director
American Institutes for Research
1000 Thomas Jefferson Street, NW
Washington, DC 20007

Dear Ms. Herman:

This letter follows up my conservation with you of today. Overall I think that you have done a good job on this report. However, I think that you have under-rated the Accelerated Schools Project by your methodology. I acknowledge that I am a "developer", but I am also a highly recognized evaluator. I am also the former President of the Evaluation Research Society (which is now the American Evaluation Association), Past-Editor of the Review of Educational Research, and author of the most widely used book on cost-effectiveness analysis in education.
The under-rating of the Accelerated Schools results is due to:

1.  No consideration given to the large number of year-to-year gains from official records of schools and school districts cited in our Accomplishments of Accelerated Schools and documented in footnotes as to source. Although these are not experimental studies, they are documented results which should count for something. For example, if PS 108 in East Harlem, New Your has moved over three years from about 36 percent of students at or above grade level in math and reading to  61 percent at or above grade level in reading and 68 percent at or above grade level in math on CTBS (a school with a 93 percent poverty rate and a 100 percent minority), shouldn't this have some weight? What if we have evidence of dozens of schools with this kind of result? Should the lack of a formal study mean that this kind of evidence is worthless?

2.  Virtually all studies comparing "experimental" schools to control or comparison schools are not as rigorous as that sounds. The reason is that the two groups of schools were not chosen randomly;y, but the treatment school in almost all of the reforms must get an 80 percent buy-in while the comparison school does not have to show any ambition whatsoever. Simply measuring results at the comparison school does not mean that it is comparable in ambition, staffing, and commitment to student achievement. And remember that the developer chooses the comparison school, not an outsider.

3.  Use of one's own achievement measures that are aligned to one's reform will get results that are not found in more "neutral" testing. For example, see Ross and Smith (1994) Elementary School Journal, pp. 121-138 for results on the developers teat, but not on district tests. All Accelerated Schools results are on whatever tests districts require, not on the kinds of gifted and talented criteria which we are dedicated to.

4.  In an evaluation by Success for All evaluators in the State of Washington using SFA measures of outcomes and first year results which do not favor our project which is developmental and doesn't look for results until the third year, Accelerated Schools got slightly better results than Success for All. Using Peabody Picture Vocabulary Test as a control, the two projects showed no difference on oral reading, word identification, and passage comprehension. But, SFA had a .28 effect size advantage on work attack and ASP had a .41 advantage on writing. At the very least one could argue that ASP has at least comparable results with SFA in a direct comparative evaluation (even when the measures and timing would have favored SFA criteria rather than those of ASP). I have enclosed the study.

I would hope that based upon this information you can reconsider the "marginal" effectiveness that you gave to Accelerated Schools.

Sincerely,

Henry M. Levin
David Jacks Professor of Higher Education and Economics