Comments on the AIR Study of Comprehensive School Reform Approaches
from the
America's Choice Design Team

We welcome the AIR study and others on this topic as beginning contributions to what we hope will be a growing literature on comprehensive school reform. But this particular study is, we believe, deeply flawed. The study rated the programs on four dimensions. We will comment on the way each of these evaluations were applied to the America's Choice Design.

Strength of Research Base. Generally , when this term is used, it refers to the quality and extent of research findings on which the design is based. The question being answered is whether and to what degree the design is based on what is actually known about the factors that account for high student performance.

Our design is in fact based on extensive reviews of the research in the many fields that are covered by the design, including learning theory (one of the nation's leading cognitive scientists is associated with our team), standards and assessments (in which we are among the nation's leading authorities), curriculum (with particular attention to reading and writing) and modern management. Our own organization has done one of the most extensive qualitative international comparative studies of education ever undertaken. And we have had our design reviewed by several eminent researchers to make sure that it reflects the best research.

Ideally, of course, we would have undertaken a formal comprehensive meta-study of the many research fields that bear on our design. AIR appeared to have been looking for such a study or for original research by the design team. But neither New American Schools nor the foundations that have supported our work asked us to prepare such studies or do such research nor did they provide the funds that would have made it possible.

Effects on Students. The America's Choice Design incorporates a commitment to the New Standards Performance Standards and the New Standards Reference Examinations, which have been available for less than two years. Because, therefore, we did not have available comparative data based on our own standards and assessments, we asked jurisdictions with which we have been working to share the data they had on whatever tests and assessments they were using. There is space here for only three examples:

In Chicago, where we had been asked to take on 13 of the city's worst-performing elementary and middle schools, the percentage of third and eighth grade students scoring at or above the  national norms on the Iowa Test of Basic Skills rose from 14.6 percent on average when we started in 1996 to 19.9 percent in 1998 in reading, and from 16.4 percent to 23.6 percent on average in the same period in mathematics.

In Rochester, New York, where we were working with three elementary and middle schools, the average proportion of students meeting the standard on the statewide reading tests went form 69 percent to 90 percent from the 94-95 school year to the 96-97 school year. That proportion went from 92 percent to 98 percent in mathematics.

After our first year in Kentucky, State Commissioner of Education Tom Boysen announced that the schools associated with our design had made greater gains on the statewide assessment than the schools associated with any of the many other outside technical assistance networks operating in the state.

We supplied data of this kind to AIR. But AIR evidently did not consider it in arriving at their judgments because it did not come in the context of formal research studies using "carefully matched control groups" and so on.

In any case, since our standards and assessments are now available and our work has been fully codified into a formal design, we have asked CPRE, arguably the nation's leading educational program evaluators, to evaluate and report on the implementation and effects of the America's Choice Design. That work begins in the fall of 1998.

In the meantime, we would be happy to share the data we have from our school, district and state partners on the effects of our work thus far.

Ease of Implementation. Unfortunately, the AIR researchers apparently did not have an opportunity to ask our school, district and state partners what it has been like to work with us and how easy it has been to implement our design. Here is a typical comment from Dr. Judith Rizzo, Deputy Chancellor, New York City Public Schools:

"We like your standards. It begins there but doesn't end there. Most of what we've purchased is your expertise and thinking and talent to move, I think, in the direction we want to take the system. The quality of the standards was the most compelling feature, as well as the NCEE's willingness and understanding to customize them to make them feel and smell like New York City kids' work, which was an absolutely essential ingredient. The quality of the NCEE's professional development work that occurred during the process was the best I've ever seen., It embodied all of what I know good solid professional development to be. It modeled for our participants the kind of behaviors that we have to demonstrate in order to get kids to achieve the standards. Because it is experiential, it did this in ways that my own words could never have done. My testament is the fact that we continue with the work, in math and applied learning. Ann Borthwick is the most sought-after consultant I've ever seen in this system. Ann's practically part of the system. She couldn't be more invested in our system if her paycheck had the name of the Board of Education on it. Consultants come and go, but she's staying in town. The superintendents adore her."

The highest score one could get on the AIR scale requires successful implementation in only five schools. There are hundreds of schools in our network and we would be happy to point any interested party to more than five of them in which the faculty members would attest to the strong and effective implementation support they received from the America's Choice team.

Costs. The AIR report showed our design as being on the high end of the expense dimension, when all costs are accounted for. That may or may not be true; we simply do not know., because we are not privy to the data that they used to make this determination. It is true that it actually costs participating schools more than we charge. The most expensive part of the additional charges is the cost of the faculty members whose time -- in whole or in part --must be dedicated to implementation of the America's Choice Design, like the Literacy Coordinator. The reality is that, in most cases, these are faculty members already on the staff who are redirected to this work from similar work that they are already doing, like Title I teachers or reading resource teachers, so there is no net additional cost for the school. Because we do not know how the AIR team treated these expenses, we cannot comment on its accuracy.