An Educators' Guide to Schoolwide Reform

Overview

This guide was prepared for educators and others to use when investigating different approaches to school reform. It reviews the research on 24 "whole-school," "comprehensive," or "schoolwide" approaches.¹

Schoolwide reform is an increasingly popular school improvement strategy, especially for low-performing, high-poverty schools. In 1994, the U.S. Congress made it easier for schools to organize themselves in this way.  For the first time, schools with student poverty rates as low as 50 percent could use Title 1 funds to improve the entire school.² Then, in 1997, Congress authorized an additional $145 million per year to help low-performing (mostly Title 1) schools raise student achievement by adopting "research-based, schoolwide" approaches.³

Thousands of schools across the country have begun to consider schoolwide reform seriously.  Unfortunately, it is difficult for these schools to obtain accurate, objective information they can use to decide whether schoolwide reform is right for them and, if so, which approaches are most likely to meet their needs. This guide provides that information.

While several reports catalog available schoolwide approaches, and some even provide limited evaluations, this is the only guide that rates the approaches against a common set of high standards or compares them to one another in terms of scientifically reliable evidence. This work was conducted by the American Institutes for Research (AIR) under contract to the American Association of School Administrators, American Federation of Teachers, National Association of Elementary School Principals, National Association of Secondary School Principals, and National Education Association.  These organizations used an open, competitive process to select AIR.⁴ The sponsoring organizations set the scope for the original project and, with the scientific advisors, provided feedback on the methodology. AIR conducted the ratings independently.

An at-a-glance summary of all 24 approaches opens the guide. This one-page table provides a snapshot of the approaches' relative strengths in three areas: (a) evidence of positive effects on student achievement; (b) support developers provide schools as they adopt the approaches; and (c) first-year adoption costs.

Brief profiles of the approaches follow. Each profile explains the ratings the approach received, provides additional information about its key features, and includes the name and address of its developer.

The guide's reference section lists papers, articles, and books about the approaches and their effectiveness. In Appendices A through G, readers will find detailed information on the procedures AIR used to review and rate the approaches, plus tables that provide additional information about each approach's effects on student achievement, implementation, and costs.  This guide is not meant to endorse, favor, or discredit any of the approaches.  Rather, it is designed to assist readers who want to critically examine the most widely available schoolwide reform approaches.  Schools can improve their performance in a variety of ways, not just by using a schoolwide approach.  However, educators interested in these approaches should find the guide useful.

TWENTY-FOUR SCHOOL REFORM APPROACHES AT A GLANCE

The table below summarizes the reviews of all 24 schoolwide approaches.

The information it provides is limited. To fully understand the ratings, readers are encouraged to review the profiles. The table provides a snapshot of the information that was available at the time this guide was prepared.  Developers continue to modify and strengthen the approaches and to gather and report information on their effectiveness.

Reading the Table

The table presents information in five areas: evidence of positive effects on student achievement; year each approach was introduced in schools; number of schools using the approaches; support developers provide schools; and first-year adoption costs (high and low estimates).

Approaches are rated in two of these areas: evidence of positive effects on student achievement, and support developers provide schools.  A full circle () indicates a strength. An empty circle () indicates a weakness.  The half-filled circle () suggests promise, and the quarter circle () indicates a marginal rating. The question mark (?) indicates that the research evidence is so limited that there really is no evidence on which to assess effects on student achievement or the support the developer provides schools.

The table can be read by rows and by columns. For example, the row on Support Developer Provides Schools indicates that the developers of 12 approaches provide schools with "strong" implementation support. Ten developers provide "promising" support for implementation, and two provide "marginal" support.

The columns in the table provide an at-a-glance summary of each approach.   For example, the column on Core Knowledge indicates that schools have used this approach since 1990. There are currently 750 Core Knowledge schools. A review of the studies that have examined Core Knowledge schools indicates it has "promising" evidence of effects on student achievement, provides schools with "promising" support for implementation, and is moderately expensive to put in place.

To the right side of the table, the column on Talent Development High School indicates that ten schools are using this approach, which was first introduced to schools in 1994. Studies of Talent Development High School show "marginal" evidence of effects on students and "strong" support provided by the developer for implementation. Costs to adopt this approach are moderate.

Understanding the Ratings

Evidence of Positive Effects on Student Achievement. The first row of the table presents ratings for Evidence of Positive Effects on Student Achievement. Each approach is rated on a five-point scale ranging from "strong" evidence of positive effects to a "no research" rating, which indicates there are no rigorous studies on which to judge the approach's effects on student achievement. Each rating is explained in the key below, and discussed in detail in Appendix A.

The ratings were compiled from individual reviews of available studies, each of which was ranked according to methodological criteria such as sample size, duration of the study, appropriateness of comparison groups, and relevance of measurement instruments.

In general, evidence of positive effects on student achievement—arguably the most important feature of any reform approach—is extremely limited. Even though many of the approaches have been in schools for years, only three provide strong evidence of positive effects on student achievement.  As a result, educators often are considering schoolwide reform without vital information on which to make decisions. More rigorous evaluations are needed, with broad dissemination of findings.⁵

Support Developer Provides Schools. The fourth row of the table presents ratings for the Support Developer Provides Schools. This four-point rating ranges from strong implementation support, in which developers provide a range of services, to weak support, in which developers only provide initial training for school staffs. Higher ratings indicate that the developers provide access to appropriate types of support, frequent support over an extended time, and tools to help schools evaluate implementation.⁷

Each rating is explained in the key below, and discussed in detail in Appendix D.

It is important to note, however, that some developers may not provide extensive support because schools do not need it to implement a particular approach successfully. To get a full picture of what kinds of changes are required of schools and what kinds of support are offered, readers should compare profiles.

Summary Table of All 24 Approaches

	Evidence of positive effects on student achievement2	Year introduced in schools	Number of schools	Support developer provides schools	First year costs3	First-year costs with current staff reassigned
Accelerated Schools (K-8)		'86	1000		$27	$14
America's Choice (K-12)	?	'98	300		$190	$90
ATLAS Communities (PreK-12)	?	'92	63		$98	$90
Audrey Cohen College (K-12)	?	'70	16		$161	$86
Basic Schools Network (K-12)	?	'92	150		$12	NC
Coalition of Essential Schools (K-12)		'84	1000		NA	NA
Community for Learning (K-12)		'90	92		$157	$82
Co-NECT (K-12)	?	'92	75		$588	NC
Core Knowledge (K-8)		'90	750		$56	NC
Different Ways of Knowing (K-7)		'89	412		$84	NC
Direct Instruction (K-6)		Late 60's	150		$244	$194
Expeditionary Learning Outward Bound (K-12)		'92	65		$81	NC
The Foxfire Fund (K-12)	?	'66	NA		$65	NC
High Schools That Work (9-12)		'87	860		$48	NC
High/Scope (K-3)		'67	27		$1304	NC
League of Professional Schools (K-12)		'89	158		$13	NC
Modern Red Schoolhouse (K-12)	?	'93	50		$215	NC
Onward to Excellence (K-12)		'81	1000		$72	$60
Paideia (K-12)		'82	80		$146	$96
Roots and Wings (PreK-6)		'93	200		$270	$70
School Development Program (K-12)		'68	700		$45	$32
Success for All (PreK-6)		'87	1130		$270	$70
Talent Development High School (9-12)		'94	10		$57	$27
Urban Learning Centers (PreK-12)	?	'93	13		$169	$159
= Strong      = Promising      = Marginal      = Mixed,Weak     ?= No Research     NA= Not Available      NC= No Change
1 This table summarizes information from An Educators' Guide to Schoolwide Reform. 2 Although many types of student outcomes are important, evidence of positive effects on student achievement is a key consideration in selecting schoolwide reforms. However, some schools may wish to consider a new approach that has not yet developed strong evidence of effectiveness, but provides the strongest match with school goals. 3 Costs are in thousands of dollars (e.g., $62=$62,000). 4 The estimate for High/Scope assumes a school of 25 K-3 teachers.
EXPLANATION OF "24 SCHOOLWIDE REFORM APPROACHES AT A GLANCE" TABLE Evidence of positive effects on student achievement: To arrive at this rating, studies that report achievement effects were reviewed. The studies report student achievement on: standardized tests, including mandated statewide assessments; assessments embedded in a specific curriculum; teacher-designed assessments; reading inventories; and the National Assessment of Educational Progress. Evidence of effects on achievement also includes students' course-taking patterns; daily attendance rates; graduation rates; within-grade retention rates; and letter grades.  The final rating reflects the amount of rigorous research and the strength of the findings from that research. Year introduced in schools: This date refers to the year schools first used the approach. This is included so that readers can judge whether the ratings are influenced by the newness of the approach. Number of schools: This reflects the number of schools developers reported were using each approach as of October 30, 1998. This includes all schools that are using the approach, regardless of the length and level of implementation. Support developer provides schools: This rating reflects: (a) access to appropriate types of support (e.g., staff development, technical assistance visits, materials, Web site access); (b) frequency and duration of support; and (c) availability of tools to help schools monitor their implementation of the approach. First-year costs: This estimates costs associated with developers' requirements or recommendations related to: (a) additional personnel schools must hire to meet developers' staffing requirements; (b) curriculum and other materials; and (c) staff development. Unless otherwise noted, costs are based on an estimated school size of 500 students and 25 teachers and other certified staff. First-year costs with current staff reassigned: This estimates costs for schools that are able to implement the approach by reassigning existing personnel, rather than hiring additional staff, to meet developers' staffing requirements. This includes costs associated with developers' requirements or recommendations related to: (a) curriculum and other materials, and (b) staff development. Most, if not all, of the approaches have been  implemented by schools with schoolwide Title 1 funding.  Such schools may be able to implement the approaches using existing resources, including staff. Special Note: Readers are encouraged to use this table in conjunction with the entire guide, which explains in detail how the approaches were reviewed and rated. The guide provides information about key features specific to each approach, as well as detailed information on: the costs of adopting the approaches; studies documenting effects on students; experiences reported by school staffs; and the characteristics of schools and students using each approach.

UNDERSTANDING THE CONTEXT FOR THE RATINGS

The table provides contextual information about the approaches in three areas: the year each approach was first introduced in schools, the number of schools using each approach, and first-year adoption costs.

Year Introduced in Schools. The second row of the table shows the year each approach was first introduced in schools. This information provides some context for interpreting findings. Depending on the approach, it may take three years to fully implement and an additional two years to evaluate.  Approaches that have been in schools for five or fewer years may have lower ratings for Evidence of Positive Effects on Student Achievement than approaches that have operated in schools for some time because there has not been a sufficient period of time for such data to be collected.

Number of Schools. The third row shows the number of schools across the country that had implemented the approaches as of fall 1998. These are the schools that each developer reports are using their entire approach; it does not include schools that only use materials or implement selected components of an approach.

The number of schools using an approach is one indication of whether it can be replicated or implemented in more than one setting. It also is included so that readers can compare the total number of schools using a particular approach to the number that have been involved in research studies of that approach (see Appendix C). The number of schools studied should be large enough (and sufficiently representative of all schools using the approach) so that educators are confident that research findings provide an accurate picture of the approach's strengths and weaknesses.

First-Year Adoption Costs. The fifth and sixth rows in the table highlight the first-year costs of adopting the approaches. The costs are presented in two ways. The first—labeled first-year costs—is the higher of the two figures. It estimates schools' costs if they must hire additional personnel to meet developers' staffing requirements. For example, some developers require schools to have a full-time school-based facilitator or coordinator. If current staff members cannot fill these roles, schools need to hire additional staff. The second cost estimate—first-year costs with current staff reassigned—is the cost schools incur if they are able to reassign current staff members to meet developers' staffing requirements. Unless otherwise noted, costs are estimated for a school with 500 students and 25 teachers and other certified staff.

HOW THE APPROACHES WERE SELECTED

Five criteria were used to select the 24 approaches reviewed:

1. They are promoted by their developers as a means to improve student achievement in low-performing schools.

2. They are mentioned by name in the federal legislation that created the Comprehensive School Reform Demonstration Program.⁸

3. They are used in many schools and districts.

4. They have obtained national visibility in the education and popular press.

5. There is some research evidence about their effects on students and/or their implementation in schools.

6. Seventeen approaches are reviewed because they are mentioned in the Comprehensive School Reform Demonstration Program legislation. The other seven approaches satisfy at least three of the other criteria.⁹

HOW THE APPROACHES WERE REVIEWED AND RATED

To prepare the table above and the profiles of each approach, AIR: (a) interviewed the developers of each approach (see Appendix G for details); (b) gathered and reviewed all available studies about the approaches (see Appendices A and D); and (c) collected additional information from schools that are using the approaches (see Appendix G). All profiles and tables were reviewed by the developers for accuracy, revised, and reviewed again. In addition, developers were offered the option of submitting written comments (see Appendix H).

For each approach, AIR reviewed: (1) evidence of positive effects on student achievement; (2) support that the developer provides schools as they implement the approaches; and (3) first-year costs associated with implementation.

EVIDENCE OF POSITIVE EFFECTS ON STUDENT
ACHIEVEMENT

AIR made an extensive effort to gather and review all relevant material for each approach. AIR reviewed three types of materials (loosely called studies) to determine whether they were methodologically sound and to assess the positive student achievement findings:

• Individual studies that reported a broad range of achievement outcomes. For example, different studies reported students' performance on one or more types of measures (e.g., standardized tests, mandated statewide assessments, assessments embedded in a specific curriculum, and teacher-designed assessments). Some studies also reported achievement outcomes in terms of students' course-taking patterns, daily attendance rates, graduation rates, referrals to special education, within-grade retention rates, or letter grades.

• Studies that are available to the public. In all cases, the studies reviewed were available to school staff or others who might want to read them. In many cases, studies were unpublished but were available from the developers.

• Changes the developers reported in test scores. AIR reviewed raw test score data provided by the developers if they were accompanied by contextual information that could be used to interpret them (e.g., number and grades of students tested, changes in student population over the testing period, and description of implementation). Many developers collect test data that suggest their approach raises test scores, but few provide adequate information to relate changes in test scores to the approach.

AIR reviewed more than 130 studies of student achievement. Each approach was rated on its number of sound studies and the positive student achievement findings in those studies.¹⁰

The review of the research occurred in two stages. First, AIR critically reviewed studies for methodological rigor, taking into consideration important distinctions among studies such as scope (e.g., number of students, period over which data were collected), quality and objectivity of the measurement instruments, and affiliation of the researcher. Second, AIR assigned to each approach an overall rating for positive evidence of effects on student achievement, based on the number of studies that met the criteria for methodological rigor and the effects reported in those studies.¹¹

In the first stage, AIR critically reviewed all studies that reported student achievement effects for one of the 24 schoolwide reform approaches, using an instrument developed for this type of research review and tailored to this project, the Evaluation of Research on Educational Approaches (EREA).  Each study was reviewed individually, using the EREA, by one of seven trained researchers.

Each AIR researcher was assigned several reform approaches and reviewed all of the studies for these approaches. In addition, each researcher reviewed a sample of studies reviewed by fellow researchers to maintain inter-rater reliability. Two or more researchers reviewed one out of every five studies.  The project director compared ratings for these overlapping studies and, in cases of discrepancies, retrained the researchers and clarified the issues for all project staff.

SUPPORT DEVELOPER PROVIDES SCHOOLS

To rate the level of support developers provide schools as they implement the approaches, AIR considered the following factors when reviewing materials developers provided and information gathered in telephone interviews with each developer:

• Access to appropriate types of support. Most developers offer a variety of types of support (e.g., visits from the developer, newsletters, telephone consultation with the developer, and access to the developer's Web site) to assist schools as they implement the approaches. Some developers provide proactive, on-site assistance to help schools work through issues with implementation before problems undermine the implementation process.

• Frequency and duration of support. Frequency of contact and duration of support indicate the level of interaction schools might expect of the developers. Developers may work closely with each school, or may provide an approach and encourage the schools to work independently towards reform.

• Tools to help schools evaluate their implementation progress.  Some developers provide benchmarks against which schools can monitor their implementation of the approach. Others help schools develop their own implementation plan, including schedules for expected progress.

However, all approaches do not require the same amount of support from developers, and some are more demanding to implement than others.  Therefore, the guide provides information on: (1) the critical steps developers require or encourage schools to take to fully implement the approaches; (2) the scope of actual implementation—that is, how long the approach has been in schools and how many schools are implementing the approach; and (3) findings from research on implementation (e.g., components of the approach that are especially easy or difficult to implement). This information can be found in the table above, in the profiles of each approach, and in Appendix E.

ESTIMATING THE FIRST-YEAR COSTS OF ADOPTING THE APPROACHES

The cost estimates in the guide are for the first year a school implements an approach. In most cases, the first year is the most expensive. AIR estimated the total amount schools can expect to pay in addition to their normal operating expenses. In some cases, this cost can be reduced if schools are able to reallocate current personnel and/or fiscal resources to support their adoption of a schoolwide approach.

To arrive at cost estimates for the first year, AIR asked each developer to provide information on the costs schools generally incur. In addition, a random sample of schools that had adopted each approach were asked to share estimates of the costs that were actually incurred. AIR confirmed or adjusted the developers' figures based on the schools' cost figures and information from studies that included cost information.

Unless otherwise noted, costs are based on a school of 500 students and 25 teachers or other certified staff, and assume that the approach is implemented across the entire school. The cost estimates also include all components the developers require, or strongly recommend, that schools adopt during their first year of implementation. For example, if the developer requires three days of professional development for each teacher, the estimates include the cost of the training sessions and the cost of release time for 25 teachers for three days each.

Costs are based on standard rates. For example, an average rate for a full-time facilitator or for a certified teacher is $50,000 per year. One day of release time for a teacher is based on a salary of $50,000/180 days, or $278 (Barnett 1996; King 1994; Odden undated).

Two cost estimates appear in the summary table above and in the profiles of each approach. The first, a high estimate, includes all the approach-related costs schools can expect to incur (i.e., training, technical assistance, materials, and additional staff). Most importantly, it estimates the cost of any additional personnel that schools may need to hire to meet developers' requirements. The second cost estimate, a lower dollar figure, acknowledges that some schools can reassign current staff members rather than hire additional staff.

Schools could spend less than the low estimate if, for example, release time for training is covered by regular district allocations, or the approach is implemented in selected grades rather than all grades in the first year. Schools could spend more than the high estimate if, for example, the school opts for additional training not required by the developer.

BEFORE SELECTING A SCHOOLWIDE REFORM APPROACH

Local policymakers, educators, parents, and community members make their decisions about schoolwide reform based on many different factors. Such decisions can be informed by seven steps, outlined below.

Step 1: Identify the school's needs. Many schools engage in self-studies or audits as part of the process of investigating available schoolwide reforms.  These procedures sharpen the schools' understanding of what is working well and what needs to be improved.  They also generate goals that schools can match against the goals (and documented effects) of various schoolwide approaches. Self-studies and audits also establish a starting point or baseline for measuring progress once an approach is adopted.

Step 2: Investigate alternative approaches. In order to identify approaches that can meet their needs, schools should investigate a number of approaches to reform.¹² First, approaches vary considerably in their philosophies, components, and ways of working with schools. Second, by casting the investigative net widely, schools increase their chances of finding an approach that matches their goals. More than an academic exercise, procedures should be used to enable schools to pinpoint exactly what kind of effects on students they can expect if they implement a program as the developer intends (e.g., improved reading, math, or science achievement; better attendance; higher graduation rates; or fewer students receiving special education services). To make an informed selection from among the alternatives, staff members need a good understanding of all the approaches under consideration. This guide is one resource for developing that understanding. Other resources are noted in a special section in the references.

Step 3: Ask the developers questions. The 24 approaches are changing constantly as new components are added or existing components are revised.   Schools should contact the developers for updates. In addition, most of the approaches are expanding to more and more schools as interest in schoolwide reform grows. Since developers may have too many interested schools to support every one, schools should contact developers early in the process of exploring approaches to find out what kind of support will be available and when that support can be provided.

Step 4: Call a random sample of schools that are using the approaches. It is wise to ask the developers for a full list of schools that are using their approaches (and schools that have discontinued using the approaches), including addresses, phone numbers, and the name of the appropriate person to contact. Some developers are willing to provide such lists.

Phone calls to a random sample of schools can reveal: why the schools chose the approach; how they addressed any implementation problems they have experienced; what it cost them to implement the approach; and, most importantly, what kind of results they have seen, especially in terms of positive effects on student achievement. Schools already implementing the approach also may have evaluated the training and support the developer provided, and they can share this information. By contacting as many schools as time permits, especially if the developers provide limited information on the approach's effects on students, the investigating school can better understand the approach.

Step 5: Visit schools whenever possible. Steps 1 through 4 should narrow the search for an approach that best meets a school's needs, which makes it possible to arrange school visits for the top few choices. Most developers recommend that schools take this step. School visits provide an opportunity to see the approach in action: to visit classrooms; to meet with school staff; and to talk to students, parents, and community members. They also give would-be adopters a sense of how an approach operates in different schools, and if visitors' resources permit, in different school districts and communities. While approaches that offer highly structured curricula and instructional strategies may look quite similar across settings, those that advance a philosophy or general approach to school reform may operate differently in different settings.

Step 6: Match the developer's requirements with available resources.  The cost information in this guide is a good starting point for comparing resources and requirements. It captures the out-of-pocket costs that developers typically charge for training, consultation, and materials, and it flags less-obvious costs that schools need to consider (e.g., hiring new staff, purchasing additional books for libraries, and asking faculty to develop curriculum materials). But the actual costs of adopting and sustaining an approach vary widely across schools and districts.

For example, the costs of training and technical support from the developer often vary, depending on whether expenses can be shared among local schools adopting the approach. Economies of scale apply when clusters of schools implement the approach at the same time. Some developers allow schools to tailor their implementation by phasing in components (and their associated costs) over time; others do not. Some approaches ask instructional staff to enroll in college courses to update their subject-matter knowledge; others ask them to meet before and after school, or to work extended hours. Such requirements have cost implications.

To determine the feasibility of adopting an approach, compare all the requirements developers ask schools and districts to meet with available resources.

Step 7: Put the decision to a vote. Because staff support is critical to success, the decision to adopt a schoolwide reform approach should involve the entire staff and be supported by a large majority. Many developers recommend that the staff vote on adopting an approach. In fact, many require that 80 percent or more of the staff vote for adoption, by secret ballot, before they will agree to work with a school.

The information in this guide is central to the second step, investigating alternative approaches. The guide will assist readers as they: examine an approach's track record for improving student achievement; learn about the support developers provide schools and districts before and after adoption; and develop an understanding of the costs involved in adopting an approach.

USING THIS GUIDE TO INVESTIGATE SCHOOLWIDE REFORM

This guide includes much of the basic information readers need to investigate the 24 approaches reviewed: (a) the table above; (b) the profiles of each approach; and (c) the more detailed information in the appendices about studies of the programs' effects on students, implementation requirements, and costs.

To support readers' further investigation, the guide references two additional sources of information: (a) the developer of each approach; and (b) additional readings on each approach. The developer's name and address, including its Web site address, is listed at the end of each profile. Papers, articles, and books written about the approaches are listed in the references.

Busy readers are cautioned against relying exclusively on the summary table.  The table provides limited information that is not sufficient to understand the ratings fully. Readers need to read each of the profiles of the approaches to understand the ratings.

Here are some suggested ways to use this guide:

• Consult the summary table to compare the ratings assigned to the approaches.

• Read the profile of each approach to understand the ratings assigned and the costs of adopting the approach. In addition, the profiles provide an overview of each approach and describe its major components.

• See Appendix B for ratings of the studies AIR reviewed.

• Review Appendix C for details about studies of the approaches' effects on students.

• Review Appendix E for information from studies that document schools' experiences implementing the approaches.

• See Appendix F for details about the costs of adopting the approaches.

• See the brief discussion above to find out how the approaches were selected for this guide.

• Turn to Appendices A and D for details about how the approaches were reviewed and rated.

• Consult the guide's references to read more about specific approaches.

• See the special section in the references to find out about other guides that describe the reform approaches.

• Follow the steps outlined in Before Selecting a Schoolwide Reform Approach to make an informed decision.

CONCLUSION

This guide is a tool to help schools investigate schoolwide reform approaches.  Information is presented in varying degrees of detail, from the summary table  to the detailed descriptions of the research in the appendices. We hope that readers will find it useful.

¹ According to the Comprehensive School Reform Demonstration Program ("Obey-Porter"), "comprehensive" approaches integrate nine components: effective research-based methods and strategies; comprehensive design with aligned components; professional development; measurable goals and benchmarks; support within the school; parental and community involvement; external technical support and assistance; evaluation strategies; and coordination of resources.
² In 1991, corporate America launched an ambitious research and development program, the New American Schools Development Corporation (now New American Schools), to develop several new approaches to school reform. NASDC design teams began piloting these approaches in 1992.
³ The Comprehensive School Reform Demonstration Program provides funds to states for grants to schools of at least $50,000 per year for three years for whole-school reform.
⁴ The project sponsors mailed a Request for Qualifications to 15 nonprofit research and development organizations on May 8, 1998. Statements of qualifications were submitted on May 22. AIR was awarded the contract, which began on June 1. AIR is an independent, nonprofit research and development organization based in Washington, D.C.
⁵ It is important to note that reform approaches that are operating in a large number of schools or have been operating for a long period of time are better situated to provide a strong research base.
⁶ Due to the limited data in many subject areas, ratings are based on evidence of positive effects on student achievement across subject areas, rather than within specific subjects.
⁷ In some cases, sources other than the developer, such as regional assistance centers or publishers, also provide support. The ratings do not reflect support from these sources.
⁸ The programs mentioned in the Comprehensive School Reform Demonstration Program legislation are: Accelerated Schools, America's Choice, ATLAS Communities, Audrey Cohen College, Coalition of Essential Schools, Community for Learning, Co-NECT, Direct Instruction, Expeditionary Learning Outward Bound, High Schools That Work, Modern Red Schoolhouse, Paideia, Roots and Wings, School Development Program, Success for All, Talent Development High School with Career Academies, and Urban Learning Centers.
⁹ Preliminary analysis was also conducted on the Edison Project and Montessori. After reviewing the results, however, both developers presented AIR with convincing arguments that additional data needed to be reviewed to arrive at an accurate rating of student achievement effects. The scope of this additional data gathering and analysis could not be accommodated within project timelines. Thus, neither approach is included in this Guide. We hope to include both in the next edition.
¹⁰ The rating criteria draw on multiple sources, including Stringfield (1998), National Center to Improve the Tools of Education (1998), and U.S. Department of Education (1998). The project's advisors and experts in educational evaluation reviewed draft versions of the criteria used to rate the evidence for each approach. The final criteria reflect their comments and suggestions.
¹¹ The reported analyses treat all studies equally. In fact, studies differ on a number of dimensions, including the number of schools studied, the number of grade levels tested, the number of outcome measures used, and the number of students included.   Given the available data, it was not possible to take all of these sources of variation into account in the analysis. But, to examine the sensitivity of the results to alternative assumptions about the studies, a supplementary analysis was conducted in which studies were weighted based on the number of schools included. The weighting made very little difference in the overall results.
¹² Some schools may choose to develop their own schoolwide reform rather than use a preexisting model. This guide, however, focuses on externally developed approaches to schoolwide reform.