An Educators' Guide to Schoolwide Reform - Home
An Educators' Guide to Schoolwide Reform - Introduction
An Educators' Guide to Schoolwide Reform - Acknowledgments
An Educators' Guide to Schoolwide Reform - Overview
An Educators' Guide to Schoolwide Reform - 24 Approaches
An Educators' Guide to Schoolwide Reform - Catalogs & Reviews
An Educators' Guide to Schoolwide Reform - References
An Educators' Guide to Schoolwide Reform - Appendices
An Educators' Guide to Schoolwide Reform - Ordering Information

An Educators' Guide to Schoolwide Reform

APPENDIX D
CRITERIA TO EVALUATE SUPPORT
DEVELOPERS PROVIDE SCHOOLS

This report presents information on four aspects of implementation: 1) the support developers report they provide schools; 2) the steps developers require or encourage schools to take to implement the approaches; 3) the scope of implementation in terms of how long the approach has been in schools and how many schools are implementing it; and 4) findings from research on implementation. Although the report contains information on all four aspects, only the first—support that developers provide schools—is rated.

Support the Developer Provides Schools

Review Process

To rate the level of support developers1 provide schools as they implement the approaches, AIR reviewed materials developers provided as well as information gathered in telephone interviews with each developer. Each researcher was responsible for several of the 24 approaches. For each approach, the researchers asked the developer a series of questions on professional development and technical assistance, reviewed informational materials provided by the developer, and conducted telephone interviews with at least three schools using the approach (see Appendix G). The researchers then described the findings in the profiles (see Support the Developer Provides Schools) and assigned a rating to each approach based on these findings and the rating criteria described below. The project developer reviewed all ratings, in many cases rereading original materials or speaking with approach developers to clarify ambiguous points. Developers were given an opportunity to review their profiles, including the support ratings, and provide supplementary evidence if they felt their approaches had been mis-rated.

Review Criteria

AIR considered the following factors in developing the rating criteria:

  • Access to appropriate types of support: Most developers offer a variety of types of support (e.g., visits from the developer, newsletters, telephone consultation with the developer, access to the developer's Web site) to meet a range of needs. Some developers provide proactive, on-site assistance, to help schools work through issues with implementation before problems undermine the implementation process.

  • Frequency and duration of support: Frequency of contact and duration of support indicate the level of interaction schools might expect of the developers. Developers may work closely with each school, or may provide an approach and encourage the schools to work independently towards reform.

  • Tools to help schools evaluate their implementation progress: Some developers provide benchmarks against which schools can monitor their implementation of the approach. Others help schools develop their own implementation plan, including schedules for expected progress.

Based on these factors, the following rating criteria for evaluating the implementation support developers provide schools were developed:

full.gif (85 bytes) = Strong support for schools

Developers provide training before implementation.

During the first year of implementation, developers schedule at least four on-site technical assistance visits to provide schools with regular guidance OR developers provide extensive training (e.g., at least one week in the first year) to a full-time on-site facilitator.

Developers provide off-site support (e.g., technical assistance through an interactive Web site, on-line assistance, over the telephone, or at off-site meetings).

Developers provide supportive materials (e.g., newsletter, non-interactive Web site).

Developers provide training and technical assistance support for at least three years.

Developers provide an indicator system or help schools develop their own indicator system to track implementation progress.

half.gif (93 bytes) = Promising support for schools

Developers provide training before implementation.

During the first year of implementation, developers schedule at least one on-site technical assistance visit to provide guidance OR developers provide extensive training to a part-time on-site coordinator.

Developers provide off-site support.

Developers provide support for at least three years.

quarter.gif (91 bytes) = Marginal support for schools

Developers provide training before implementation.

In addition to training before implementation, developers provide access to on- or off-site support in response to schools' requests through the first year of implementation.

empty.gif (88 bytes) = Weak support for schools

Developers do not provide any specialized technical assistance beyond training before implementation.

As AIR applied these rating criteria to the information developers had provided, we clarified three areas of ambiguity. The first relates to developers' efforts to support schools' full implementation of their approaches. Many developers help schools evaluate the outcomes of their approaches, particularly changes in student achievement. But these evaluations may not be sufficient to support the implementation process. Therefore, the rating system also focuses on developers' efforts to help schools monitor and/or evaluate their implementation of the approaches by tracking changes in curriculum, instruction, governance, school organization, and so on.

The second area of ambiguity is the distinction between proactive and reactive assistance. Most developers emphasize that they are available in response to questions or concerns voiced by school staff. To earn a high support rating, however, the developer must provide frequent guidance throughout the process as a matter of policy, without waiting for problems to surface.

Finally, some developers train on-site coordinators rather than sending their own staff to schools for technical assistance visits. These rating criteria recognize that the developers' requirements for coordinators affects the level of implementation support that schools experience. If a developer requires the coordinators to make a full-time commitment, and trains them accordingly, the approach receives a higher implementation support rating than if the developer requires the coordinators to make a part-time commitment, and trains them accordingly.

Implementation Requirements

Each profile describes the steps that developers require or strongly recommend schools take as they implement the approaches (see Implementation Requirements and Schools' Experiences). This information was drawn from interviews with the developers (see Appendix G) and supplemented with informational materials provided by the developers.

Scope of Implementation

The year the approach was first used in a school and the number of schools using the approach as of October 30, 1998 are reported in the profiles and on the table on page 4 of the guide.

Research Findings

In addition to information on implementation from the developers, AIR examined implementation successes and challenges reported in studies. This information is reported in Appendix E, and summarized in the profiles (see Implementation Requirements and Schools' Experiences).

AIR drew implementation data from studies that: reported student achievement outcomes (which also were reviewed for methodology, using the EREA process described in Appendix G); and focused on implementation exclusively.2 AIR reviewed these studies in different ways. For the outcome studies, we evaluated whether the schools in the studies fully implemented the most important aspects of the approach. For implementation studies, we evaluated implementation of the most important aspects of the approach, as we had done with outcome studies, and we also summarized major findings using a series of questions on implementation. These steps—evaluating implementation and summarizing implementation findings—are described below.

Evaluating Implementation

To evaluate the information each study contained about schools' implementation of an approach, AIR: 1) identified critical components of each approach; 2) reviewed studies for evidence that schools had implemented these components; 3) assigned an implementation rating for each component; and 4) assigned an overall implementation rating. Each researcher was responsible for reviewing outcome and implementation studies for several approaches.

To identify critical components of the approaches, AIR researchers interviewed the developers (see Appendix E for the Interview Guide) and reviewed promotional materials. During telephone interviews, the researchers asked the developers to describe the following elements of the approach:

  1. time and scheduling requirements;
  2. student placement and scheduling requirements;
  3. staffing and professional development requirements;
  4. instructional materials;
  5. professional development;
  6. procedures to monitor student progress and performance;
  7. administrative support needed to support implementation; and
  8. family or community involvement.

AIR researchers asked each developer to rate each element as "critical," "not critical," or "not relevant" to successful implementation.

Next, the researchers reviewed both outcome and implementation studies for evidence that the schools in the studies had implemented the approaches consistent with the developers' expectations. For each approach, a researcher completed information about each component (e.g., time and scheduling requirements); indicated whether the component was "critical," "not critical," or "not relevant," to implementation according to the developer; noted relevant information about a school's implementation (e.g., "grouping was homogeneous, as recommended, but regrouping did not occur as frequently as recommended"); and judged whether the school's implementation of each component, was "good," "adequate," "poor," or "not clear."

The researcher then gave each study an overall implementation rating. A study was rated "good" if the implementation of all critical components was good and the implementation of all non-critical components was good or adequate; "adequate" if the implementation of all critical components was good or adequate and the implementation of one non-critical component was poor; "weak" if implementation of no more than one critical component was poor and the implementation of all other components was good or adequate; and "inadequate" if more than one critical component was poorly implemented.

The process of maintaining inter-rater reliability for outcomes studies is described in Appendix G; this process also applies to the implementation ratings section of the review. For implementation studies, the project director reviewed implementation ratings for 10 percent of the studies. In cases of discrepancy, the project director and researcher reviewed the data and reached consensus on the rating. In such cases, the discrepancies and supporting evidence for the final rating were documented.

In practice, this rating system was limited by the information provided in the studies. Many studies provided little or no information on the schools' implementation of components the developers considered critical.

Summarizing Implementation Findings

To summarize findings from implementation studies, AIR evaluated the implementation of critical components of each approach using the procedures discussed above. In addition, AIR researchers reviewed findings from each implementation study using the following questions as guidelines:

  1. How long had the school been using this approach at the time of the study?

  2. How did the school select the approach?

  3. If the approach involves a planning period, how long did the planning period last? Was that enough time to prepare? Too much time?

  4. How did the study evaluate quality of implementation? For example, did the study look at progress towards milestones? Survey teachers on their satisfaction with or the effectiveness of the approach?

  5. How did the school implement the approach—were all components operationalized simultaneously, schoolwide? Were some components or grades phased in? How did this approach to implementation seem to affect the quality of the implementation?

  6. What components of the approach are especially difficult to implement?

  7. Typically, what happens when the school tries to implement these difficult components?

  8. What components of the approach are especially easy to implement?

  9. What qualities of the approach/school/staff/technical assistance/etc. contribute to successful implementation?

  10. Did the school make any adaptations to the original design? If so, give an example of the adaptation, and describe the reasons for making that adaptation. Did the adaptation work?

  11. How much technical assistance did the school receive? What types of support? What was the source?

  12. How much professional development related to the approach did the school receive? What was the source?

  13. If the study includes information about costs, describe those data. What expenditures were beyond the expenditures described by the developer?

  14. Describe major findings of the implementation study, if they are not captured by the questions above.

Findings for each implementation study are reported in Appendix E, and findings across studies are summarized in the profiles.

1 For some approaches, such as Accelerated Schools and Direct Instruction, implementation support is available from sources other than the developer.

2 In many cases, the distinction between outcomes and implementation studies was ambiguous. AIR used this decision rule: studies that reported any outcome data at all were considered outcomes studies. Researchers reviewed those studies using the EREA (including the implementation rating). All other studies that reported implementation data were considered implementation studies. Some of the "outcomes" studies that were reviewed with the EREA focused primarily on implementation. These studies tended to not pass the EREA criteria for rigorous methodology. Although initially identified as outcomes studies, these studies also were reviewed as implementation studies. Thus, the studies whose classification was most ambiguous—both outcomes and implementation—were reviewed as both types.