An Educators' Guide to Schoolwide Reform

An Educators' Guide to Schoolwide Reform

APPENDIX G
DATA COLLECTION ACTIVITIES

Data collection activities consisted of three major tasks: collecting available documentation and research on all of the approaches, interviewing the developer of each approach, and interviewing personnel from a random sample of three to five schools using each of the approaches.

Document Collection

To provide a comprehensive and definitive review of the research available on each of the 24 schoolwide reform approaches, AIR:

Asked each developer to identify research related to their approach during a telephone interview. Copies of this research were then obtained.
Conducted a search of the Educational Resources Information Center database from 1976 to the present.

Reviewed the previous ten years of nine key education research journals, using the Reader’s Guide to Periodical Literature and the Education Index:

	American Educational Research Journal
	American Journal of Education
	Educational Leadership
	Educational Research and Evaluation
	Educational Researcher
	Harvard Educational Review
	Journal of Education for Students Placed at Risk
	Review of Educational Research
	School Effectiveness and School Improvement

Searched Education Week (1988-present) to locate references to approaches and research on the approaches.
Collected relevant materials from Robert Slavin and Sam Stringfield, Center for Research on the Edu-cation of Students Placed at Risk, Johns Hopkins University, who have previously performed overview research on comprehensive school reform approaches.
Located and obtained articles referenced for each approach in the Catalog of School Reform Models: First Edition (NWREL 1998).
Reviewed bibliographies of research articles collected through the above methods to locate further related research.
Asked colleagues in the following organizations to review the collected references to identify and help fill potential gaps:

Regional laboratories and research centers, through the National Education Information Knowledge Association;

RAND Corporation;

Representatives from each NAS design, through New American Schools; and

Educational Research Service.

AIR focused its review on research that had been conducted within the past 10 years. However, we expanded the range of years covered when an approach remained largely unchanged and the bulk of research related to the approach was older.

Developer Interview

Interviews were conducted with developers or developers’ staff for each of the 24 approaches using the following questions. Interviews lasted approximately 30 minutes and were conducted after reviewing background material on the approach, but before reviewing the research. The questions asked in the interview are listed.

Guide to Developer Interview

Rationale Underlying the Approach
1. Please describe the rationale or theory underlying the approach.
2. Please describe the goals of the approach with respect to student-level outcomes (e.g., achievement, enrollment, or affective outcomes).
3. Please describe the goals of the approach with respect to school-level outcomes (e.g., school climate).
Background Information
1. Describe briefly what happens in the classroom when the approach is used.
2. Which subject areas are included in the approach?
3. Which children is the approach directed toward?
4. How many children can one teacher serve with the approach in a school period?
5. How long has the approach been used in schools?
6. How widespread is the use of the approach (i.e., approximate number of schools and districts implementing the approach)?
Description of Components of the Approach (For each component, specify if it is critical to the success of the approach.)
1. Time and Scheduling Requirements (_critical _not critical _not relevant)
  1. What are start-up time requirements for the approach?
  2. Please describe the start-up process (e.g., whole-school, gradually phase in, pace of phase in).
  3. How many days of the school year is the approach to be used?
  4. How many minutes of instructional time are devoted daily to using the approach?
  5. How many sessions (days/minutes) of the approach are typically necessary to achieve educationally significant effects?
  6. What are the unique scheduling requirements?
2. Placement and Grouping Requirements ( critical not critical not relevant)
  1. How are children to be grouped for instruction? Whole class, small group, or individually? Homogeneously or heterogeneously?
  2. How many children are to be in groups?
  3. Does grouping transcend classroom boundaries?
  4. Describe procedure for determining instructional/developmental level of students.
3. Staffing Requirements (__critical __not critical __not relevant)
  1. Are additional personnel required in addition to existing classroom teachers? Specify the number of personnel, in which grades, and what qualifications and certification levels they should have.
  2. Is there a limit to the number of children per classroom? If so, what number?
4. Instructional Materials (__critical __not critical __not relevant)
  1. Specify the instructional materials needed for each classroom.
  2. Specify the source(s) of materials (e.g., developers, other publishers, teachers).
  3. Specify how much and what kinds of instructional materials teachers are expected to create/develop.
  4. Are appropriate materials available for children whose primary language is other than English? If yes, describe materials and why they are appropriate.
5. Professional Development (_critical _not critical _not relevant)
  1. Inservice training—prior to students beginning school year. What are the requirements for staff development prior to staff beginning to teach the approach (e.g., summer training)? Describe how many hours of training are required prior to each of the first three years of implementing the approach.
  2. Inservice training—during school year. What are the requirements for staff development during the school year? Describe how many hours of inservice training are required during each of the first three years of implementing the approach.
  3. In-class coaching. Is in-class coaching part of the approach? If so, how much coaching is to be provided for each teacher during each of the first three years of implementing the approach?
  4. Professional developers. Please describe the source(s) of professional development (e.g., developer, regional center, professional organization, contractor, district). What are the qualifications and availability of trainers/consultants who provide professional development?
  5. Technical assistance. What other forms of technical assistance are available, and from what source?
6. Monitoring of Student Progress and Performance (_critical _not critical _not relevant)
  1. Does the approach include ongoing monitoring of student progress and performance? Describe monitoring procedures.
  2. Does the approach provide remedies when student progress (and/or performance) is inadequate? Describe procedures.
  3. Are there provisions for accelerating high-achieving students? Please describe.
  4. Does the approach target any particular category of students (i.e., by SES, region, ethnicity, disability status, level of schooling, etc.)? Please describe.
7. Administrative Support (_critical _not critical _not relevant)
  1. What are the critical elements of support needed from the principal?
  2. What are the critical elements of support needed from the central administration?
8. Family or Community Involvement (_critical _not critical _not relevant)
  1. Please describe the family or community involvement called for by this approach.
9. Please describe other critical components of the approach (e.g., peer involvement, computer technology).
Cost of Approach (These data should be gathered in interviews with approach developers, independent research reviews, and, if necessary, random calls to sites implementing the approach.)
1. What are the extra costs associated with the approach for the first year of implementation? (Specify how many students the costs are for.)
  1. Additional personnel—Roles/titles, full-time, part-time.
  2. Materials—Required and supplementary, please describe.
  3. Staff Development—Release time, stipends, travel/per diem, trainer, fees, other expenses.
  4. Other Costs.
2. What are the extra costs for subsequent years?
  1. Additional personnel—Roles/titles, full-time, part-time.
  2. Materials—Required and supplementary, please describe.
  3. Staff Development—Release time, stipends, travel/per diem, trainer, fees, other expenses.
  4. Other Costs.
Information on Implementation
1. What information on implementation requirements and procedures is available to schools considering using this approach? (Select one.)
  1. General description of approach requirements.
  2. Non-detailed description of implementation requirements of the approach, including at least some of the following: staff development, curriculum, instruction methods, materials, and assessments.
  3. Specific description of implementation requirements and procedures of the approach, including all of the following: staff development, curriculum, instruction methods, materials, assessments, and costs.
2. What information on costs is available to schools considering using the approach? (Select one.)
  1. General information about approach costs.
  2. Estimated cost of approach, including whether or not costs of materials, staff development, additional personnel, etc. are included in the approach’s purchase price.
  3. Specific costs of the approach, including whether or not costs of materials, staff development, additional personnel, etc. are included in the approach’s purchase price.
Replication of Implementation
1. How stable and widespread is approach implementation? (Select one.)
  1. Not fully implemented at the original site.
  2. Fully implemented in the original pilot site(s) for a minimum of one school year.
  3. Fully implemented in the original site(s) for more than three years.
  4. Fully implemented in multiple sites for more than three years.
2. How many additional schools are expected to implement the approach according to the developer?
  1. 1998-1999 _______
  2. 1999-2000 _______
  3. 2000-2001 _______
  4. 2001-2002 _______
  5. 2002-2003 _______
Request Materials and Sites

Thank you very much for taking the time to talk about (approach). Thank you, also, for agreeing to send the materials we’ve requested. I have just two more requests. First, could you recommend articles and evaluations of the approach that we could read? Second, could you please give us a list of all of the sites that are implementing your approach, with contact names.

School Interviews

AIR conducted telephone interviews with staff at three to five randomly selected schools that had adopted each approach. A greater number of sites were chosen for those approaches that were more widely used. In addition, schools were chosen that were not in their first year of implementation. Fifteen-minute interviews were conducted using the following questions.

Guide to School Interview

How long has your school been using this approach, and why did your school select this particular approach?
In your opinion, what are the strengths of this approach?
Do you have any concerns regarding this approach?
(a). Has your school made any adaptations to the original design of the approach provided by the developer?
(b). Can you provide an example of a recent adaptation to the developer’s original design? [If answered "yes" to question 4(a).]

(c). Why did your school decide to make that particular adaptation?
(a). How did you begin implementing this approach in your school? [i.e., all at once, over time]
(b). What do you think were the benefits and disadvantages of this style of implementation?
(a). In your experience, is this approach equally effective for all of your students, or have you found it to be more/less effective with some? [i.e., gifted and talented students, special needs students]
(b). If there is a difference, what do you think explains that difference?

(c). Is this approach equally effective for all types of schools and school districts?
(a). Do you think most of your teachers share your feelings concerning this approach?
(b). If not, why not, and what do you think would help your teachers to use the approach more effectively?

(c). Has this approach either helped your teachers to work more cooperatively or created/exaggerated differences among staff? Please describe.

(d). Have staff attitudes concerning this approach changed over the course of the time you have been using it? Please explain.
(a). Discuss the costs of the approach. Namely, what are the extra costs associated with the approach for the first year of implementation? (Specify how many students the costs are for: _____).
1. Additional personnel—Full and part-time
2. Materials—Required and supplementary
3. Staff Development—Release time, stipend, travel/per diem, trainer, fees, other expenses
4. Other Costs
(b). What are the extra costs for subsequent years?
1. Additional personnel—Full and part-time
2. Materials—Required and supplementary
3. Staff Development—Release time, stipend, travel/per diem, trainer, fees, other expenses
4. Other Costs
Are your school’s costs the same now as when you first started using this approach? If not, what is the same and what is different?
Given a second chance, would your school spend its money in the same manner, or have you learned things while using this approach that would lead you to spend your money differently?