Feature

Growth Measures for Systemic Change

Through periodic learning assessments, you can analyze which instructional programs are most effective, make student groupings and reallocate resources to areas of need by Allan Olson

Educators are becoming more aware of the limitations of testing that simply measures student achievement at a single point in time, such as benchmark tests, locally constructed formative tests, conventional standardized tests, and state assessments used to determine adequate yearly progress under No Child Left Behind.

Not surprisingly, school administrators are implementing assessments to measure individual student growth during the school year and from year to year. In states like South Carolina, Indiana and Minnesota, anywhere between 60 and 90 percent of the school districts statewide are using a formative assessment based on growth measures to make informed decisions about each student’s education.

Accurate measures, combined with a vertical measurement scale, have provided an opportunity never before available to K-12 educational settings. Now not only do educators understand how their students are performing at one point of time, they can quickly see how much students have grown from quarter to quarter or from one year to the next. As a result, educators can understand and influence growth for all students, regardless of achievement status, age and class groupings. Likewise, growth measures point to how student achievement is aligned with district or state-defined content standards while ensuring instruction challenges each student appropriately.

The principal question is how growth data might be used to continuously improve the effectiveness of educational systems. Can we focus more clearly on each job function within a school district in ways that align those functions to improve student learning?

Access to accurate student growth data informs a wealth of decision making, including program and teacher effectiveness, the ability to challenge all students given their status, adequacy of instructional programs or resources, school staffing and scheduling, as well as the impact of environmental factors, such as bus schedules and classroom materials. With the use of accurate measures and timely access to the analysis of school/district progress, schools now can determine the amount and nature of academic growth that each student needs and then organize themselves to accomplish these learning goals.

Gauging Changes
At the most fundamental level, a growth measure provides educators a quantifiable way to gauge the difference in scores for a single student from one point in time to another. By comparison, the achievement level is the score that a student has at a fixed point in time, such as a standardized test score. The quality of a growth measure, however, depends on the integrity of the testing methodology and measurement scale.

Northwest Evaluation Association, a national nonprofit organization providing research-based assessments, has championed the use of computerized adaptive testing to measure growth in student achievement as the foundation for the Measures of Academic Progress, or MAP. It now is used by more than 2,200 districts nationwide.

MAP reports achievement on a RIT (Rasch Unit) scale, an equal-interval vertical measurement scale that enables educators to measure growth independent of grade level and to evaluate and compare performance data across years. The RIT is infinite, although most students’ scores fall between the values of 140 and 300. The scale is equal-interval, meaning the distance between 170 and 182 is the same as the distance between 240 and 252. This allows educators to apply simple mathematical equations to the scores to determine the mean and median scores in a class or grade.

RIT scores are used by teachers to plan instruction around students’ strengths and weaknesses relative to state curriculum standards. Educators can choose curricula that are aligned with NWEA’s Continuum of Learning, and therefore it becomes easier to place students in well-targeted, differentiated instruction, which leads to improved performance and growth.

The aggregated results from periodic assessments, ideally implemented at nine-week intervals, provide a wealth of information for administrators. In an analysis of instructional programs, educators can use growth data to determine the effectiveness of one instructional program over another, make decisions about the groupings of children given their current achievement levels and instructional needs and reallocate instructional resources. Likewise, it is possible to see whether programs are adequately effective for all disaggregated groups of students or perhaps just one subgroup.
With an analysis of school data, juxtaposed to a virtual comparison group of like students and districts, administrators can make informed, objective decisions about teacher, counselor and staff performance. Tough questions about teacher preparedness, school staffing levels and student/teacher assignments can be addressed objectively.

Virtual Comparisons
The National Heritage Academies, which operate 53 public charter schools in five states, including Indiana, Michigan, New York, North Carolina and Ohio, is an example of an organization using the NWEA virtual comparison group reports. The academy schools began using MAP growth measures in all of its schools in 2003. Individual student scores demonstrate how each student is progressing and learning over time, and these scores provide teachers with guidance on where to place students in the curricula and how to adjust instruction based on student needs.

By selecting a virtual comparison group, including comparable students based on metropolitan service district characteristics, grade levels and socio-economic status, the academy schools’ administrators and teachers can see how student growth compares with students nationwide in similar surroundings.

To date, students enrolled in National Heritage Academies are showing growth at the 73rd percentile on the last fall-to-spring administration. More importantly, the comparison data, in combination with MAP test scores, has revealed specific instructional areas in both math and reading in need of improvement. Likewise, these schools now can obtain a clear picture of growth at each of its 53 schools, thereby identifying high-growth schools with replicable best practices and lower-performing schools where additional resources should be applied.

“The comparative data provided by MAP supplements the data we receive from our students’ individual standardized state test results, allowing us to leverage this information to help all students achieve academic growth,” says Robert Theaker, director of assessment and measurement at the National Heritage Academies. “Overall, we see where grades are hitting home runs, where instructional practice has an impact on achievement and where we can adjust resources to help both teachers and students. We can now target both our professional development and instructional practice efforts most appropriately.”

From a building or district view, it is possible to learn whether time is used most effectively to generate student learning. Which schools are more or less effective? Are there environmental circumstances, such as long bus routes, meal services or facility conditions that have a negative impact on learning?

At the Beaufort County, S.C., School District, administrators have used growth measures to see how they can best use professional staff time. For example, the district discovered that when teachers did not have enough preparation time or adequate time to make thoughtful decisions about students and their instruction, there was a correlation with lower growth scores. The district thus changed structures regarding who does planning and when. Additionally, best practices were shared across buildings and the result has been improvements attributed to non-instructional factors.

The school district actively responded to its learning about time by implementing a calibration model for staffing, which ensures that adequate resources are in place to support learning. This model incorporates core, supplemental and intervention staffing requirements. By implementing this belief system, the district sought to increase the likelihood that staff are performing those roles most critical for student success.

A second strategy involved piloting a full-time substitute teacher program to ensure instructional time would not be compromised when the regular classroom teachers were absent from the classroom for professional development purposes. These full-time substitutes were trained on district initiatives, familiar with all policies and practices and developed relationships within the district’s schools. By addressing time in these ways, Beaufort County recognized the real issue related to time as a factor in determining student growth is about managing the time with students and staff well.

The Beaufort County schools combine results from the MAP growth measures with data acquired through a districtwide survey of professional and administrative staff, as well as food service employees and school bus drivers. The addition of data on process, as revealed in this survey, aligned with data on student growth, allows Beaufort administrators to make more informed decisions about school and district operations to produce greater student growth.

In short, an assessment built on growth measurement not only assures that educators can appropriately challenge all children and raise student learning, it also can provide the foundation for better decision making at a district level, thereby improving how schools are organized and programs delivered.

Answerability
The ability to organize educational institutions around individual student growth has to start with district leadership. School board members, superintendents and principals all play a role in setting policy, adopting programs and organizing the delivery of instructional services to meet the needs of children.

At the policy level, school boards must create a culture that is focused on the values and principles that foster constant improvement in learning. If the core values of the district are to organize and deliver instruction in a way that allows all children to be appropriately challenged, then specific curriculum, programs, schedule changes, teacher effectiveness or school performance can be assessed based on how well it fosters academic growth in students.

School boards, as well as district superintendents, then must own and manage a system of answerability. In the Horry County, S.C., school district, the superintendent and administrative staff would meet at least three times a year to examine student data regarding achievement, growth, growth targets, attendance, attitude and other criteria. The data are disaggregated across all of the important local factors, such as race, gender, program and status (high-achieving, average or low-achieving students).

Periodic reporting should help the school board to:
• Understand the relationship between student needs and resource allocation by building and program, and likewise, that teacher effort is appropriately focused.

• Have access to the context variables that are important to support effective leadership.

• Support staff development that is responsive to locally derived evidence.

• Support changes in school calendars and school schedules to assure the use of time maximizes learning.

Periodic reports not only keep a board current regarding school and district performance, they also inform how the district is ranked compared with like districts or with national averages. NWEA has created a data warehouse of longitudinal growth data. Although data are currently used by NWEA staff researchers and partner research organizations, one day the data will be open to individual school districts. Stripped of all student identifiers, these data then can provide a robust pool for creating virtual comparison groups. Districts can compare the costs and effectiveness of programs, organizational structures and the use of time or structure for grouping of students. In addition, the longitudinal data offer a quick comparison with state and national averages of student growth and achievement.

Robust Analysis
While the board fosters the culture and focus in a district, superintendents must manage both district policy and practice so that time, talent and resources are used to maximize learning. Few superintendents would argue with this mandate, and many are pouring their efforts into creating a model that will foster improved learning for all children.

Although a school district may recently have purchased a formative testing program, the data received through the administration of the tests may provide an insufficient measure of growth. Unfortunately, without a reliable growth measure in place, superintendents are ill-equipped to know what works and what needs to be changed. Which curricula are effecting the greatest growth? What are the best assignments for teachers, given their effectiveness lifting achievement in particular student subgroups?

All aspects of school or districtwide functions can have an impact on how much growth can be produced in students. In districts where outdated measures are in place, there is not quantifiable evidence to suggest what changes to make throughout the district.

Superintendents then become active champions for a culture that is organized around ways to optimize student growth. They own the mission of accelerating academic growth of every low-performing student and challenging every child with growth-provoking learning experiences.

For example, Linda Clark, superintendent of the Meridian, Idaho, School District, provides school principals and staff the license and encouragement to make structural changes to effect improvement in student learning. With this encouragement, educators can embrace change more eagerly, based on the evidence revealed in growth data from students. Clark cites several schools that have been successful in overcoming major odds to attain rapid and sustained improvement.

As the superintendent of Mesa County Public Schools in Grand Junction, Colo., J. Tim Mills created a culture focused on growth. Most evident is the manner in which school staff now engage parents in conversation about growth, resetting expectations about success in the school and how the district can measure progress.

Changing the conversation about education in the community, stimulated by the data available on how students are growing and what instruction, resources and staff are creating greatest benefit, similarly leads to systemic change focused on student growth in the schools.

Gerald Hill, superintendent in Glenview, Ill., joins his administrative team in systematically reviewing districtwide data on student growth. They encourage principals to review growth data with classroom teachers and other professional staff at the site level. By following up on their commitment to create growth in student learning, district staff are changing the culture through their communication and making sure everyone stays in alignment with district aspirations for raising student learning.

In districts where this model has been successfully implemented, superintendents and their executive teams use data from assessments to:

• See that all students receive instruction that is challenging.

• Allocate resources by program or school based on student needs.

• Assign teachers so that appropriate talent is assigned to each school based on the student needs of the school or program.

• Provide for staff development that is responsive to the evidence within each school setting.

• Meet with each school principal at least three times a year to examine student data and local efforts to improve.

The ability of superintendents to be effective in their role is dependent on timely and accurate reporting, robust analytics and student growth data over time. In turn, principals need the context variables that are important for leadership. This includes the necessary flexibility and support to respond to the rapidly changing needs of students throughout the school year.

Technology Platform
Certainly it is the quality of the growth measure as well as the timeliness of both the assessments and reporting that allow administrators to implement systemic change. However, the integral role of technology cannot be overlooked. Only through the advances in computer and software technology over the past few years have school districts been able to realize the potential for using growth data to create meaningful change.

Most testing companies are offering computer-based or online assessments, but many lack capacity to measure growth. Others are offering assessments that are computer-based adaptive tests, which present test items to students adjusted for difficulty depending on how students have responded to previous questions. Likewise, current technology returns test results immediately to teachers and administrators.

As a best practice, administrators need desktop access to online information and the ability to query data to see program effects in producing growth. Online query tools allow administrators to disaggregate the growth data at any level. With this insight, administrators can evaluate the effectiveness of school programs, policies and functions and resource allocation.

Administrators also can compare and analyze district, school and individual growth targets, spot district or school-level patterns and trends and quickly drill down through groups to pinpoint student data.

Newer technology also is allowing organizations like NWEA to import additional data into its longitudinal data set, to access all known district data, regarding attendance, schedules, personnel and finance. It is also possible to import non-traditional data sets, such as U.S. government census data, to more fully inform decision making.

Future Measures
Educators continue to have a healthy conversation about student achievement, school performance and the nation’s ability to serve the educational needs of all children. Clearly school districts want to produce an environment where all children are appropriately challenged, able to realize growth in learning year after year and can achieve state-mandated benchmarks.

Benchmark tests, locally constructed formative tests, conventional standardized tests and state-mandated assessments only demonstrate where students are performing at one point in time. Without adequate information about the activities, programs, curricula and educational structure that produce the greatest growth, it will be difficult for administrators to effect and sustain growth in student achievement, that is to create the systemic changes that will produce long-lasting benefits to students and their learning environments.

Allan Olson is president of the Northwest Evaluation Association, 5885 S.W. Meadows Road, Suite 200, Lake Oswego, OR 97035. E-mail: Allan.Olson@NWEA.org