Districts Pilot Value-Added Assessment

Warwick School District, a 4,100-student district located in southeastern Pennsylvania, tested its students even in years when state-mandated exams were not required.

But Iovino didn’t know exactly what to make of all this test data his district was accumulating.

"We weren’t doing much with longitudinal data," he says. In other words, the Warwick schools weren’t tracking individual student achievement over time, but rather looking at how a particular grade scored on a test year after year—a snapshot assessment.

Iovino knew there were other ways of looking at his district’s yearly test data. "A way that would predict how well a student would perform [in the future] and what kind of value was added to a student’s achievement in a given year," he says.

He was right. Pennsylvania launched in 2002 the Pennsylvania Value-Added Assessment System with Warwick as one of the 32 pilot districts. The system is based on the value-added assessment model statistician William Sanders created in Tennessee in the 1980s.

As Iovino explains, value-added assessment carries two basic goals: to assess how much growth a student achieved from the beginning of the school year to the end and to predict a student’s future growth. Value-added requires, though, a minimum of three consecutive years’ worth of test data for an accurate assessment. When Sanders created the model few school districts in the country conducted annual testing.

But with the No Child Left Behind Act’s yearly testing requirements, every school district in the country soon will have enough test data to use value-added assessment. Sanders and others hope they do.

Teacher Usage

"It took us a long time in education to realize we could do more with test scores than simply slice and average them," says June Rivers, who co-directs with Sanders, her husband, the Value-Added Assessment and Research Program at the SAS Institute in Cary, N.C. Rivers is working with officials in Pennsylvania to implement that state’s value-added assessment system known as PVAAS.

In 2002 the state added a value-added component to its student assessment process and launched a pilot project with 32 districts that had been using yearly tests in grades three through eight. Last year, 30 more districts were added to the pilot and this year an additional 50 to 60 districts are participating, according to Kristin Lewald, a project manager for PVAAS, which is overseen by the state’s department of education.

It’s her job to work with superintendents such as Iovino to smooth the transition.

"What I tell them is that it’s taking information they already have and using it in a different way. We use data they already have, without using another test," she says.

In Warwick, Iovino began the value-added assessment project by looking at his middle school students. "The plan was to take a look at total scores as well as student grouping," he says.

Iovino’s teachers have used the value-added measures that allow a teacher to know in more detail where individual students stand academically and to regroup students before beginning a new curriculum topic. Iovino cites a social studies unit on feudalism as an example. "Let’s say we have a group of 7th-grade students who aren’t reading at grade level. That group receives a pre-teaching lesson before getting into the study of feudalism."

Warwick this year will begin value-added assessment at its four elementary schools. "I want to create a climate where this is something worth looking at and where we are developing future action for improvement," he says.

A Full Spectrum

An important aspect of value-added, according to both Lewald and Sanders, is that the information allows school officials to examine the progress of all students, not merely the students who are struggling. States have to go beyond the minimal NCLB requirements to ensure all students are making progress every year, Sanders says.

"What about kids from the other end of the spectrum? Kids not in jeopardy of falling behind?" Sanders asks rhetorically. "The next question is, ‘Can we sustain that kid on [an upward] trajectory to get that kid in a position to go to a university and be successful?’ We want sustained growth for all kids, not just the low-end students."

The Oak Hills School District in suburban Cincinnati is well aware of Sanders’ views that challenge most state accountability tests that only show when kids are proficient.

"One of the things we’re able to do with value-added is highlight areas where we’re doing things really well," says Jay Kemen, Oak Hills’ director of curriculum and instruction. The information is personalized and detailed at the student level, the grade level and the building level.

"Where we have very high performance based on growth, we can go in and say, ‘What are you doing well?’ It works in districts and schools that have developed a culture of learning," he says. "I think in part what this data and other sources give us is a broader picture."

In Oak Hills, Kemen says, the district might go into one building to look at the test scores and find that nine out of 10 students are doing well, but one isn’t. "Is that student a lower- functioning student or is it a higher-functioning student? Are we not stretching them as we should?" he wonders.

Major Scaling

Like Pennsylvania, Ohio has been rolling out a value-added assessment system known as Project SOAR (Schools’ On-line Achievement Reports) in pilot districts. The project is run by Battelle for Kids, a Columbus-based organization created in 2001 by the Ohio Business Roundtable. Jim Mahoney, a former superintendent, is Battelle’s executive director. (See related story, page 16.)

Mahoney says the challenge at hand is to scale up from the current 78 participating districts to all 718 Ohio public school districts, a goal he expects to meet by August 2008.

At Westerly Elementary School, a school near Cleveland that houses only 3rd and 4th grades, Principal Sylvia Cooper says the annual state exam administered to her students was not especially valuable because each year it compared a different cohort of students.

"You’re not comparing apples to apples," she says. "You don’t know how well your program is affecting individual students."

The idea of value-added is appealing, she says, and her school has joined the pilot program. The previous assessment system, she says, looked at two different groups of students. Now, though, value-added will allow her to see how individual student perform over time. "We get a more specific picture of individual student progress and academic areas that need to be honed better," she says, adding that she and her staff have had little time to analyze the data enough to make curriculum adjustments.

Cooper is required to import her school’s test data into a software program that will conduct the value-added measures. During the start-of-school crunch, the test data was put on the back burner.

The time crunch is one of the hurdles value-added advocates must clear to convince school leaders across the country on the benefits of the system. Another hurdle is convincing teachers.

Evaluating Teachers

Bill Sanders has been talking about a new way to analyze school test data for more than 20 years. He first started looking at test scores in the mid-1980s as a researcher at the University of Tennessee at Knoxville. Then-Governor Lamar Alexander was looking for a way to evaluate teachers and Sanders thought he could find a solution.

Sanders, instead of trying to control for differences in a student’s background, such as family income and parents’ education levels, decided each student should act as his or her own control. In other words, the focus should be on gains rather than raw test scores so that each student’s performance is measured not against that of similar students but against his or her own past performance.

The overall idea is that even if all students don’t achieve at the same levels, schools and teachers should at least be adding value to each student’s performance.

But Sanders’ early work mostly fell on deaf ears. It wasn’t until 1992 that state policymakers adopted his novel assessment notion. A key yet controversial aspect allows an examination of not just school-level performance but an individual teacher’s performance, based on the gains made by his or her students.

Using Sanders’ system it was easy for school officials in Tennessee to see which teachers were the biggest gainers and the state included the value-added measure in teacher evaluations. In Chattanooga, high-gaining teachers are rewarded with bonus pay and other districts throughout the state are considering following suit. But outside of Tennessee, the notion of using a student’s test score gains to evaluate teachers is not exactly popular.

"Our teacher unions will never go for this," says David Plank, co-director of the Education Policy Center at Michigan State University. "Any move will be blocked."

Plank says Michigan, like every state, soon will have annual test data thanks to No Child Left Behind, making it possible to track classroom-level performance. "Technically it’s feasible, politically it’s not," he says.

Identifying Excellence

For states that have adopted a value-added assessment system, the way around the sticky politics of value-added has been to, thus far, pledge to steer clear of using it to evaluate teachers.

"PVAAS was not intended for teacher accountability," says Pennsylvania’s Lewald.

Patricia Brenneman, Oak Hills superintendent, says neither is Project SOAR in Ohio. "We don’t want our teachers being threatened by someone else’s best practice," she says. If the data identifies a teacher who has increased student learning significantly, then "let’s have that teacher do staff development with other teachers," says Brenneman.

Rather than fear value-added assessment, teachers in the Oak Hills district have embraced it, the superintendent says. "Data don’t make change in the classroom, people make change."

Using the teachers who show the highest gains as best-practice tutors for other district teachers is exactly what officials in the Hamilton County, Tenn., schools, which include Chattanooga, have been doing for the past several years, says Dan Challener, president of Chattanooga’s Public Education Foundation.

"Through precise research we’ve been able to identify about 75 teachers who consistently posted well above national and local average gains," he says. "And what we then created is a research network with those teachers."

The district also has created a set of classroom videos featuring the high-gaining teachers so that if other teachers want to study the classroom practices of those teachers, they can simply refer to the videotapes, Challener explains.

Sanders sees the teacher evaluation component inexorably tied to the value-added measure. "Once you make the investment of testing each kid, each year, then you have the opportunity to make these measurements at various levels," he says. "Now, I’m a strong advocate that districts should evolve to get [value-added] measurement at the classroom level," but he concedes he is aware of the reality and knows Ohio and Pennsylvania educators will not be using their value-added systems for teacher evaluation, at least not in the near future.

Last Hurdle?

So if states have removed the teacher evaluation component from value-added assessment, then what’s holding education officials from fully embracing Sanders’ model?

In Michigan, researcher and policy analyst David Plank is about to release a paper that examines potential pitfalls of value-added assessment.

For instance, he says, state tests used in value-added calculations typically produce an overall reading or language arts score that may be derived from several different skills, such as spelling and grammar. So if a 3rd-grade student scores a 30 on the spelling portion and a 90 on the grammar portion of that test, his or her overall score would be 60. If the next year that same student, scores a 90 on the spelling portion of the test and a 30 on the grammar portion his or her overall score would again be 60, showing no gain.

"I have been and continue to be a strong advocate for value added," says Plank. "Technically, though, it’s even more difficult than any of us had realized in our early enthusiasms."

Plank believes one solution would be to increase the frequency of testing, an idea he realizes would prove costly and probably unpopular. Some school districts, though, already have taken precaution against potentially misleading value-added results by using a combination of tests.

Gerrita Postlewait, superintendent of the Horry County, S.C., School District, has been working with Sanders for several years. "He created a database using state- and national-normed test and Northwest Evaluation Association tests (see related story) tests to help create a broad picture of value-added measures," she says.

The approach provides a depth of information as well, she says. Her 47-school district now can more accurately assess everything from student achievement to certain districtwide programs. "It’s taken the guesswork out," she says.

Sanders is aware that his system is not perfect and concedes that if, like in Plank’s example, the student’s performance results on the spelling and grammar portions of a reading test are not separated out, no way exists to accurately measure student gains. "We can say that Mrs. Jones’s students across the scale measured in the test made progress greater than the district average. Can we say more than that? Of course not because we don’t have the data."

Still, Sanders and his SAS colleagues continue to refine the system, and Sanders believes the strength of value-added is what it reveals over time. As schools become increasingly data-rich, and collect year after year of test data, flaws in value-added will be greatly reduced, he says.

The key to value-added’s future is having educators not only accept it as an assessment model but make the best use of the information it provides, he says. "You can do the best analysis in the world but if you don’t have people trained and coached to use the information, not much is going to happen."

Getting to that point, though, requires a final leap of faith from the education community. "This is a paradigm shift in the way most educators think," Sanders says.

Brett Schaeffer is a free-lance education writer based in Philadelphia. E-mail: brett@brettschaeffer.com