Punchback: Answering Critics

Reasons for Wariness About ‘Evidence-Based’ Claims

by WILLIAM J. MATHIS

Torrents of school improvement ideas flood superintendents' in-boxes. "Total school reform" is promised, with glossy color brochures featuring the attractive, smiling faces of expensive professional speakers promising "evidence-based results."

Returning from professional meetings, administrators and teachers wave studies extolling the curative qualities of popular programs prefaced with the phrase "Research says ... ." Then, there are the special education panaceas promoted by advocacy groups and anxious parents.

William MathisWilliam J. Mathis


While superintendents are dealing with this cacophony, laws are being passed. Then mandates cascade down as faits accompli. All too often, these notions are advanced by vested interest groups or partisan think tanks. Whether coming from the political right, left or center, these studies are designed with the purpose of influencing policymakers.

And they have huge and under-recognized effects. Arguably, the greatest impact has been made by free-market-oriented think tanks such as the American Enterprise Institute, the Hoover Institution, the Cato Institute and the Friedman Foundation. For example, 40 states now have charter-school laws, even though the research evidence shows charter schools, on average, produce no higher test scores (and actually produce lower scores according to some well-designed studies) and result in further segregation in our already fragmented society.

Vested Interests
School administrators have a professional and moral responsibility to weigh in on proposed policies. Yet, all too often, this does not happen. One reason is that superintendents do not have the time to sit down and study a particular issue. Even when they are not being lobbied by board members, teachers, parent and citizen groups, finding a few minutes to read about the topic of the day inundates them with advocacy pieces, conflicting reports and studies cloaked in statistical exotica. The task becomes overwhelming.

Adding to the problem, vested interest groups have figured out that education policy issues can now be pre-decided by political action at the state and federal levels. There is no greater example than No Child Left Behind. Unfortunately, our recent review of the research behind the Obama administration's blueprint for reauthorizing the Elementary and Secondary Education Act finds the plan endorses faulty solutions such as high-stakes testing, school takeover models, test-based teacher evaluation and charter schools.

This means if superintendents want to truly lead, they have to refocus their efforts in the following ways:

•  Engage early and work at higher levels. Important policy decisions are no longer local. The solution is through state- and national-level organizations. However, these groups must act fast, with clarity and with courage. All too often, organizations get mushy in fear of offending some members, losing their seat at the table or being criticized. Rest assured, those whose agenda is contrary to yours are not slow, reticent, opaque or mushy.

•  Know the issues. One of the biggest problems is developing high-quality, on-time knowledge about the issues. In the cyber age, traditional texts and reports have given way to electronic publications, frequently sponsored by think tanks with important-sounding names. The question is whether a study is good research, partisan advocacy cloaked as research or something in between.

Things to consider:
Who sponsored the report? Is there an ideological bent to the organization? In many cases, the orientation and funding are plainly visible on the group's web page. Do they embrace bias such as market-oriented reform models?

Do their research articles have independent peer review? If they don't mention it on their website, they probably don't.

Check external and independent reviews of the work.The National Education Policy Center , conducts independent, peer-reviewed studies on major policy ideas. We also conduct the think tank review project, which evaluates the quality of the science behind these reports. Searching our website can provide you with an objective review by a national scholar. Other groups offer literature reviews, but always read with a discerning eye.

Be wary of appeals to authority. Just because the study was sponsored by a prestigious group such as Brookings, Broad, Gates or a prominent state group doesn't mean it is either good or bad science. Read critically.

Review curriculum improvement strategies. The scientific foundation for most commercial reform packages is shaky. A visit to the federal government's What Works Clearinghouse is advised. Like weight-loss schemes, beware of people selling potent plans and four-step programs guaranteed to boost your achievement.

•  Advocate.Many superintendents are reluctant to be policy advocates, even though it is essential that they be. Advocacy can take many forms, including supporting professional groups, being active in associations, writing op-ed columns, blogging, testifying to state legislative committees and state boards, publishing articles, and letting your elected representatives hear from you.

We are in uncertain times when the very notion of a democratic and equal education is being threatened, often with bad science. Instead of merely managing schools, we must lead in building a just and fair society.

William Mathis, a former superintendent, is managing director of the National Education Policy Center in Boulder, Colo. E-mail: william.mathis@colorado.edu