Survey Says: Data To Guide Policy Decisions

Superintendents and boards can find formal research — rather than anecdotes and impressions — an advantage on key issues by Edgar H. Thompson, George H. Stainback and James G. Stovall

The 3,600-student school district of Albertville, Ala., faced a classic dilemma in 2003. The district’s sole high school was showing its decades of wear and tear and was badly in need of renovation. Some citizens wanted to abandon the building and build a new high school on the outskirts of town, while others were adamant about preserving the existing building for economic and historic reasons.

Regardless of the final decision, the superintendent and the school board would have to ask the taxpayers to come up with more money. To help them make the decision, the board and superintendent hired a survey research company to conduct a citywide survey of taxpayers’ opinions. The survey professionals helped the district formulate a balanced questionnaire, conducted a telephone survey, analyzed the results and presented the board with a written report that was made public soon after.

The survey results indicated the public was aware of the problem with the dilapidated high school. Nearly two-thirds of the respondents classified the problem as “very serious” or “serious.” The survey also revealed that more than 60 percent favored keeping the high school at its present location while nearly 30 percent wanted a new building in a new location. Support for a new location increased about 10 points when respondents were told the old building could be renovated and put to some civic use.

On the key question of a tax increase, 44 percent of the respondents said they would vote for a tax increase if they were assured it would be used to construct a new building in a new location. About 50 percent said they would support a tax increase if the board were simply renovating the old building.

Although the survey results were not definitive, the research gave the superintendent and school board members some solid information on which to base a decision. Faced with committing millions of dollars to a construction project, the board felt justified in spending a few thousand dollars to get accurate information from an independent source.

Mining Data
Today’s superintendent makes key decisions about the school system that affect not only the direction of the system, but also the lives of the people associated with it. Those decisions include making sure people, money, resources and assets are distributed in ways that most benefit the students.

When competing interests attempt to dictate school policy, the superintendent and the school board must serve as the arbitrators. Special interest groups are becoming more prevalent and powerful in school governance and have developed relatively sophisticated strategies to promote what they consider the most appropriate policies and programs for the schools.

These groups, which include people both inside and outside the system, are generally responsible stakeholders in the schools and have the students’ best interests at heart. They are able to support their position with information that is valuable but whose value is tempered by its source. School superintendents may rightly question the validity of this information, but they often do not have the resources to counter it if they believe it is leading the system in the wrong direction.

For example, a teachers organization may claim that teacher morale is low in the district or a taxpayers association may claim that citizens generally are against new taxes or tax increases for schools. Too often such assertions are made without true substantiation. The superintendent should ask, “How has this information been obtained?” “What questions were asked?” “How was this information processed that led to the conclusion?”

When important decisions and millions of dollars are at stake, school boards and superintendents must have accurate, reliable and unbiased information on which to develop policy and programs. Districts often can gather such information through scientific surveys conducted by companies that have little or no connection with the school system, as the Albertville district did.

Survey research has become increasingly sophisticated, accurate and affordable. A good research firm with a good track record of working with school systems can bring its experience to bear on the problems the school system is facing and provide efficient, cost-effective research with quick results.

Areas in which survey research might be useful range from the mundane (when to run buses) to the mighty (tax increases). In addition, survey research can:

• provide information from wide-ranging constituencies;

• give superintendents and boards independent, unbiased information to evaluate the effectiveness of policies and programs;

• provide guidance for identifying issues and formulating solutions; and

• allow for strategic planning in building support for policies and proposals.

The public relations aspect of conducting a survey is also significant. Good survey research gives constituencies a voice in the decision-making process and sends the message that the superintendent and the board are interested in what the public has to say.

Solid Research
Most people who go through graduate degree programs in the social sciences or related fields know the basics of good research: Ask the right questions of a good sampling of the defined population; take into account confounding factors; and analyze data in a dispassionate fashion. These principles are easy to state; they are much harder to put into practice.

How do you construct a question that will gather the information needed? How do you define a population and then draw a sample from that population? Should a survey be conducted via telephone, mail, personal interview or the web? How can the results be coded and analyzed?

These are not easy questions to answer, especially for those who are inexperienced in survey research. That is one of the reasons superintendents should not hire someone within their own school system to conduct a survey. Even if that person understands the basic principles of surveying, he or she may lack the resources to produce adequate and reliable data that interest groups cannot challenge as biased. In-house surveys ultimately may cost a school system more than if a professional firm was hired to conduct the survey.

A high-quality research survey firm can provide a complete range of services — creating the questionnaire, sampling, interviewing, analyzing the data, and writing and presenting the report. These professionals also provide independent voices that are more credible when dealing with controversial issues.

Participatory Role
Hiring a reputable firm to do the research does not mean bowing out of the process entirely. The superintendent and school board should actively participate in the initial stages of the development of the survey and should ask questions throughout the process. In particular, superintendents and board members should pay attention to the following areas:

• Wording. Experts in survey research will tell you that the single most important factor in determining the results you get is the way the questions are worded. Slight variations in wording can yield big differences in the results. Survey questions and possible responses should be understandable, unambiguous, unbiased and complete.

Questions should focus on areas that are within the realm of experience of the respondents. For instance, surveys administered to the public should be free of educational jargon. A question such as “Do you favor teaching phonics rather than whole language?” might be appropriate for a survey of teachers, but it would not be appropriate for a general population survey.

• Sampling. Good sampling is a big part of ensuring the accuracy and reliability of the results of the survey. Sampling involves a two-step process — defining a population and drawing a sample.

At the outset, superintendents and school boards should have a good idea of the population (or populations) from which they want information. Is this a survey of parents of school children, teachers within the system or the general public? A survey of all adults in a county is relatively easy because the surveyors can reach this population at home by telephone. A survey of high school students, however, may be more difficult and cost more time and money.

The advantage of systematic sampling is that you can make sure that a population is defined and that each person in that population has one and only one chance of being selected for the sample. Also, the results of a survey can be generalized to an entire population with a known chance of error. A good sample will collect information from all segments of the population, not just from those who are most committed or extreme in their opinions.

• Data collection. Survey research traditionally draws from three methods of data collection — personal interview, mail survey and telephone survey. A fourth method, the web survey, is available, but to date seems to work only with specialized populations.

The personal interview survey is both expensive and time-consuming, and few school systems have the resources to invest in this kind of survey. Mailing a questionnaire to respondents and letting them fill it out and send it back is a more attractive strategy because it seems relatively inexpensive. However, mail surveys often elicit very low response rates and thus produce potentially biased results. A mail survey can achieve a good response rate, but that requires time for follow up, which can increase costs. The fact that the researcher has no control over who actually fills out the questionnaire can skew results.

Telephone surveys are the most efficient and cost-effective surveys. The interviews themselves are conducted by trained professionals. These professionals know how to select the appropriate respondent within a household and can ask questions in a neutral manner. Responses are fed directly into computer systems that quickly provide data for analysis.

• Data analysis and reporting. Data analysis and written reports should be part of the package you purchase from a professional survey research firm. Superintendents and school board members should make clear their expectations about how they want the data to be analyzed and what information they want in a final report.

This clarity about expectations is important because even a short, simple survey can produce a large amount of data. It is the professional researcher’s responsibility to handle these data and present the results in a form that those who commissioned or who have an interest in the survey can easily understand and use.

Reports should be simple, straightforward and unbiased, even if the survey produced what the superintendent or board consider “bad news.” Again, this is where the independent voice of a professional researcher, one who does not have a stake in the issues of the survey, can benefit the school system.

Time and Money
District officials should seek survey companies that understand that efficiency and timeliness are qualities of good research. A survey company that wants two months to conduct simple surveys probably is not working efficiently. Not only does such a timeframe compromise the timeliness of the data, the additional time translates into a higher cost for the district.

Several factors can play into calculating the cost of a survey, but two of the most significant are the length of the questionnaire and the size of the sample. The length of the questionnaire affects the time necessary to develop the questionnaire, administer it and collect and analyze the data. Believe it or not, a survey that takes 12 minutes to conduct can cost significantly more than a survey that takes 10 minutes to conduct.

There is no single answer to the question, “How many people should we include?” That depends on what information the district wants from the survey and the quality of the data it desires. A larger sample often means a higher quality of data, but there is a point at which the amount of money required to survey a large sample is not worth the investment. A professional survey researcher can work with you to help you spend your money most effectively.

One way survey companies price projects is by determining what an individual interview will cost and then letting the client choose the size of the sample. For instance, a single interview in a general population survey may cost $20. A survey that needs 300 respondents then would cost about $6,000. Knowing that, a superintendent might consider paying for a higher quality of data and ask for a survey of 400 respondents at a cost of $8,000.

Whatever the cost for the survey, school superintendents and school boards should understand exactly what they are buying when they contract for survey research. They should be clear about timetables and deadlines, the procedures the researchers will use, what the survey will cost and how the data will be stored and accessed.

Applying Results
Contracting for and conducting a survey are just the starting points, of course. The ultimate goal is to be able to use the survey results to guide decision making. The responsibility for using the results falls to the superintendent and the board, not the researchers.

Before contracting for a survey, superintendents should think carefully about how they will use results to implement a plan. Survey results do not speak for themselves. Superintendents and board members should ask for a variety of interpretations of the data. Any finding must be set in a context. Determining the appropriate context must be a group effort.

Conducting a survey can be a positive public relations move for a school system if the results are shared far and wide. Results of a survey can spark discussion based on good information rather than on impressions and anecdotes.
When school systems include research as a permanent part of their budgets, they are able to conduct surveys regularly and benefit from the solid survey results.

The Road Map
In Albertville, Ala., the school board is still struggling after more than two years with the issue of improving its high school facility. The board formulated one proposal that the voters narrowly defeated in May. Now it is working on another.
So was the money it spent on the initial survey worth it?

“Absolutely,” says board member Allan Champion. Even though the results of the survey were not definitive on what the board should do, Champion says they “provided us a road map. They showed us what the possibilities were.”

Some board members had argued that the public would never accept tearing down the old school building. Champion said the survey results showed this was not the case and it gave the board the opportunity to seriously consider that option. Champion said he would definitely commission the survey again and he would favor doing another survey as the board refines its options.

The experience in Albertville demonstrates the complex task of operating a school system and making decisions about its future. Part of that task is responding effectively to the special-interest groups and to the taxpaying public. Survey research is an effective, cost-efficient means of obtaining the best information for this task and making the superintendent’s decisions just a bit easier.

Edgar Thompson is director of the William F. Neff Center for Teacher Education at Emory & Henry College, P.O. Box 947, Emory, VA 24327. E-mail hthompson@ehc.edu. George Stainback, a former school superintendent, is associate professor of education at Emory & Henry College. James Stovall is the Edward J. Meeman Distinguished Professor of Journalism at the University of Tennessee and a co-director of Southern Opinion Research.