Tech Leadership

Unconventional Wisdom About Buying Technology

by Michael F. Sullivan

Conventional wisdom says you shouldn’t buy anything in education until you see the research. You’re supposed to ask these questions: Does that particular technology enhance learning? Does that piece of software increase test scores? Do those machines reduce absenteeism?

Of course the answer is always yes. No vendor is going to provide research that says a product doesn’t work. Research is considered just another price of doing business. But, you say, this must be a good thing because even federal regulations require that some products be research based.

Well, here is how the research field of product effectiveness works. First the vendor hires a professor or consulting firm to test the product. Then the consultants go to friends or to schools that will agree to test the product under prescribed conditions. The professor or consultant must be able to deliver the research sites, which may be the toughest part of the job. Most school districts dislike any disruption to the normal process, and most teachers resist disruptions to their routines. Blackmail and bribery work best when gaining the participation of educators.

Once the research sites are identified, the process is fairly simple. A few questions asked, a T test or two and, voilá, the results are evident. An upbeat report is written, bound and submitted to the contracting company. And guess how many more contracts show up if the research report says the product doesn’t work? Thus every product is shown to do what it purports to do.

Why then is it so difficult to get the results you paid for? Why isn’t every student breezing through those tests and offering to stay after school for more work? In your setting the product wasn’t used in the way in which it was designed, or your students represent a different population, or you didn’t train adequately, or you didn’t have adequate support, or you bought it during a month that didn’t have an “r” in it. You get the idea: It’s your fault. When was the last time you saw a company report that the research it published was flawed and the product didn’t deliver the results forecasted?

Trustworthy Research

Let’s be honest. The research on many products and services is worthless. If you don’t like the studies provided, you just need to do a little research of your own to find an entirely contrary position. You could decide to buy nothing because you can’t find that defining study. Few researchers do studies for fun. Most have a vested interest or an ax to grind. Plus, educational research is infamous for its lack of reliability. Remember how Harvard University President James Conant showed us that large high schools are better? And how James Coleman, the sociologist at the University of Chicago, showed that money didn’t matter to schools? Or how about the whole-language approach to learning and that body of research? (We need not bother to discuss my dissertation on the promise of the open classroom.)


You may wonder why education research is so often poor. The answer is that educators aren’t great consumers of research. Many don’t really understand the principles of research design and aren’t likely to question assumptions and procedures. Of course, they also know the free market drives competitors to attack each other’s research and there is no education equivalent of the Food and Drug Administration to oversee product evaluation to make sure consumers are protected. Therefore, they tend to pretty much ignore all research unless it supports something they already believe. And much of the research is done by people with no real background in education, leading them to miss some pretty important stuff.

So if one can’t or won’t trust the research, how do you make a purchase decision? Just as with buying a car, you research the company. If it hasn’t been around for at least a few years, odds are it may not be around for many more. Of course even established businesses have been disappearing in this economic climate so even a lengthy past isn’t a guarantee.

Check with friends and colleagues. Does the company stand behind its products? Test the product yourself. If the company believes in its products, it’ll give you a trial run. Check out the competition. Is this product really worth more than a competitor’s product that costs $20,000 less? Don’t hesitate to haggle. If you pay list price for an expensive piece of hardware or software, you probably paid too much.

Paying for Results

Please don’t assume that you shouldn’t buy anything. You need a certain amount of hardware and software to operate a meaningful instructional enterprise of any size. You don’t have to decide between believing the claims or throwing the vendor out. Just don’t accept the claims as proof that the product will work in your setting.


The real test is in your schools with your students—and you want and should be able to expect results. Here’s a novel idea: Offer to pay the company $100 for every one of your students who tests out of calculus or $10,000 if every 5th-grader passes the high-stakes test. You’ve identified the issues that are important to you, and you’re certainly willing to pay for results.

Just make sure everyone agrees on the desired results. If you ask to have your students prepared for the real world, they may just end up drinking bad coffee in a cubicle while waiting for tech support to get the computer working. That’s the real world, never mind the conventional wisdom.

Michael Sullivan, former executive director of the Agency for Instructional Technology, is
education director of Take Charge America, P.O. Box 83330, Phoenix AZ 85071. E-mail: