June 22, 2021
Evaluating EdTech: A Strategy for Selecting Digital Tools
Guidance from Common Sense Education for district-level decision making
BY JESSICA LINDL/School Administrator, March 2017
Navigating through EdTech options can feel like a tightrope walk for school administrators hoping to balance the desire for innovation with the need to make fiscally sound choices. Fortunately, it does not need to be a binary choice. Over many conversations I’ve had in recent months with digitally savvy educational leaders, I’ve learned that it’s possible to do both.
I’d like to share with you what I’ve learned.
Where to Start?
Most educators I’ve spoken with agree that defining your need is a good way to begin.
“Let’s say you want kids to learn more math,” says Steven Hodas, former executive director of the New York City Department of Education’s Office of Innovation and now practitioner in residence at the Center on Reinventing Public Education at the University of Washington. “An administrator might see the problem as kids doing poorly in math. But a teacher might say that the problem is they’re unable to give students sufficient individual attention. Both point to kids learning more math, but they define the nature of their objectives differently.”
Hodas suggests administrators engage people who work with the problem on a daily basis to help refine the need before jumping to possible solutions. That also includes parents, school staff, classroom aides and even the community at large. Once the need is better defined, it becomes easier to narrow down the universe of digital tools to the ones that address your need.
Next, Barton Dassinger, principal of the Cesár Chavez Multicultural Academic Center in Chicago, likes to proceed with a “research question” — turning the need into a testable hypothesis. This forces people to think of ways to quantify their objectives. “An example of a research question is: ‘Do students using product X show higher gains in Y than similar students not using the product?’” where Y is the metric used to gauge progress, whether that’s behavioral referrals, attendance, college admissions or test scores, Dassinger explains.
“Before you make a purchase, you can work with the company to provide the data that will help you answer your research question,” adds Dassinger, who recommends pressing vendors for weekly data reports. “That way, you can evaluate their claims on the metrics that matter to you.”
With tens of thousands of choices, finding the right tool can seem like searching for a needle in a haystack. Many educators have devised their own systems for filtering in the most relevant tools while remaining open to changing possibilities.
The Omaha Public Schools in Nebraska developed an app approval process that allows any educator in the school district to submit recommendations for digital products throughout the year. Since the process launched last fall, Omaha’s teachers have submitted more than 200 apps. Most receive a decision within five to seven days. If the app is rejected, the teacher receives feedback on why the product didn’t pass muster.
“Before this system, it was a manual process that would take months” to process a recommendation, says Rob Dickson, Omaha’s executive director for information management services.
In Chicago, Dassinger prefers to interact directly with teachers within the classroom environment and have informal conversations to surface new products, keeping a list of prospects on a spreadsheet.
|Jessica Lindl, general manager of Common Sense Education, during a classroom visit.|
These examples suggest there’s room for a wide range of approaches. But the most effective systems tend to have several characteristics in common — they are ongoing, transparent and adaptable.
Ongoing. Technological advancements occur nonstop, and teachers’ needs don’t always coincide with adoption cycles. The best systems continually evaluate what’s new in the market.
Transparent. Danny Wagner, manager of Common Sense Education’s STEM Education Content and a former science teacher in Boone County, Ky., believes teachers need feedback so they can better understand why their submissions did or didn’t make the grade. This is critical. Feedback shows teachers their voices are valued, and it will help them stay involved, providing a fresh flow of ideas.
Adaptable. Dickson calls his system in Omaha “a workflow for change” because he can update it with new curriculum requirements and learning standards. “If we create a sound system for evaluating apps, it shouldn’t matter what new technologies are thrown at us,” he says. “We can still do what’s best for our kids.”
If your district’s discovery process involves issuing a request for proposals, Hodas recommends focusing on the learning goal, not the solution. “If your RFP contains a list of specifications, people try to check the boxes and tell you what you want to hear. That’s not likely to be productive,” he advises. “You need to reach out in a way that gives people the freedom to give you creative solutions.”
Let’s assume your discovery phase leads to a flood of options. An efficient way to narrow the field, Hodas says, is to ask teachers to watch a five-minute demo of each product (most tech products have video demos posted on their websites). After viewing, ask the teachers to answer this question: Would you use it in your classroom? If the answer is no (or mostly no), toss it out of consideration.
Once you have a manageable list of candidates, you can put them through a more rigorous evaluation. In Omaha, reviewers use a wide range of criteria, from compliance with the federal Children’s Online Privacy Protection Act and best practices to pedagogy and alignment to P21’s Framework for 21st Century Learning.
In the Charlotte-Mecklenburg School District in North Carolina, Valerie Truesdale, the chief technology officer, says she first determines whether the products are interactive and adaptive. “Students have to be interacting with the software and not just be given digitized worksheets,” she says. “And the software has to adjust its pace to individual progress so every student can have a personalized learning experience.” Products then are evaluated using North Carolina’s Standard Course of Study.
At Common Sense Education, where we’ve reviewed more than 2,500 educational technologies, our rubric includes engagement (how compelling the product is), pedagogy (how well it supports deep learning) and support (how accessible the experience is to students and teachers).
Recently, we added a fourth pillar: privacy. Over the past two years, we’ve worked with more than 100 school districts to create two free resources for schools: (1) a program that evaluates educational software products on their safety, security and privacy policies, along with how well they comply with government regulations; and (2) an open source toolkit for districts that want to do their own evaluations. By the end of the school year, our director of privacy review, Bill Fitzgerald, plans to have completed evaluations for 300 educational technology products on our site https://privacy.commonsense.org/.
Finally, once you’ve selected a technology to try out, start small and iterate.
“Think of the first implementation as a pilot,” Hodas advises. “Don’t think of pilots as free software to try out. Think of the vendor as someone who is collaborating with you to create a product that will result in the changes you both want to see.”
This can be a cultural leap for school districts used to being given fully developed materials from established publishers. With technology companies, particularly the younger and more innovative startups, products are in constant evolution, and their developers are continually adding features, fixing bugs and improving the overall experience. While it can be frustrating to deal with a moving target, schools can take advantage of that flux by jumping in to influence which features get added and how the product is improved.
At the end of the day, I am reminded of some sage advice for maintaining perspective, provided by Truesdale, a veteran school leader in the Charlotte-Mecklenburg Schools. “The key is to remember that these are all digital tools,” she says. “Just like building a house, you have many tools in your toolbox. What tools we use depends on the job we want to accomplish. But there’s no substitute for quality teaching.”
Jessica Lindl is vice president and general manager of Common Sense Education in San Francisco, Calif. E-mail: email@example.com Twitter: @CommonSenseEd
Common Sense Education, a nonprofit organization, provides educators with free, research-based classroom tools to help schools harness technology for learning. The website www.commonsense.org/education includes a primer for evaluating educational software with particular attention to child privacy, teaching strategies and practical ideas on digital citizenship.
Also, Common Sense Education suggests these information sites:
» ConnectEd, www.whitehouse.gov/issues/education/k-12/connected
» Digital Promise, http://digitalpromise.org
» EdSearch/Lumos Learning, www.lumoslearning.com/llwp/edsearch.html
» EdSurge, www.edsurge.com/product-reviews
» Federal Trade Commission’s explanation of the Children’s Online Privacy Protection Act for educators, www.ftc.gov/tips-advice/business-center/privacy-and-security/children%27s-privacy
» LEAP Innovations, www.leapinnovations.org
» Learning Innovation Hub, http://svefoundation.org/ihub
» P21’s Framework for 21st Century Learning, www.p21.org/our-work/p21-framework
» Technology for Education Consortium, http://techedconsortium.org
» U.S. Department of Education, Office of Educational Technology, http://tech.ed.gov
Questions to Cut Through the Marketing Spiel
Whether you’re putting out a formal request for proposal or you just want a conversation with the vendor, here are some questions to help educators move past the marketing script and on to how the proprietor’s products can improve student learning. I’ve collected some of these from Danny Wagner, head of Common Sense Education’s STEM Education Content, to spark your thinking.
Customization: Can the company customize the product to fit the needs of a wide range of schools and populations in your district? Can teachers create custom content? How well does the tool integrate with other tools teachers are using in your classrooms?
Connectedness: How does the technology help connect stakeholders, including parents, students, teachers, administrators and community members? The stronger these connections, the more students will take ownership of their learning, teachers will become engaged facilitators, parents will become decision makers and empower learning at home, and businesses will give their time, sponsorship and donations to educational initiatives.
Equity: How can a variety of student populations access the product? Is it available in multiple languages? Can resources be printed or read out loud? Are there options for library access or home access?
Technology Requirements: What are the platform and systems requirements? Does it work with existing school devices? Does it require the school to upgrade its infrastructure? What support does the company offer schools to help integrate the product? Can the product or service scale to support thousands of students?
These will surely prompt a dozen other questions for school district leaders to ask. If so, you are well on your way to being an expert educational technology evaluator.