What to Look for in Online Learning Apps for K-12

May 9, 2020

The sudden need for distance learning imposed upon schools worldwide by the COVID-19 pandemic has created a huge market for online learning applications. Some have been around for decades while others seemed to have popped up overnight.

For teachers and school leaders who may have scant knowledge of how to assess which distance learning applications are most appropriate for their needs, GSE Professor Zachary Pardos offers his expert advice in the Q&A below.

GSE: Where do educators even begin when looking at online learning applications?

Professor Pardos: There are two main categories of education resources that teachers can seek out. It is also important to remember that these resources are also seeking out teachers so you have to be aware of their marketing, and ask good questions about what they’re offering.

One category can be described as an “open educational resource” (OER). The content is free, it’s open-licensed, and it is often contributed by other teachers. It’s crowd-sourced, if you will.

Open educational resources can be attractive in this moment because they are already created online content that can be used asynchronously. That is, activities like videos and problem sets can be selected and assigned to help compensate for temporary lack of live class sessions.

These resources are plentiful, but that also makes them difficult to sort through.

These are criteria that can help evaluate the the quality of an OER:

  • Do they use the standards you care about? OER websites organize content using a mix of in-house taxonomies and state standards. Check that the site uses a standard that makes it convenient for you to align resources to your course.
  • Are the standards aligned properly to the content? OERs often rely on the community to tag their content with standards without much oversight, leading to inconsistencies. If you see any obvious misalignments, it’s probably best to move on to another source.
  • Has the curating OER site provided a way to give feedback on contributions and a process for contributors to iteratively improve their content? Some OERs can appear outdated or low quality, often because the site does not provide a mechanism for feedback, such as a discussion board or comments section. If the OER website has these social feedback forums, it’s a good sign.
  • Can it be used with a learning management system (LMS)? If you use an LMS, you know how convenient it can be when assignments are given and graded within the LMS. Some OER assessments allow for this type of integration. You can spot these OERs by their use of the term “learning tools interoperability” or “LTI.” This makes connecting them to a gradebook possible, but will require technical know-how. Many school’s info tech professionals will be familiar with LTI.

Another category is “adaptive technology.” These apps provide formative assessment; are connected with some theory of learning; and are off-the-shelf tools to adapt curriculum depending upon individual student progress.

A provider of a high quality adaptive learning technology should be able to answer certain questions easily. Some don’t have an answer, and that’s when educators should be skeptical. It’s important to ask questions because “adaptive” is too often used as a marketing ploy and when you take a closer look, the learning application isn’t adaptive at all.

Below are a few important questions to ask of an adaptive technology:

  • What is the app teaching and what size chunk of curriculum is it meant to address? Just like with OERs, it’s important to ask what standards the app is aligned with. It is also important for teachers to know what part of their curriculum they want an adaptive app to address. Some are only designed to cover all topics in a course from beginning to end, while others allow teachers to assign topics in piecemeal fashion.
  • What aspect of the system is adaptive? The most common form of adaptivity is a personalized sequencing of problems that depends on periodic formative assessment performed by the app. If the provider doesn’t describe, concretely, what makes their software adaptive, it most likely isn’t.
  • What is the learning mechanism behind the technology? It is very reasonable for educators to ask why a technology provider believes their product produces learning and how that relates to being “adaptive.” Immediate feedback, hints, scaffolding, and a personalized differentiation of instruction are all common mechanisms used in adaptive systems which have been found to be effective in a wide variety of school settings. 
  • How is the adaptive system assessing progress? For example, does it use percent correct or a psychometric model? The technology should use an established assessment mechanism and be able to quickly tell you what that is.

GSE: What evidence is there, if any, showing that one of these models is better than another?

Professor Pardos: There has been research showing that the psychometric models can be improved in a scientific way and that these improvements have led to learning gains. It is not a deal-breaker if the system uses a simpler approach to assessment, but if they are incapable of clearly defining how the system assesses students, that’s a red flag.

GSE: What about the early grades, say K-3? Is it appropriate for kids to be learning on an electronic device?

Professor Pardos: That’s an important question and one being actively discussed by researchers in early childhood development. It’s also a personal question for teachers and parents to answer, themselves. I will say that, with regard to technology, learning from home using personal devices is creating a privacy grey zone with regard to data collection across K-16. Most companies are behaving responsibly, but it’s important to be vigilant and not let that zone expand too far.

GSE: In general, are single subject learning platforms better than those offering multiple subjects?

Professor Pardos: Multiple subject platforms, like OERs and online course platforms, offer a consistency of experience and quality across subjects, but have relatively weak pedagogy compared to single subject platforms that take advantage of evolving discipline specific best practices. There isn’t always an option. Most subject specific platforms are limited to STEM.

GSE: We’ve been talking about online learning platforms from an academic and cognitive development perspective. Is there anything from a social/emotional (SEL) standpoint that educators should be looking for in an online learning application? Or is teaching and developing a students’ SEL pretty impossible with distance learning?

Professor Pardos: There has been nascent work on measuring SEL in online environments. How to then respond to the measurement in a way that fosters growth using the technology is still an open question. It may not be impossible, but it’s not a current strength of distance learning platforms.

GSE: Are there any other resources you can point educators to?

Professor Pardos: For open educational resources, OER Commons is a good place to start. For a list of technologies that have “made the grade” as scored by the U.S. Department of Education, the Institute of Education Services has the What Works Clearinghouse. A student in my class on “Digital Learning Environments” recently shared an impressive index of online educational resources. It’s a sprawling, wild west sampling of what is out there.

GSE: Please share any other advice you feel is important for educators to have when assessing and selecting an online learning application.

Professor Pardos: The above advice is meant to help educators quickly navigate the space of learning technologies in this challenging time. Under less urgent conditions, don’t forget the professional development. Technology implementation takes time, involving a series of adjustments in order to be effective for students and to adapt to the goals of educators.

About Professor Zachary Pardos

Zachary Pardos studies the representation of knowledge as communicated by student behavior and engineers personalized supports leveraging big data in education. His current projects focus on increasing upward mobility in the California postsecondary system and using behavioral and semantic data to map out paths to cognitive and career achievement in K-16. He earned his PhD in Computer Science at Worcester Polytechnic Institute. Funded by a National Science Foundation Fellowship (GK-12), he spent extensive time with K-12 educators and students working to integrate educational technology into the curriculum as a formative assessment tool.