Leveraging AI to improve adaptive tutoring systems

First of all, what is adaptive tutoring?

Adaptive typically means that the tutoring system is making instructional decisions that are tailored to assessments it’s making of the student in real-time.

How does OATutor benefit from AI?

The initial release of the tool took a year to produce every textbook worth of adaptive content. Intermediate algebra took a year. College algebra took a year. Elementary algebra took a year.

What took the longest was the creation of the hints. What are you going to do when a student says they need help? Are you just going to show them the steps for the solution, which has been shown to work in other systems? We took four or five of our lessons and replaced all of the human tutor hints with ChatGPT-generated hints that had been screened for errors by a domain expert. We then did a learning gain experiment to see what the effect was. Do students learn as much from the ChatGPT generated stuff? The results showed a statistically insignificant difference between learning from the manually-created hints and ChatGPT hints. This means, for some subjects, adaptive tutoring systems could be produced many times faster with the aid of AI.

What are some applications for this tool that you’re excited about?

I am looking forward to the ways that this work intersects with other work we’re doing with community colleges around credit equivalency and credit acceptance.

Let’s say you’re coming from high school and you think the high school course you took was rigorous enough that you should get credit for it in college. Or you’re at a community college and you think the course you took is rigorous enough for you to get credit at a four-year university like Berkeley.

If it’s determined that the course only covers 50 percent of the material and it needs to cover at least 70 percent in order for you to get credit for it, maybe you can take some custom-created lessons on OATutor to make up the difference, instead of retaking the course at college or a university.

Do you plan to involve teachers in generating the queries that develop the hints?

Yes. Getting the teachers’ voice into the content and pedagogy of the system, that’s where generative AI may shine. Our system is very lightweight right now, [meaning] it’s not designed to have a graphical user interface for a teacher to edit and author content. If states, districts, or schools adopt this open source system, they can completely have control over the content. You could conceivably have a small IT department, as in just one person, be responsive to a teacher’s request to change content. Anyone can pick it up, make it their own and there are no restrictions, other than some technical knowledge needed to edit things.

However, with the help of generative AI, teachers could directly describe the background of their student cohort and give instructions to the system on the desired tone and tenor with which the system should interact with students. This is the direction we’re headed in with the next major version.

And it’s free.

Yes, the system has been released under an MIT License, which means it’s free for everyone to use and modify, including industry. More than just the system, we have also made our five textbooks worth of full semester-long adaptive content freely available and remixable under a Creative Commons license. This open and free approach should open the doors to more transparent and accessible innovation in this area of education technology.

Does OATutor have potential in non-STEM learning contexts?

I think it does. And perhaps it’s particularly appealing to use in those cases because of the generative AI/ChatGPT integrations we’ve made. I am meeting with a philosophy professor from the Royal Institute of Technology in Sweden. The appeal to him is he used ChatGPT to give feedback on some questions and responses, and it looked like ChatGPT had something valuable to offer in the way of feedback. But he couldn’t come up with the appropriate structure to suggest how students use ChatGPT. Should he tell them to go to chat.openai.com? That feels a little bit disjunct from the experience of the course. But when I gave a presentation about OATutor [in Sweden] and how we’ve been using ChatGPT to generate hints, he thought he could define a small assignment and then have ChatGPT give the feedback within the framework of the tutoring system. The log data would give him the indication of the student’s performance on the assignment and the interactions with ChatGPT, and our system works with Canvas, so he can link the OATutor assignment to the grade book in the learning management system.

Any closing thoughts?

I’m excited to get this tool out there. This is the first project I’ve spent three years working on before even attempting a publication. Now, finally, we’re making good on that pledge to have it open sourced and have presented it internationally in Germany, Sweden, Denmark, Japan, China, and Israel. The paper based on design of the system and its piloting in classrooms was accepted at our top choice of venue, a prominent conference on Human-Computer Interaction (CHI) where past seminal work on education technology has debuted.