Leveraging AI to improve adaptive tutoring systems

associate professor zachary pardos smiling at camera with circle graphic with O A tutor in center open source top left researcher friendly top right creative commons bottom center

For those who worry that the black box of AI is a threat to education, that teachers and students may become overly reliant on the technology, there’s good news. Researchers are using AI in an open way to build adaptive tutoring systems to help teachers differentiate instruction and students learn math and other subjects.

At the forefront of this work is Associate Professor Zachary Pardos, whose team in BSE’s Computational Approaches to Human Learning lab has developed a tutoring tool called Open Adaptive Tutur (OATutor) and released it in 2023. An open-source system, the tool has potential to powerfully assist research in the use of AI in learning scenarios.

OATutor is gaining recognition in the field. The tool is a finalist in the 2023-24 Tools Competition in the track titled "Engaging Adult Learners in Higher Education" (Growth Award category), and if selected could be awarded $150,000 (winners will be announced in summer 2024); and was awarded a second $50,000 microgrant from Berkeley's Office of Undergraduate Education in order to expand its use in select chemistry and mathematics courses.

This conversation has been edited for length and clarity.

First of all, what is adaptive tutoring?

Adaptive typically means that the tutoring system is making instructional decisions that are tailored to assessments it’s making of the student in real-time.

How does OATutor benefit from AI?

The initial release of the tool took a year to produce every textbook worth of adaptive content. Intermediate algebra took a year. College algebra took a year. Elementary algebra took a year.

What took the longest was the creation of the hints. What are you going to do when a student says they need help? Are you just going to show them the steps for the solution, which has been shown to work in other systems? We took four or five of our lessons and replaced all of the human tutor hints with ChatGPT-generated hints that had been screened for errors by a domain expert. We then did a learning gain experiment to see what the effect was. Do students learn as much from the ChatGPT generated stuff? The results showed a statistically insignificant difference between learning from the manually-created hints and ChatGPT hints. This means, for some subjects, adaptive tutoring systems could be produced many times faster with the aid of AI.

What are some applications for this tool that you’re excited about?

I am looking forward to the ways that this work intersects with other work we’re doing with community colleges around credit equivalency and credit acceptance.

Let’s say you’re coming from high school and you think the high school course you took was rigorous enough that you should get credit for it in college. Or you’re at a community college and you think the course you took is rigorous enough for you to get credit at a four-year university like Berkeley.

If it’s determined that the course only covers 50 percent of the material and it needs to cover at least 70 percent in order for you to get credit for it, maybe you can take some custom-created lessons on OATutor to make up the difference, instead of retaking the course at college or a university.

Do you plan to involve teachers in generating the queries that develop the hints?

Yes. Getting the teachers’ voice into the content and pedagogy of the system, that’s where generative AI may shine. Our system is very lightweight right now, [meaning] it’s not designed to have a graphical user interface for a teacher to edit and author content. If states, districts, or schools adopt this open source system, they can completely have control over the content. You could conceivably have a small IT department, as in just one person, be responsive to a teacher’s request to change content. Anyone can pick it up, make it their own and there are no restrictions, other than some technical knowledge needed to edit things.

However, with the help of generative AI, teachers could directly describe the background of their student cohort and give instructions to the system on the desired tone and tenor with which the system should interact with students. This is the direction we’re headed in with the next major version.

And it’s free.

Yes, the system has been released under an MIT License, which means it’s free for everyone to use and modify, including industry. More than just the system, we have also made our five textbooks worth of full semester-long adaptive content freely available and remixable under a Creative Commons license. This open and free approach should open the doors to more transparent and accessible innovation in this area of education technology.

Does OATutor have potential in non-STEM learning contexts?

I think it does. And perhaps it’s particularly appealing to use in those cases because of the generative AI/ChatGPT integrations we’ve made. I am meeting with a philosophy professor from the Royal Institute of Technology in Sweden. The appeal to him is he used ChatGPT to give feedback on some questions and responses, and it looked like ChatGPT had something valuable to offer in the way of feedback. But he couldn’t come up with the appropriate structure to suggest how students use ChatGPT. Should he tell them to go to chat.openai.com? That feels a little bit disjunct from the experience of the course. But when I gave a presentation about OATutor [in Sweden] and how we’ve been using ChatGPT to generate hints, he thought he could define a small assignment and then have ChatGPT give the feedback within the framework of the tutoring system. The log data would give him the indication of the student’s performance on the assignment and the interactions with ChatGPT, and our system works with Canvas, so he can link the OATutor assignment to the grade book in the learning management system.

Any closing thoughts?

I’m excited to get this tool out there. This is the first project I’ve spent three years working on before even attempting a publication. Now, finally, we’re making good on that pledge to have it open sourced and have presented it internationally in Germany, Sweden, Denmark, Japan, China, and Israel. The paper based on design of the system and its piloting in classrooms was accepted at our top choice of venue, a prominent conference on Human-Computer Interaction (CHI) where past seminal work on education technology has debuted.

This interview was conducted by contributing writer Kate Rix. Rix is an education reporter based in Oakland, Calif. Her reporting about school mental health services won the Education Writers Association's 2021 Award for Education Reporting. Her work has been published in USA Today, NBCNews.com and The Guardian.

Additional Resources