AI in EdTech Should Help Teachers Because It Can't Replace Them
Lessons from the failure of Khanmigo, Khan Academy's AI-powered Tutor
RIP Khanmigo & Edtech Industry Dreams of AI Tutors, says Dan Meyer. What happened?
Three years ago, Khan Academy launched Khanmigo, a ChatGPT-powered AI Tutor, which was expected to start a “revolution in learning”. It was supposed to be “probably the biggest positive transformation that education has ever seen.”
Given how well-known Khan Academy itself was all over the world, the amount of traction the announcement got was phenomenal:
Khan’s announcement of Khanmigo generated millions of views and precipitated widespread media attention, a book deal, and a steady flow of philanthropic and government subsidies. Newark Public Schools received $25,000 from the Bill & Melinda Gates Foundation for Khanmigo licenses. Indiana schools could apply for $50,000 in state funds for Khanmigo use. Palm Beach County received a $2,000,000 commitment from the Stiles-Nicholson Foundation to help schools purchase Khanmigo licenses. Microsoft eventually sponsored Khanmigo licenses for every teacher in 49 countries.
But, in spite of all the attention, the funding, and the initiatives in various schools, Khanmigo went nowhere. As Sal Khan himself admitted last week, “For a lot of students, it was a non-event. They just didn’t use it much.”
Isn’t 1-on-1 Tutoring the Holy Grail of Education?
A famous finding in education is called Bloom’s 2-Sigma Problem. In the 1960s, Benjamin Bloom conducted some studies which showed that average students who got 1-on-1 tutoring using mastery learning techniques performed 2 standard deviations better than students educated in a regular classroom environment. 2 standard deviations is a huge number: that is the equivalent of a 100 IQ student performing at 130 IQ level or a CBSE 12th std percentage going from 72% to 92%.
And it is called a “problem” because until now, we did not have a scalable way of providing 1-on-1 tutoring or mastery learning at scale.
AI tutoring was supposed to be a perfect solution to this “problem.” An AI tutor is excellent at providing 1-on-1 tutoring and it can easily implement mastery learning. It can seamlessly adjust explanations to the level of the student, give examples that student can connect with, answer any questions the student might have with infinite patience1, and keep circling back until the concept truly clicks—no shame, no time pressure, no moving on before the student is ready.
And Khanmigo was ideally placed to solve Bloom’s 2-Sigma Problem. What went wrong?
The 5-Percent Problem
The problem, as is clearly seen from Sal Khan’s lament, is that most of the students are not using the tutor, so obviously they’re not going to see any improvement. As Naval Ravikant puts it:
The means of learning are abundant, the desire to learn is scarce
In short, the real problem isn’t Bloom’s 2-Sigma Problem. The real problem is Education’s 5-Percent Problem.
Here are studies showing that technology is indeed a great tutor:
hed the results of a massive, 99-district study of students. It showed an effect size of 0.26 standard deviations (SD)—equivalent to several months of additional schooling—for students who used the program as recommended.
A 2016 Harvard study of DreamBox, a competing mathematics platform, though without the benefit of Sal Khan’s satin voiceover, found an effect size of 0.20 SD for students who used the program as recommended. A 2019 study of i-Ready, a similar program, reported an effect size in math of 0.22 SD—again for students who used the program as recommended. And in 2023 IXL, yet another online mathematics program, reported an effect size of 0.14 SD for students who used the program as designed.
These gains, even though less than 2 SD, are quite impressive. And these are pre-ChatGPT, so the situation now should have been even better. So what’s the problem?
A clue is in those wiggle words “students who used the program as recommended.” Just how many students do use these programs as recommended—at least 30 minutes per week in the case of Khan Academy? The answer is usually buried in a footnote, if it’s reported at all. In the case of the Khan study, it is 4.7 percent of students. The percentage of students using the other products as prescribed is similarly low.
Coursera completion rates have a similar number. I’ve been complaining about this myself since 2019. Once you realize that only 5% of the students are self-motivated enough to use edtech tools, the problem becomes clear:
Imagine a doctor prescribing a sophisticated new drug to 100 patients and finding 95 of them didn’t take it as prescribed. That is the situation with many online math interventions in K–12 education today. They are a solution for the 5 percent. The other 95 percent see minimal gains, if any.
“Khanmigo did not fail the way bad software fails. It failed the way gym memberships fail: the equipment was there, the promise was real, but most people did not show up.” —ChatGPT5.5
Pivoting to Humanity
Dan Meyer has a follow-up article pointing out that the only way AI in particular and EdTech in general can make a difference is by “pivoting to humanity”. For quite some time now, EdTech has tried to solve the 2-Sigma Problem by replacing classrooms+teachers with technology. And, in the pre-ChatGPT days, it was possible to make the claim that the reason EdTech hasn’t succeeded so far is because the technology was not mature enough. But, now, it is time to admit that technology is not going to solve the problem by replacing the humans, so any hope of making any headway lies in technology helping the human teachers2.
Why? Because teachers do much more than teach:
“As soon as I started hearing about tutors being replaced by AI, I knew that the people responsible for such nonsense had never tutored a day in their life. 40% is remembering to ask about the novel they are writing, the tea3 they spilled about their friends, or the language test they’ve been studying for all year. 40% of my time is spent just building confidence and reassuring students they’re doing the right thing. 20% is actually teaching math.” —Katelynn Petersen
“EdTech keeps mistaking the most visible part of teaching for the whole of teaching. The explanation is visible. The worksheet is visible. The answer is visible. But the real work often lies elsewhere: noticing who has stopped trying, who is pretending to understand, who needs confidence more than content, who needs a nudge, who needs the class to laugh with them and not at them, who needs patience and who needs impatience. A tutor can explain fractions. A teacher can make a child willing to try fractions again.” —ChatGPT5.5
What does it mean for EdTech to pivot towards humanity? Instead of trying to replace teachers, EdTech should try to solve the problem that human teachers face. Dan has an interesting list of hard problems:
Kids want to work on paper and it’s hard for teachers to know what they’re doing.
Teachers struggle to support a student’s thinking if it isn’t visible. Lots of thinking happens on paper and teachers often lack the time necessary to review and respond to it. How can we make that paper-based learning more legible for more teachers?
Coaches want to support teachers and teachers want their support.
Coaches often have too many teachers on their roster to support with model lessons and walkthroughs. Also, many teachers want support but not in the form of a model lesson or walkthrough. The desire to give and receive support are misaligned here.
Teachers want support in leading whole-class discussions.
Whole-class discussion is some of the most satisfying work for teachers and productive learning for students. But it is very hard work. Misaligned.
Caregivers want to support their kids but don’t know how.
Many parents and caregivers want to do more to support their kids’ learning than they do currently. But they often lack visibility into student learning and may need some education themselves. The reports that schools send home are frequently summative, low resolution, and a waste of ink or pixels overall. Teacher emails are much more useful but time-consuming for the teacher. What can schools and edtech companies do to help align caregivers, teachers, and kids here?
TMKK
So what’s the takeaway message4?
If your child (or you!—you are also a student who can benefit from 1-on-1 AI tutoring) is happily using AI as a tutor, you should absolutely encourage it. Because this means that your child is in the 5% who will actually benefit. The main thing to make sure is that your child is using the AI to learn (i.e. to get explanations for concepts) and not using it to avoid learning (i.e. to do the homework). Here are some interesting ways to use AI as a personal tutor from Ashish: Learn by Doing, Learning, Doing, Thinking
If you are a teacher or educator, I’m sure you’re being bombarded by various AI tools and initiatives (even I teach one such course 😂). Evaluate those through the lens, “How does it enable the teacher to help the average student better?”
If you are techie/builder in the EdTech space, consider pivoting to humanity. Focus on the problems Dan listed, or try to find other real problems that good teachers are facing when helping the average students, by treating good teachers as the experts (you understand tech, they understand education—the latter is more important).
(I don’t use AI to write any of my articles. However, this time, I found a couple of its suggestions good. So I’ve included those in callout blocks with appropriate attribution.)
When Shridhar Shukla retired, he decided that improving education was one area that he wanted to spend his time and money on. And after studying that space for a while, he reached the conclusion that the key to any improvement in the area was to focus on improving the support systems for the teachers so they can focus on the child. This article is saying something similar.
I hope you know what “tea” means in this context. If not, ask your favourite AI!
TMKK stands for “Toh MaiN Kya KarooN”