AI Tutors Fail to Meet Tutoring Criteria

Interesting findings that will resonate with all my peers who are researching AI in education and coaching: Generative AI is not naturally very good at tutoring - it is designed to give answers and suggest solutions. You have to really cajole it to listen* to the user and hold back answers - and even then it fails a lot of the time. (*or diagnose misconceptions in this case)

Can AI Tutors Actually Teach? We Measured It We ran 779 simulated tutoring conversations to test whether AI models can teach, not just answer. We simulated an 8th-grade math student with a planted misconception (drawn from documented algebra errors), six frontier models acting as the tutor, and a rubric for what counts as teaching. Three things stood out: 1. Out of the box, models don't tutor. Without a tutoring prompt, they handed the student the answer in 97% of conversations. Zero successful tutoring outcomes in 298 tries. 2. A tutoring prompt helps, but unevenly. Answer-giving dropped sharply, but no model was good at both halves of the job, diagnosing the misconception and holding back the answer. 3. Price didn't predict quality. The cheapest model (about $0.002 per conversation) scored best on the strict rubric. One that costs 12x more did worse. The models were rarely factually wrong. We found 3 false statements across all 779 conversations. The problem is that handing over answers isn't teaching, and there's no standard way for buyers to tell the difference ahead of time. Funders are putting a lot of money into AI tutoring. Independent, ongoing evaluation is inexpensive by comparison, and it's the gap they're best positioned to close. Full methods, results, and caveats: https://lnkd.in/gJC9NhyV

I’ve been teaching Mathematics for over a decade now, and I realise that a lot of what I do in a lesson is not really “explaining maths.” I’m constantly watching, listening and making small decisions based on what my students do. My biggest takeaway from building an AI tutor is that tutoring is much more complex than it looks. The hard part is not explaining something. AI is already very good at that. The hard part is knowing what the student needs to do next, in the moment, and then explaining it in the right way. Yes, because “tutoring happens in conversations.“ If a student gets a question wrong, why? Did they misunderstand the question? Is it a misconception? Did they forget something they knew previously? Did they make a careless mistake? The tutor has to work that out and decide what to do almost immediately, one step at a time; visually, verbally and sometimes even through movement. And if the student still doesn’t get it, the next explanation shouldn’t just be another version of the same explanation. It should find and respond to what the student is actually struggling with. We’ve had to think about this deeply while building Lumi, and it has made me appreciate even more the work we do almost instinctively as teachers. A tutor can’t just have a library of great explanations. They need to adapt the next teaching move based on the student’s response. They need to know when to slow down, when to ask another question, when to draw something, when to give a simpler example, when to let them struggle a little longer, and when to change the approach completely. So far, our experience is that a good AI tutor can do a lot of the co-teaching: helping students understand concepts, breaking ideas down and giving them a strong foundation. But the deeper learning — application, reasoning, responding to unexpected thinking, and knowing exactly when and how to push a student — is still where human tutors bring something very special. That’s why this quarter, we’re integrating Lumi AI into HiPrep Online , where Lumi will continue to support students as their AI learning companion, but now alongside our 250+ expert tutor pool. We will share what this looks like in practice. #edtech #k12education #AIineducation

This is a great study, and the data is fascinating. But I think it points to a bigger problem in the AI tutor race: we're confusing AI capabilities with AI tutoring. AI can generate the right answer, explain a concept, or even produce a Socratic-looking response. None of that necessarily means it is teaching. The real job starts with understanding the learner: what do they know, what don't they know, where are the learning gaps, and what misconception is causing the error? The tutor then needs to design the right intervention to help the learner close that gap not just give them the answer. And even getting the learner to the right answer isn't enough. Can they now solve the problem independently? Can they apply the same mental model to a different problem?  Because that's where transfer happens, and that's when learning has actually occurred. I also think the best AI tutors will involve humans in the loop. There will be moments where context, emotion, motivation or a deeper learning challenge requires human judgment.  The system should know when to hand off to a human, and when to take the learner back. So perhaps the real AI tutor is a system that can understand the learner, diagnose the gap, intervene, verify mastery and then transfer to a human tutor.

AI tutors will typically give out the answer which makes them bad tutors. Here is a different method. A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior https://lnkd.in/gVUCXtxG

High-quality instruction and human interaction are still essential for learning at all levels. An LLM can’t tease out your misconceptions.

So many thoughts from reading this article. Definitely worth a perusal. Some quick thoughts: (1) Tutoring whether via human or chatbot would fall under ESSA mandates as an intervention, so research to verify they are evidence-based interventions sounds like a given. (2) A drywaller once told the homeowner, that was inspecting their work and pointing out shoddy or sloppy finishing, "It's whatever you're willing to accept." The homeowner said, "So, you wouldn't fix any of this if I didn't point it out?!" The reply, "Like I said, it's whatever you're willing to accept." Are leaders going to... (a) ask for ongoing measurement of strict clean-win rate /API cost per cost per conversation, prompted condition (what should be labeled Table 4), (b) ask for a particular LLM version to remain unchanged for the duration of a school year, and (c) be aware if vendors/partners/providers change out the LLM without notice for profitability as API costs change? (3) My thoughts don't even go onto privacy and data protections here because even if public schools do not opt for AI tutors in the near future, then there will still be the private market that will. Individual parents seeking have their student tutored (https://lnkd.in/g2RBBTv9) or private institutions looking for a tutor that can always be accessible. (4) Regarding the PNAS article, we cannot let FOMO guide us on policy or adoption.

Too many educators and institutions fear letting go. So they hold on to old methods. Old teaching patterns. Old assessment systems. Old ways of preparing students. And slowly, they limit what students could become. They have smart teachers. Talented students. Powerful technology. That’s where the real opportunity is being missed. Because when students are always dependent on someone for every answer, a quiet message gets created: “You cannot learn this on your own.” And students adjust. They stop exploring. Stop questioning. Stop experimenting. Stop thinking beyond the textbook. But what if technology could change that? What if AI could become a learning partner, rather than simply an answer generator? That is the idea behind Edumath AI Buddy. I believe the future of education is not about replacing teachers. It is about amplifying teachers and empowering learners. Here’s where AI can unlock learning 👇 1. Give students the right guidance at the right moment Instead of immediately giving an answer, AI can help students understand where they are stuck and guide them toward the solution. 2. Make learning more personalized Every student learns differently. AI can support different learning speeds, different levels of understanding, and different learning needs. 3. Turn doubts into learning opportunities A student's question should not be a weakness. It should be the starting point of deeper understanding. 4. Move beyond passive learning Students shouldn't just read formulas and memorize answers. They should ask, solve, practice, analyze, and improve. 5. Give teachers more time for what matters When technology supports repetitive learning tasks, teachers can spend more time on mentoring, explaining concepts, identifying gaps, and inspiring students. 6. Make practice smarter AI-powered practice can help students identify weak areas and focus their effort where it matters most. 7. Build independent learners The ultimate goal isn't to make students dependent on AI. It is to make them confident enough to think independently. That is the difference between: AI that gives answers AI that helps students learn how to find answers. This is why I created Edumath AI Buddy. A learning platform designed to bring AI-powered guidance, academic resources, practice, and interactive learning into one learning ecosystem. The future of education won't belong to those who simply use more technology. It will belong to those who use technology better. Teachers shouldn't be replaced. Students shouldn't become dependent. AI should become the bridge between curiosity and understanding. If you are a student, teacher, parent, school leader, or education enthusiast, I invite you to explore what we are building. 🌐 Explore Edumath AI Buddy: https://lnkd.in/ef-zih8A #EdumathAIBuddy #AIinEducation #ArtificialIntelligence #EdTech #FutureOfEducation #EducationTechnology #PersonalizedLearning #StudentLearning #Teachers #AIForStudents #EdumathBySereddy

When I was in elementary school and even middle school, math came easily to me. However, when I hit 9th-grade algebra, something changed. The first thing was that the teacher did not seem to care. He did not show us anything on the board or explain what we were working on. He simply said, “Read chapter… and do the problems at the end of the chapter for homework.” I didn't do so well in algebra. I don’t remember, but I suspect there were others in the same boat. The next year, I had Mrs. Johnson for geometry. She was amazing and brought the content to life. She helped me catch up in algebra. I would come in at lunch, and she would work with me. It is still not my strongest area, but I can work my way through most problems. I like to think AI-personalization would have helped me in 9th grade more than my teacher did! AI‑driven personalization is ushering in a new era of learning, one where experiences adapt in real time, respond to individual behavior, and evolve as learners progress. For the first time, we can design learning ecosystems that feel less like static pathways and more like dynamic journeys. Adaptive sequencing, intelligent recommendations, and responsive content flows are transforming personalization from a design aspiration into a scalable reality powered by AI‑driven personalization. Adaptive pathways are the foundation of this shift. Instead of forcing every learner through the same linear progression, AI can analyze engagement signals, performance trends, and behavioral patterns to determine the next best step. Learners advance at the right pace, with the right level of challenge and support, creating momentum that traditional models simply can’t replicate. Intelligent recommendations deepen that momentum. AI can surface targeted resources, practice opportunities, and micro‑learning moments precisely when learners need them. These recommendations aren’t generic; they’re context‑aware, informed by real‑time data, and aligned to each learner’s evolving goals. It’s the difference between navigating a catalog and being guided through a curated experience. Dynamic content sequencing is where personalization becomes truly transformative. AI can reorder modules, adjust complexity, introduce new scenarios, or scaffold additional support based on how learners interact with the experience. Learning becomes fluid, responsive, intuitive, and deeply relevant. This accelerates skill development by ensuring learners spend time where it matters most, not where the curriculum assumes they should. For learning leaders, AI‑enabled personalization doesn’t replace human judgment; it amplifies it. When AI handles the adaptive mechanics, learning teams can focus on strategy, creativity, and designing experiences that feel human, intentional, and transformative. How is AI reshaping the way your learning experiences adapt to individual learners in real time? Photo by The New York Public Library on Unsplash