tl;dr: most educators think ai tools like ChatGPT instantly memorize everything students say and use it to train better models. that’s not how they work. but the real privacy risks in edtech are actually much weirder and more concerning.
one of the biggest misconceptions i see among educators, administrators, and ed policy folks involves how ai “training” actually works in tools like ChatGPT, Claude, and other large language models (LLMs) flooding into classrooms.
the concern is totally understandable: teachers don’t want to pilot these tools because they’re worried about contributing student data to training datasets. with recent incidents like the PowerSchool breach affecting 62 million students and the LAUSD chatbot failure, these fears feel even more justified.
but here’s the thing - this concern often stems from an incorrect mental model of how these edtech ai systems actually work.
short version: ChatGPT and similar classroom ai tools don’t directly learn from and memorize everything students say to them.
this can feel counterintuitive, especially in educational settings. when we work with students, we constantly update our knowledge based on what they tell us - their learning style, misconceptions, progress. and since computers have perfect memory, surely these ai tutoring systems would remember every detail of student interactions. isn’t that what “training” means in machine learning?
that’s not how these systems work.
llms are stateless functions in the classroom
from a technical standpoint, it’s best to think of LLMs as stateless function calls. given this input text, what should come next?
in a “conversation” with an ai chatbot - whether it’s ChatGPT, Claude, Copilot for education, or Google’s Gemini - that function input consists of the current conversation (everything said by both the student and the bot) up to that point, plus the student’s new prompt.
every time you or your students start a new chat conversation, you clear the slate. each conversation is an entirely new sequence, carried out entirely independently of previous conversations from both that student and other users across your district.
understanding this is key to using these models effectively in educational settings. every time anyone hits “new chat” they are effectively wiping the short-term memory of the model, starting again from scratch.
this has important consequences for classroom use:
there’s no point in “telling” a model something to improve its knowledge for future students. i’ve heard from teachers who spent weeks pasting curriculum content into ChatGPT sessions trying to “train” a better tutor for their class. that’s wasted effort!
this explains why “context length” matters so much for educational applications. different LLMs can consider different amounts of conversation at once - measured in “tokens” (roughly 3/4 of a word). if a tutoring session gets too long, the ai will “forget” what happened at the beginning.
sometimes it’s helpful to start fresh conversations when an ai tutor gets confused or starts making obvious mistakes. that reset might get it back on track for your student.
features like retrieval augmented generation (RAG) and ChatGPT’s new “memory” only make sense once you understand this fundamental limitation.
if you’re excited about locally-hosted ai models because they can’t possibly send student data to external servers, you’re mostly right - you can run them offline and monitor network traffic. but if you’re hoping for an ai that learns from your students and gets better at helping them over time… that’s probably not going to work the way you think.
so what is “training” in the context of educational ai?
when we talk about model training for educational ai, we’re talking about the process used to build these models in the first place.
there are roughly two phases. first, companies pile in several terabytes of text - think all of wikipedia, large portions of the web, books, academic papers, and more - spending months and potentially millions of dollars identifying patterns in how language works.
this gives you a model that can complete sentences, but not necessarily in ways that will help students learn effectively. the second phase aims to fix that - this includes instruction tuning and reinforcement learning from human feedback (RLHF) to teach the model to respond helpfully to educational prompts.
the end result is the model itself - an enormous file of floating point numbers that captures both statistical relationships between words and some version of “pedagogical taste” for assembling responses to student questions.
once trained, the model remains static and unchanged - sometimes for months or years. as one Anthropic engineer noted, the model is stored in a static file and loaded across thousands of identical servers. the model file never changes during actual classroom use.
these models don’t update very often!
why educators should still be cautious about student data
here’s the frustrating part: we can’t confidently say “don’t worry, ai chatbots don’t train on student input.”
many edtech ai providers have terms that allow them to improve their models based on classroom usage. even when opt-out mechanisms exist, they’re often opted-in by default - and most educators don’t even know these settings exist.
when OpenAI says they “may use content to improve services,” it’s unclear what they mean for educational contexts. recent policy changes in 2024-2025 have improved protections - ChatGPT Edu now offers enhanced privacy controls and 30-day retention policies - but implementation varies dramatically between consumer and enterprise education accounts.
the powerschool incident and lausd chatbot failure show the real risks aren’t always about training data. over 62 million student records were exposed through basic security failures, while the $3 million lausd “ed” chatbot collapsed with unclear data protections. the opt-out mechanisms are confusing, and many people don’t believe vendors when they claim not to train on educational data.
there’s also a trust crisis. when teachers tell students to disregard sensitive information they’ve shared, we know full well students won’t forget it. the same skepticism applies to ai vendors’ privacy claims.
like any cloud service used in education, there’s always risk of data exposure through security breaches - as the powerschool incident proved.
what about new “memory” features in educational ai?
to make things more confusing, edtech ai tools are introducing features that work around the stateless limitation.
ChatGPT recently added memory where it can “remember” details across conversations - your teaching style, student needs, curriculum preferences. this happens through a simple prompting trick: during conversations, the bot records short notes that get included in future chat contexts.
you can review and modify these remembered details anytime, and ChatGPT shows when it’s adding to memory. but for educational use, this creates new FERPA compliance questions about persistent student information storage.
bad edtech policy based on bad mental models
one of the most concerning results of these misconceptions affects institutional ai policies.
does your district ban all ai tools because they don’t want student data leaked to model providers? they’re not entirely wrong - see privacy concerns above - but if they’re acting based on the idea that everything students say is instantly memorized and could appear in responses to other students, they’re working from faulty information.
even more worrying is what happens with policymakers. with 25 states now having official ai guidance (up from 2 in 2023), how many education officials are crafting regulations based on science fiction ideas about how these systems work?
recent research shows ai detection tools flag black students’ work as ai-generated at twice the rate of white students, while the hingham high school lawsuit reveals how unclear policies lead to arbitrary punishments. if people believe classroom ai instantly memorizes and learns from student interactions, there’s real risk of supporting measures that address imaginary rather than genuine risks.
the 2024-2025 period revealed that our current privacy frameworks, designed for pre-ai educational technology, are inadequate for large language models. but understanding how these systems actually work - versus how we imagine they work - is the first step toward creating better policies that protect students while enabling beneficial ai applications in education. imagine they work - is the first step toward creating better policies that protect students while enabling beneficial ai applications in education.