Opinion: The School AI Problem Isn't Adoption, but Visibility - GovTech

For years, school technology leaders have been dealing with shadow IT. Applications get adopted by teachers, staff or students without going through the school’s formal procurement or security processes. AI is accelerating the problem. A teacher can create an account with an AI service in minutes, connect it to a school Google Workspace or Microsoft 365 account, upload a document or authorize access to files — all without involving IT.

For example, a recent security incident at the machine learning company Hugging Face involved an improperly secured testing environment. It showed how a gap in configuration could create consequences that even sophisticated technology environments struggle with. Now, Hugging Face also runs a dedicated education program with a model hub that teaches machine learning to students. Launched in 2022, their ML Demo.cratization tour had experts teaching hands-on classes to more than 1,000 students from 16 countries.

This means educators and students are already working with and learning on a platform affected by the AI security breach, one that is not directly part of the K-12 curriculum, yet is a vital learning tool.

For K-12 leaders, the takeaway is not that AI is inherently unsafe. It is that technology is advancing faster than many organizations’ governance and security practices. This gap deserves attention before districts expand their use of AI.

AI IS CHANGING AN OLD K-12 PROBLEM

AI is making the problem harder because it is increasingly embedded in the applications schools are already using. It’s no longer standalone AI tools; AI capabilities are now appearing across software and platforms that already have access to school data.

At the same time, AI systems are becoming more capable. While a traditional chatbot generates a response based on a prompt, more agentic systems are capable of interacting with files, workflows and applications, and this changes the security requirements.

Schools now need to monitor what information a tool with AI capabilities can access, where this information goes, what permissions it requires, what actions it can take and what third-party services or models it relies on.

YOU CAN'T GOVERN WHAT YOU CAN'T SEE

The biggest blind spot for many districts is the simplest one. They do not know all of the AI tools being used by their employees and students, and that makes traditional approval-based policies difficult to enforce. A school might have a careful review process for software it purchases centrally, but that process does not necessarily cover an application that a teacher or student finds independently and begins using in a classroom.

A user can connect an application to a school account, authorize access to files or paste information into an AI prompt without understanding what happens to that data afterward. The cybersecurity risk thus extends beyond the AI model to the application, account, permissions and data.

That is why visibility has to come before governance. Before a school can establish a meaningful AI policy, it needs to understand what is already happening inside its environment.

DON'T TRY TO BUILD AN AI RULEBOOK FOR EVERY TOOL

District leaders may be tempted to respond by creating long lists of approved and prohibited AI applications, but this approach will be difficult to sustain. AI is changing too quickly for a school to review an application once and assume its security profile will remain unchanged. Features, integrations, permissions and underlying models are always evolving.

A better approach is to establish a set of questions that apply across applications:

- What information can this application access?

- What information should users be permitted to enter?

- Does it connect to district Google Workspace or Microsoft 365 accounts?

- What permissions does it request?

- How is student information handled?

- Who is responsible for reviewing the risk?

Using an AI tool to brainstorm a classroom activity is different from uploading student records or connecting an autonomous agent to a district account. Those activities should not necessarily face identical controls. This is where governance has to be practical rather than restrictive.

GIVE TEACHERS ROOM TO EXPERIMENT — SAFELY

The solution cannot simply be to tell teachers and students not to use unapproved AI. AI is already too accessible and too useful for that approach to be realistic. Like with Hugging Face’s machine learning tool for educators and students, various large language models are constantly lowering their access barriers to accommodate newer coding tools and platforms that schools can use. The next AI-powered hack could be just days away.

Instead, schools need practical guidance about what information should never be entered into an AI system, when an application requires IT review, and how to recognize when a tool is asking for more access than it needs.

IT teams should regularly review third-party applications and OAuth permissions, monitor account activity for unusual behavior, enforce least-privilege access and maintain a process for quickly investigating suspicious activity.

Teachers also need to understand their role in protecting student information, and administrators need to understand the risks created by uncontrolled application access. District leaders need to make sure their AI policies align with instructional goals rather than treating cybersecurity and innovation as competing priorities.

Recent K-12 technology discussions increasingly reflect this shift. Districts are moving from early experimentation with AI toward questions of governance, security and trust.

THE GOAL IS SAFE EXPERIMENTATION

Schools should never have to choose between protecting their systems and allowing educators and students to explore new technology. But experimentation without visibility creates risks that districts may not discover until after something goes wrong.

For schools, the priority should be getting the fundamentals right, as AI will continue to evolve faster than most district policies can. That means schools should not build governance around predicting which tool will be popular next year. They should build it around principles that remain relevant regardless of the technology: visibility, appropriate access, data protection and accountability.

The goal is not to put AI behind a locked door. It is to make sure that when schools open the door to new technology, they know exactly what they are letting in.

Charlie Sander is CEO and chairman at ManagedMethods, a Boulder, Colo.-based cybersecurity and student safety platform for K-12.