Artificial intelligence is transforming American schools. Across the country, school districts are adopting software that monitors students’ activity on school-issued devices. Companies such as GoGuardian report that thousands of school districts use their products and monitor the activity of millions of students. Recently, NYRA discussed how using intensive AI surveillance of students raises many constitutional questions about student rights. However, in this article, I want to be examining the harms of all types of surveillance on students, and the ways that schools surveilling students’ devices can violate their rights, threaten their safety, and undermine their privacy.
It is important to note that this surveillance existed before modern AI as we know it was widely used, and that the usage of AI is not inherently the problem. The problem lies within the surveillance itself, and how school surveillance of student devices puts them at unnecessary risk, while also threatening their privacy. While previous surveillance programs mostly focused on monitoring students’ search history, robust modern AI surveillance frameworks have expanded school surveillance capabilities. Now, AI Programs can scan emails, Google Docs, internet searches, chat messages, and other online activity for problematic behavior, usually targeting signs of violence, self-harm, bullying, or drug use. If certain words or patterns are detected, the software can automatically alert teachers or administrators so they can intervene before a crisis occurs.
Although these are good intentions, can we really trust that there will only be positive results of this increased surveillance? Can we really justify violating the privacy of every student in order to intervene in these types of situations? Well I would argue that all digital surveillance of students—AI or not—is inherently too risky, and more likely to harm vulnerable youth rather than help them.
Table of Contents
- False Positives of Student Surveillance – Why we Can’t Trust AI or School Administrators
- The Harms of Google Docs Scanning
- The Importance of School Devices for Vulnerable Youth
- How Forced Disclosures of Student Struggles by Schools Can be More Harmful than Beneficial
- How AI Surveillance in Schools Could be Used to Harm, Target and Profile LGBTQ Youth
False Positives of Student Surveillance – Why we Can’t Trust AI or School Administrators
When discussing the potential harms of excessive AI surveillance of students it is important to realize that false positives are a major concern. No AI system is perfect, and when the program mass-scans hundreds of thousands of words in google docs, internet searches, and digital activity, it is bound to make mistakes. In fact, there have been many reported instances of these mistakes happening.
In one case, a 16-year-old was pulled out of calculus class after her school’s AI monitoring system flagged her as a potential suicide risk. She had not written about harming herself. Instead, she had been researching teen anxiety and suicide for a debate competition. The software detected the keywords but failed to recognize the context. The student later said she had no idea why she had suddenly been called into the counselor’s office until she realized that AI had mistaken her academic research for a mental health crisis.
In a more serious example, in Tennessee, a 13-year-old student’s messages triggered an alert through the school’s AI monitoring software. Although investigators later concluded there was no credible threat, the student was interrogated, strip-searched, and placed under house arrest after the software flagged the conversation. The case became one of the most widely discussed examples of how AI-generated alerts can quickly escalate into life-changing consequences for students. In this situation, no second-thought was given to the case, no nuance was discussed. The student was unjustly punished due to AI surveillance making a mistake. And as shown from the intensive consequences she received, these mistakes have the potential to drastically harm students’ lives.
When I was in school, even before AI surveillance of students was regularly used, I observed some of the unfair consequences from the monitoring of school devices. For example, in high school, one of my friends used another one of my friend’s school chromebook to google search lyrics to a song. Nobody thought anything of it, until the friend whose chromebook was used got called into the consular’s office and reprimanded. Apparently, the lyrics to the song were flagged as “promoting violence”, by the school’s surveillance system, which caused my friend to get into trouble. Although this was a completely innocent action, and neither one of the friends involved had any intention of actually exhibiting any violent behavior, the surveillance still viewed the lyrics as problematic, and thus, the school intervened.
I think about this scenario often when it comes to AI surveillance in schools, because there were so many innocent behaviors like this one that me and my friends partook in, that could put us at risk for punishments if AI flagged it. AI cannot understand context, AI cannot understand human mannerisms and behavior like a human can. This is why, the more intensive the AI surveillance is used, the more risky it is. I remember how me and my friends used to use shared google docs and google slides with our school chromebooks to communicate and make jokes with each other during class. Everything that was said between us was all in innocent fun, without any serious connotations behind it—like how teenagers often joke with each other. Someone annoys you, or makes a corny joke, and you come back with an “im gonna kill u” in response. Or we may have jokingly insulted each other and made fun of each other for random things. Of course, there was nothing serious to it, just a few teenagers messing around. However, AI doesn’t see it that way. AI doesn’t understand the context of jokes, or friends ironically throwing shade at each other. To an AI surveillance system, it sees something that it deems as a threat or bullying, it flags the text, and the school will end up punishing the individuals involved.
Many people may argue that the solution to this is not to remove surveillance of students, but to instead increase human intervention, and make sure that school administrators examine the facts of each individual case before punitive action is taken. However, this is not realistic. First of all, it is commonly known that schools will always default to the easiest solution. This is perfectly illustrated by Zero Tolerance policies in schools, where blanket punishments are given out for the same type of behavior regardless of context. These policies have led to several ridiculous punishments, such as a student being suspended for violating the school’s “Zero tolerance phone policy” by using their phone in class to record a teacher using racial slurs. This is only one instance of the unfair and unjustified punishments that result from schools taking the easiest way and instead opting to use “one size fits all” punishments for behavior they deem problematic.
Along with this, even when there is human intervention and the school administrators are the ones making the decisions without AI, that still is not a guarantee that they will make the right choice. Similar concerns existed even before AI monitoring became widespread. In Connecticut, an honors student was suspended and removed from his specialized high school program after submitting a fictional horror story for an AP English assignment. Although a court-ordered mental health evaluation found no evidence that he posed a threat, the student was disciplined because school officials interpreted the fictional story as dangerous. The case illustrates how schools can mistake creative expression for genuine threats—and also shows how human intervention in student surveillance is not immune to mistakes and unjustified punishment.
Simply put, we cannot trust AI to properly dictate what behavior is actually problematic, and what behavior isn’t. We also cannot trust school administrators to properly examine the context to a situation that arises from surveillance over student work. Both leave room for error, and those errors have debilitating consequences for the students caught in the crossfire. The solution is not to have more human involvement; the solution is to decrease the overall amount of surveillance that students are subject to, in order to minimize the risk of dangerous false positives and other privacy violations.
The Harms of Google Docs Scanning
The story about my friends and I using google docs for jokes during school was only the tip of the iceberg of how using AI to scan google docs for problematic text is potentially harmful for students, as well as being a blatant violation of their privacy. First and foremost, it is important to understand that google docs is not just limited to being used for schoolwork and research. It is a free notepad that is organized, and allows anybody to write down anything they want at any time. Because of its accessibility and efficiency, it is often used for personal uses that expand far beyond just what students are required to use it for at school.
In my own personal experience, as someone who is neurodivergent with social anxiety, school could often get overwhelming and stressful for me. This isn’t something that is unique to me, and is actually something that most students in the US experience. According to studies by the APA, 83% of U.S. teens say school is a significant source of stress. This comes from the American Psychological Association’s Stress in America survey, which found that school was by far the most commonly reported source of stress among teenagers. In similar studies, 31% of teens say they feel overwhelmed because of stress.
One of the ways that I helped manage the stress caused by school, is to have a “boredom doc” that I used during class. Basically, this was just a google doc on my school chromebook, that I would open every time I was stressed, overwhelmed, bored, etc, and I would type out everything I was feeling at the moment. Whenever I was upset about something and it was distracting me, I used this google doc to type my thoughts as a coping mechanism. This was a great way for me to relieve my stress and calm myself down when I felt anxious. Later in the year, me and two of my close friends opened a shared boredom doc, where we would all collectively use it to write everything on our minds, vent about stress, and communicate with each other during class. Again, this was a great way to keep us connected, help us manage stress, and help us get to vent about stressful situations during the school day.
However, due to the personal nature of the google docs, the stuff we would write about often included sensitive subjects, that were very private, and could have potentially been flagged by AI monitoring software, if our google docs had been getting scanned by them at that time. This means that students in school now, using similar methods to vent about their day, could potentially face unwanted school intervention for AI flagged text in personal google docs. This takes away a healthy coping mechanism for school stress, and takes away a safe place for students to vent about issues during the day.
Some may say that there is no reason for a student to be doing this on a school computer, and that everything that occurs on a school device has the right to be surveilled by the school. However, school policies often make any other forms of coping like this completely unaccessible. For example, many schools institute all day phone bans, with most schools banning phones at all times during class and some schools even requiring lockable phone pouches all day. These phone bans threaten student safety in many ways, but they also effectively make it to where students wouldn’t have any other device they could use to do this venting method during the day at school. So students who are stressed, overwhelmed and need to vent have no other option besides using a school computer, and thus opening up their private thoughts to being scanned and used against them.
Along with this, private venting isn’t the only type of content that could be scanned and flagged by AI in google docs. When I was in high school, I often finished my work early, and of course, couldn’t spend my free time talking with others because of school rules, or on my phone because of phone bans, so I spent my extra time working on my own personal projects. One of these projects that I often found myself working on while at school, was my own personal novel. When I finished my school work, I would open up my novel’s google doc that I shared to my school chromebook, and pick up where I last left off in the story. However, the book did include content that AI surveillance could view as harmful, such as references to guns, drugs, bullying, violence etc. None of these things were signs that I personally was going to engage in that behavior, it was simply part of a fictional story that I had created.
AI surveillance wouldn’t see things that way though, and it would only scan the google doc for problematic text, flag it, and send it to the school administrators for intervention. As previously mentioned, school administrators have even punished students for fictional stories, if the stories contained violent content, so it’s reasonable to assume they would do the same, even if the content was found from a scanned google doc and not even submitted as an assignment. Should students’ private works, and creative expression be subject to constant monitoring, putting them at risk of punishment or unnecessary intervention. Should students’ private thoughts and experiences typed within personal documents always have a school’s watchful eye over it. The answer is simple: No. Scanning of school devices puts students’ sensitive information at risk, and should not be a normalized process.
The Importance of School Devices for Vulnerable Youth
One of the other major flaws in school surveillance policies is that the surveillance doesn’t end at school. Unlike a traditional search, AI surveillance is continuous. These systems monitor thousands of students’ devices simultaneously, at all times. This means that the AI surveillance doesn’t just monitor school devices during school hours while the student is on campus. Many schools give out chromebooks and laptops to all students, which are on the student at all times, taken home with them, and brought to school each day. If a student takes their school chromebook home, and uses it in the privacy of their own house after school has ended for the day, they are still subject to the surveillance and potential flagging of their digital activity, despite being at home. This is a violation of student privacy, and also disproportionately affects vulnerable youth who rely on school devices more than others.
First of all, not all students can regularly afford their own devices. Students from low-income families may not have computers of their own, that they can use as their personal devices. This means that if they receive a school device that they can take home overnight, that device would end up becoming their main device, which they would then rely on for personal uses outside of school. Because personal uses of a device outside of school often involve sensitive and private information, searches and expression, a school’s AI program scanning it would put the low-income student’s privacy at risk. Again, because of the fact that the AI system cannot understand context, the risk of false positives increases astronomically when everything the student is doing on the device is subject to scanning and flagging. Imagine a student reading an ebook late at night on their school device while at home, and the book happens to contain something about drugs, bullying, violence, etc, and now suddenly, that student’s device gets flagged and now they’re having to explain themself to a school administrator.
Along with this, students in oppressive and controlling home environments may have similar struggles. Oppressive parents often place heavy device/internet monitoring and restricting programs onto their children’s personal devices, making it so students would face the same problem, of their entire private device use being scanned and reported to an authority. Because of the fact that parents who institute heavy digital restrictions are likely to be emotionally abusive in other ways that wouldn’t technically be considered child abuse, it is extremely important that they have private devices in order to talk to peers about sensitive topics, and vent about their struggles. School devices can allow youth in these situations to access beneficial online communities and connect with their friends in order to get support with their struggles. However, if a school device’s AI monitoring system was scanning all chat messages that student was sending and all online communities the student was a part of, and found any information about their mental struggles, it would most likely be flagged, once again subjecting the student to unnecessary intervention.
I have seen these types of scenarios play out first-hand, with people I cared about growing up. In high school, I had this online friend who I met using social media messaging platforms. I quickly became very close to her, and learned that she was dealing with oppressive, strict, controlling and abusive parents that didn’t want her communicating with individuals over the internet. Living with these parents gave her immense mental health struggles, which ergo caused her to have trouble making friends in-person. Making online friends became her main way to connect with others and have a strong support system to deal with her struggles. Because of her parents’ overbearing control, she had no way of accessing her online communities on her personal device, and therefore relied on her school computer to be able to talk to us. Luckily, the device was not heavily monitored with AI software. But now, in modern schooling where AI surveillance is prevalent, she wouldn’t have this same luxury. Her messages of venting to her friends would be scanned, sent to school administrators, and unnecessary intervention would immediately be triggered. This would likely lead to her parents being informed of the situation, her receiving punishment for communicating with others online, and subsequently, her being cut-off from ever being able to contact her online support system again. Given the fact that cutting off vulnerable young people from online communities could devastate their mental health and potentially contribute to their suicide, the school’s use of AI survilence here would be directly leading to a horrible outcome for this student.
Because of these types of scenarios, it is instrumentally important for young people in struggling situations—either because of oppressive parents, being in a low-income home—to be able to access school devices as their personal devices for private digital usage. This becomes impossible when every click on the device is being scanned for potentially flaggable information, and funneled directly to school authorities which are likely to mishandle a situation or even unjustly punish the young person in question.
How Forced Disclosures of Student Struggles by Schools Can be More Harmful than Beneficial
I’ve mentioned a few times that the intervention from schools would be unnecessary and expose the students to harmful consequences. Some people would argue against this, saying that if a student is struggling with challenges in their life, violating their privacy in order to intervene and get them help is always justified. However, this is incorrect, because oftentimes, forced disclosures of student struggles—espcially mental health struggles—from schools lead to worse outcomes for the student.
For example, The Washington Post documented cases in which schools invoked Florida’s Baker Act after children made statements about harming themselves or others. One involved a boy with autism who, after being teased at school, said that he wanted to hurt himself. Police became involved and he was involuntarily taken for psychiatric evaluation. According to his family, he was handcuffed and taken to a psychiatric facility, where the 10-year-old refused to eat or shower and was housed with patients as old as 17. His family obtained an attorney and secured his release the following day. His father told the Post that the experience did not end when he was released: the child subsequently experienced nightmares for two years, became more anxious, and remained afraid that saying the wrong thing could cause authorities to detain him again.
This story shows how the intervention from the school—triggered because of concerns for a student’s mental health—actually ended up putting the student in a worse environment. Another young woman described beginning to self-harm around age 12 or 13 while experiencing depression and suicidal thoughts. Friends eventually contacted her school’s head of year out of concern. The school official tried to help but informed her parents without her permission. She said that learning this had happened made her angrier, more anxious and did not help her with her struggles.
This brings up an interesting element to this as well—school interventions and disclosures regarding student mental health struggles often involve one key aspect: informing the parents. Now, in a perfect world, where all parents have their children’s best interests at heart, this wouldn’t be an issue. All parents should theoretically care about supporting their child and working towards their best interests, including getting them the help they need to fight their mental health concerns. However, this is not the case. In fact, oftentimes the mental health struggles that youth are dealing with are a direct result of the toxic home environments they are living in. Along with this, many oppressive and abusive parents react harshly to their child’s mental health struggles, and opt to punish them instead of supporting them.
A particular 2024 study in the Journal of Clinical Psychology surveyed 1,495 adolescents with histories of suicidal thoughts, suicide attempts, or nonsuicidal self-injury who had disclosed these issues in therapy. Researchers specifically asked what happened when therapists informed their parents. A substantial number described reactions that they perceived as harmful or punitive.
One participant said their mother subsequently made fun of them for their self-harm. Another reported being told they were “dramatic and trying to get attention.” Another said their parents became angry because the teenager had leaned on friends for emotional support. The study’s authors specifically noted that restrictions imposed after disclosure were often described negatively and experienced by adolescents as punitive or unhelpful.
Among adolescents describing their worst disclosure experience, nearly 73% of those whose parents had previously been unaware said at the time that telling their parents had not been the right decision, and roughly half still felt that way when surveyed later. Some explicitly said the experience caused them to conceal subsequent self-harm.
This claim is backed up by first-hand stories as well. According to a first-person account published by The Guardian, a young woman describing her adolescence said that when she began self-harming, her mother responded by taking away the hobby that had been one of her outlets, screaming at her during a panic attack, and slapping her after discovering that she was self-harming. She was writing because she saw a similar dynamic occurring with her 16-year-old cousin. According to the story, the cousin was struggling severely, saying that she hated her life and no longer enjoyed anything. Rather than interpreting her behavior as possible depression, the girl’s parents reportedly treated her exhaustion and withdrawal as laziness, prohibited activities she enjoyed such as video games, and resisted getting her mental-health treatment.
These studies and stories show how in many situations, a parent learning about the mental health struggles of their children will lead to a direct negative outcome, including punishment and emotional abuse by their parents. These types of situations are oftentimes why youth are so hesitant to inform their parents of their struggles in the first place. While in some situations, children informing their parents of these issues will lead to them getting increased access to support, this is not always the case, and the school should not make the decision for them. The fact of the matter is… the child in question knows their own home life and knows their parents better than any school administrator does. If a young person doesn’t want their parents to be informed of their struggles, then it is often for a good reason, most likely to avoid situations like previously described, because they know their parents will react harshly. So young people with strict and emotionally abusive parents should not be forcefully outed by schools, just because schools used AI to scan their devices and discover their issues.
How AI Surveillance in Schools Could be Used to Harm, Target and Profile LGBTQ Youth
AI surveillance systems aren’t just risky when it comes to false positives and forcing unnecessary disclosures that could harm students. AI surveillance in schools also builds a powerful surveillance framework that opens the doors for it to be used in much more malicious ways than just for the perceived purpose of protecting students. One of the most vulnerable groups of young people that are often targets of unfair discrimination is LGBTQ youth. In order to understand the risks that AI surveillance poses to them, it is important to understand the rising stances against LGBTQ youth in our government.
Many mainstream politicians in positions of authority believe that LGBTQ and gender ideology is inherently harmful to youth, and that young people should never be exposed to it. Florida Gov. Ron DeSantis has repeatedly framed LGBTQ related school content as harmful or inappropriate for children. His office described a 2023 bill package as protecting children from “permanent mutilating surgical procedures, gender identity politics in schools, and attending sexually explicit adult performances.” In the same release, Florida’s education commissioner praised the laws as keeping schools focused on education “not indoctrination,” and the state’s healthcare agency secretary said Florida was protecting children from “harmful drugs and surgeries.” CBS also reported DeSantis saying that “elementary school kids should not have woke gender ideology injected into the curriculum,” calling it “inappropriate.”
Because of beliefs like this that are becoming more prevalent among red state politicians, we’ve seen a rise in policies that directly target and harm LGBTQ students in public schools. Texas is one of the most direct examples of this. Texas’ “Parents’ Bill of Rights” banned all LGBTQ aligning student clubs, and prohibited teachers from teaching about LGBTQ material or discussing LGBTQ topics with students. Along with this, several states have introduced Parental Notification laws within schools. These laws often legally mandate teachers and school administrators to notify parents if a student requests to go by any name or pronouns that do not align with their biological gender. Some harsher versions of these laws make it so teachers are only allowed to refer to students by these chosen names/pronouns if their parents approve. These laws are inherently discriminatory because they enforce the idea that young people are not allowed to express themselves outside of their parents’ wishes, and that parents should have default knowledge of private information about students’ identity. Informing parents of their children’s LGBTQ identities without the students’ consent is harmful in the same way that informing their parents of their mental struggles can be.
The fact of the matter is that parents are not perfect, and homophobic parents often discriminate against their children’s LGBTQ identity in many ways. Surveys show that over 57.4% of LGBTQ+ youth have experienced at least one form of parental rejection after sharing their preferred gender or sexual orientation. These students report higher levels of emotional distress compared to other students. Along with this, parents may become abusive towards their children after learning that they have an LGBTQ identity. Parents have been known to react with kicking children out of their homes, extreme isolation, severe beatings, forcible institution into “Conversion Therapy”, and even torture and murder of their queer children. This behavior has been known to cause devestating consequences on the mental health of youth, and has even resulted in the suicide of queer children.
While these laws are definitely a violation of student privacy, they also set a dangerous precedent for the profiling and forced outing of LGBTQ youth. Parental notification laws are showing that state governments can get away with enforcing laws that forcefully inform parents of students’ LGBTQ identity. So if they can get away with it in one way, it is reasonable to assume that they will try to increase the ways they can enforce these types of policies. AI surveillance could play a dangerous role in this profiling. If a school is already using a robust AI surveillance system to automatically scan all students’ devices for keywords to be flagged, it makes it incredibly easy to use the same AI framework for even more harmful reasons. The AI surveillance system could potentially be set to scan for any information on a students’ device related to them having an LGBTQ identity, or not conforming to their biological gender. This information could then be flagged, sent to school administrators, and then given to parents, all without the consent of the student.
If red state politicians already have parental notification laws, it is not unreasonable to assume that they will expand them to require school AI surveillance systems to scan for LGBTQ identity information, and then legally require school administrators to turn over this information to parents. While this may seem like a stretch, with the amount of discriminatory policies being made specifically against LGBTQ youth, and the rise of AI surveillance being used by both our government and our schools, this remains a scary possibility. LGBTQ youth could be profiled, forcefully outed, and subsequently face parental abuse, all because schools got too comfortable using AI surveillance of students.
Even if there aren’t any widespread laws requiring AI surveillance to be used for forced disclosure of students’ LGBTQ identity, there are still many schools that could use AI surveillance specifically to profile and punish queer youth. Many religious schools in the US have much less government oversight than public schools, and can therefore get away with very strict policies regarding morality, behavior, and conformity to religious standards. This has led many religious schools to punish or even expel students for discovering they were gay, or identified with a gender outside of their biological sex.
For example, James Barnett was a senior at Trinity Christian Academy in Texas when administrators discovered that he operated a website intended to provide support and community for LGBTQ+ young people. Barnett said officials initially suggested that he describe himself as confused, remove the website, and attend counseling. When he refused, the school expelled him and disclosed his sexuality to his parents without his consent. The consequences extended beyond school: Barnett said the resulting conflict threatened his housing and his parents’ willingness to support his college education. What began as school discipline over private online expression therefore expanded into a serious violation of privacy with repercussions throughout his family life.
This, and many other similar stories like this illustrate how in private and religious schools, students identifying as LGBTQ can not just lead to forced disclosures, but actual punishment from the school. This is where AI surveillance once again becomes incredibly dangerous, because it allows these strict religious schools to scan student devices for any hint that they may be LGBTQ, or may simply be engaging in any behavior that the schools deems problematic or against the religion. Students deserve basic privacy, and should not face punishment from schools simply for their identity.
The fact of the matter remains: AI surveillance of students, and all digital surveillance over students is not safe. This surveillance runs a dangerous risk of false positives, leading to unnecessary punishment and school intervention. Along with this, many students need to rely on school devices for personal and private reasons, where their usage of them should not be subject to constant monitoring, as it puts them at risk of mistreatment from their parents due to forced disclosures. Finally, mass AI surveillance of students builds a framework that could be used by politicians to profile vulnerable groups of young people, which will only lead to their harm. Surveillance of students is not okay, and we should stop pretending like it is.
The National Youth Rights Association
If you’re interested in Youth Rights, consider volunteering with us. We are always looking for new members and would love to have you on board. If you have a personal story to share about how school surveillance has negatively impacted your life, or about a general youth rights violation, consider sending us an email at nyra@youthrights.org. We’d love to help get your story out to the world.
The text of How AI Surveillance in Schools Violates Student Rights & Threatens Vulnerable Youth’s Safety © 2026 by Zane Miller is licensed under CC BY-SA 4.0.
