Students Have Been Called to the Office - Or Arrested - for False Alarms from AI-Powered Surveillance Systems (apnews.com)
- Reference: 0178622344
- News link: https://news.slashdot.org/story/25/08/09/0624254/students-have-been-called-to-the-office---or-arrested---for-false-alarms-from-ai-powered-surveillance-systems
- Source link: https://apnews.com/article/ai-school-surveillance-gaggle-goguardian-bark-8c531cde8f9aee0b1ef06cfce109724a
But when the school's surveillance software spotted that joke, "Before the morning was even over, the Tennessee eighth grader was under arrest. She was interrogated, strip-searched and spent the night in a jail cell, her mother says." Her parents filed [2]a lawsuit against the school system, according to the article (which points out the girl wasn't allowed to talk to her parents until the next day). "A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl."
> Gaggle's CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. "I wish that was treated as a teachable moment, not a law enforcement moment," said Patterson.
But that's just one example, the article points out. "Surveillance systems in American schools increasingly monitor everything students write on school accounts and devices."
> [3]Thousands of school districts across the country use software like Gaggle and Lightspeed Alert to track kids' online activities, looking for signs they might hurt themselves or others. With the help of [4]artificial intelligence , technology can dip into online conversations and immediately notify both school officials and law enforcement... In a country weary of school shootings, several states have taken a harder line on threats to schools. Among them is Tennessee, which passed a 2023 zero-tolerance law requiring any threat of mass violence against a school to be reported immediately to law enforcement....
>
> Students who think they are chatting privately among friends often do not realize they are under constant surveillance, said Shahar Pasch, an education lawyer in Florida. One teenage girl she represented made a joke about school shootings on a private Snapchat story. Snapchat's automated detection software picked up the comment, the company alerted the FBI, and the girl was arrested on school grounds within hours... The technology can also involve law enforcement in responses to mental health crises. In Florida's Polk County Schools, a district of more than 100,000 students, the school safety program received nearly 500 Gaggle alerts over four years, officers said in public Board of Education meetings. This led to 72 involuntary hospitalization cases under the Baker Act, a state law that allows authorities to require mental health evaluations for people against their will if they pose a risk to themselves or others...
>
> Information that could allow schools to assess the software's effectiveness, such as the rate of false alerts, is closely held by technology companies and unavailable publicly unless schools track the data themselves. Students [5]in one photography class were called to the principal's office over concerns Gaggle had detected nudity. The photos had been automatically deleted from the students' Google Drives, but students who had backups of the flagged images on their own devices showed it was a false alarm. District officials said they later adjusted the software's settings to reduce false alerts. Natasha Torkzaban, who graduated in 2024, said she was flagged for editing a friend's college essay because it had the words "mental health...."
>
> School officials have said they take concerns about Gaggle seriously, but also say the technology has detected dozens of imminent threats of suicide or violence. "Sometimes you have to look at the trade for the greater good," said Board of Education member Anne Costello in a July 2024 board meeting.
[1] https://apnews.com/article/ai-school-surveillance-gaggle-goguardian-bark-8c531cde8f9aee0b1ef06cfce109724a
[2] https://storage.courtlistener.com/recap/gov.uscourts.tnmd.99558/gov.uscourts.tnmd.99558.14.0.pdf
[3] https://apnews.com/article/ai-school-chromebook-surveillance-gaggle-investigation-takeaways-381fa82978f27eb85f20d03236820711
[4] https://apnews.com/article/ai-school-chromebook-surveillance-gaggle-investigation-takeaways-381fa82978f27eb85f20d03236820711
[5] https://lhsbudget.com/news/2024/02/09/art-students-push-back-against-potential-gaggle-censorship/
So what's the license say? (Score:2, Insightful)
> Gaggle's CEO, Jeff Patterson, said in an interview that the school system did not use Gaggle the way it is intended. The purpose is to find early warning signs and intervene before problems escalate to law enforcement, he said. "I wish that was treated as a teachable moment, not a law enforcement moment," said Patterson.
If the license doesn't prohibit leaping to law enforcement when the alleged purpose is to "intervene before problems escalate to law enforcement" then fuck you Jeff Patterson, you lying fuck.
Re: (Score:2)
Future Crime Unit
Everybody Runs
2023
Normal for Florida
but who am I kidding, not myself, so... normal for everywhere
I feel like I'm the only one puzzled that no one else seems to know or care when I bring it up with chumps
(chumps: technical term for "regular" people)
This is a symptom (Score:3)
Of the numerous school shootings and legions of hungry lawyers. The school would rather see a child arrested than be the site of a mass shooting or lawsuit.
Re: (Score:2)
You don't have to build a new school every time there's a shooting, that's just done to protect the guilty.
Re: (Score:2)
> The school would rather see a child arrested than be the site of a mass shooting or lawsuit.
Setting aside your "or lawsuit", would you rather have a school be the site of a mass shooting than see a child arrested?
And this is the fucking cause. (Score:1)
> Of the numerous school shootings and legions of hungry lawyers. The school would rather see a child arrested than be the site of a mass shooting or lawsuit.
If that’s a symptom, then the fact that damn near every single school shooter was under the influence of prescribed psychotropic drugs is the fucking cause.
50 years ago a high school senior in rural America would drive his truck onto a school campus with a shotgun hanging in the rear window. A school teacher smoking outside might stop the student on the way to class and say “have any luck in the duck blind this weekend?”
No SWAT teams called. No child arrested under the Baker Act to be p
The Hellmouth is active as ever (Score:2)
This is the first of several rightly famous Jon Katz pieces:
[1]https://news.slashdot.org/stor... [slashdot.org]
[1] https://news.slashdot.org/story/99/04/25/1438249/voices-from-the-hellmouth
Re: (Score:2)
How about... maybe the kids don't need to be on social media? The kid doesn't need a $500 cell phone, and a tablet, and a computer in their bedroom, and a TV, and a PS5.
Back in the day, we went to eachother's house or picked up the wall-mounted phone if we wanted to talk to a friend.
Where were the parents? Why did the kid have unsupervised (or no parental controls) social media time? Does a public school need 'surveillance software' to monitor a student... isn't that kind of a parental thing?
greater good (Score:3, Insightful)
> Sometimes you have to look at the trade for the greater good
a 13-year-old girl ... was interrogated, strip-searched and spent the night in a jail cell ... A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl ... over an offensive joke
The person that made the greater good statement is an idiot and should be subjected to the same treatment because I find it offensive.
Re: (Score:2)
>> Sometimes you have to look at the trade for the greater good
> a 13-year-old girl ... was interrogated, strip-searched and spent the night in a jail cell ... A court ordered eight weeks of house arrest, a psychological evaluation and 20 days at an alternative school for the girl ... over an offensive joke
> The person that made the greater good statement is an idiot and should be subjected to the same treatment because I find it offensive.
I suspect if they or a family member received that treatment they would not say the same thing. Where I live the parents would lawyer up a lawsuits start flying; so the school district would never pull that stunt. What I don't get is why does teh school have access to their social media accounts? If a school asked for it they's get a big F off it's none of your business. I suspect many kids ahve the 'offical' account and a real account. What happens when a bunch of students decides to monkey wrench the sy
Actually gun problem is easy to solve without bans (Score:1)
> The problem is we don't have a solution to gun control in this country.
Sure we do. Universal background checks for all transfers. Check that included criminal and mental health issues, the latter is currently lacking. As is the universal part. The system already exists, providing near real-time info to dealers (a YES/NO to proceed with transfer). Then have required safety training, as we do with hunting licenses. Again, the system exists, state approved instructors and curriculum exist, hunter safety training is mostly general firearms safety training. Drop the hunting bit and
Re: (Score:3)
> Sure we do. Universal background checks for all transfers. Check that included criminal and mental health issues, the latter is currently lacking. As is the universal part. The system already exists, providing near real-time info to dealers (a YES/NO to proceed with transfer). Then have required safety training, as we do with hunting licenses. Again, the system exists, state approved instructors and curriculum exist, hunter safety training is mostly general firearms safety training. Drop the hunting bi
Re: (Score:1)
>>> Sure we do. Universal background checks for all transfers. Check that included criminal and mental health issues, the latter is currently lacking. As is the universal part. The system already exists, providing near real-time info to dealers (a YES/NO to proceed with transfer). Then have required safety training, as we do with hunting licenses. Again, the system exists, state approved instructors and curriculum exist, hunter safety training is mostly general firearms safety training. Drop the hunting bit and you have a general purpose class. These classes include safe handling and safe storage.
> We have something politicians call "background checks", but they're not background checks. So the GP was right, and you're using sophistry to pretend he isn't, together with throwing in a whole bunch of stuff that's not actually required in most of the country to get a gun.
You misunderstood. I was not saying we have an implemented solution, I was saying we have a solution on paper. Apologies if I wasn't clear enough. If you read what I wrote you will notice I pointed out problems with the current background check, it's not universal, it's not including mental health.
> NICS, is actually just a poorly maintained blacklist based upon court records. And, contrary to your assertion, it does not include mental health issues
OK, you flat out misread. I said universal and mental health is missing. .
> it cannot contain mental health issues even if you order every psychiatrist in the country to add people who fit a specific criteria to it because that would only kick in if you actually visit a psychiatrist and get a diagnosis.
A voluntary psychiatric visit is not the only path to noting behavior that is potentially a harm to others or oneself.
> \That's what people think about when they say "Background check", not a blacklist.
Nope. That's the sort
Re: (Score:1)
Neither the [1]FBI's assessment [fbi.gov] nor the [2]CDC's assessment [cdc.gov] agree with your recommendations. They have a lot of heavily-researched recommendations filled with rigorous citations and backed by tragic data.
Research over the past 30 years is clear on how to do it: First and foremost, socioeconomic disparity needs to be addressed. Second, availability of mental health care to minors, especially by decoupling it from parental employment, or said differently, universal mental health care for minors at the very leas
[1] https://www.fbi.gov/file-repository/stats-services-publications-school-shooter-school-shooter/view
[2] https://stacks.cdc.gov/view/cdc/127279
bully comeuppance AI software (Score:2)
In some regards I prefer the nail sticks up gets hammered down approach over the squeaky wheel getting grease approach... I'm also pretty sure school bullies are the dumbasses getting pings on this software. The punishment the kid had to go through seems a bit excessive, but my guess is that she was a huge pain in the ass,and I bet her parents are probably just as dysfunctional. If this kind of software makes it easier for schools to deal with very problematic students, hopefully students will behave.
Re:bully comeuppance AI software (Score:4, Interesting)
Just as likely that she was the one being bullied. IDK why you would assume that an incompetent software package being implemented wrongly by a public district would have been correct. This claim that it was all done automatically without any human intervention is also just wrong. There's no way to perform an arrest without the officers involved making a determination of probable cause.
"Hurr durr, computer said you're guilty" isn't a defense.
As expected (Score:3)
This is always the way it goes
Something bad happens. People demand solutions. Poorly thought out and ineffective solutions are implemented, along with a press conference
Innocent people are overwhelmingly targeted while the guilty find workarounds
The problem persists
Panic culture (Score:4, Insightful)
This is the death of freedom in America. We've traded the land of the free and the home of the brave for a place children are arrested, strip-searched, and jailed overnight for making jokes. That's not the disease, it's a symptom. The child faced house arrest and a psychological evaluation not because surveillance tech failed, but because the system treats every minor infraction as a crisis. This isn't about tech companies or their tools. The real problem lies with a culture that encourages schools and states to enforce draconian policies by no means limited to turning private conversations into criminal cases.
Tech companies guard their data on false positives, but that secrecy isn't the root issue; it's the broader erosion of rights under the guise of safety. We've let fear of rare tragedies justify a surveillance state that assumes every child is a suspect, policed by unyielding laws and overzealous administrators. The First Amendment crumbles when words are criminalized without due process. The Fourth Amendment vanishes in strip-searches over bad jokes. This is a deliberate stripping of liberties, using tech as a convenient excuse. The blame falls squarely on the systems and people who wield it recklessly. Liberty won't return until we dismantle these policies, not the tools they hide behind. In fact liberty won't return until we do away with the safety/panic/cancel culture injected by idiots misled by authoritarians.
Re: (Score:2)
Indeed. In civilized nations, a 13 year old cannot even be arrested and making an inappropriate joke cannot get you arrested either. Sounds like an advanced form of surveillance-fascism to me.
Why does Snapchat do this? (Score:2)
Why is Snapchat so dead set to be spyware for authorities? I see no commercial value in it, they can't really use snap content for anything else given their privacy policy. All they are doing is harming their brand, Whatsapp does not get brand damage from having E2EE.
I hate to see... (Score:2)
I would hate to see what my school record would look like if I went to school today. I might even have a criminal record.
Re: (Score:2)
Not every place on this planet does this sort of crap. In fact, most do not.
Sounds like surveillance-fascism (Score:3)
And not the "light" variant anymore, but already an advanced form.
Don't remember it like that. (Score:2, Informative)
> School officials have said they take concerns about Gaggle seriously, but also say the technology has detected dozens of imminent threats of suicide or violence. "Sometimes you have to look at the trade for the greater good," said Board of Education member Anne Costello in a July 2024 board meeting.
So it's better to throw a 1000 innocent people in prison than let 1 guilty one go free? Don't remember that saying like that. On the upside, sounds like a good motto for ICE...
Re: (Score:2)
> So it's better to throw a 1000 innocent people in prison than let 1 guilty one go free?
Pol Pot only said arrest one innocent rather than freeing one guilty. So, this is 1000 times worse.
Re: (Score:2)
Sometimes it really does make sense to sacrifice some privacy for the greater good. And sometimes it doesn't. You have to look at the evidence. What are the real benefits? What are the real harms? The answer isn't always the same.
Oh, the evidence is secret? The companies consider it proprietary information and won't let anyone see it? You'll just have to trust us then. We Know Best.
This story seems fishy (Score:4, Insightful)
Maybe AI played a role. But AI didn't sentence the girl to eight weeks of house arrest. A court did that. Courts don't order house arrest based solely on an AI report, or for an offensive joke.
The story as presented, doesn't pass the smell test.
So, do you kids ... (Score:2)
... still insist on bringing your electronic fidget spinners to school?
Careful (Score:3)
I am a teacher in a school with nice kids compared to others in the neighborhood. It is an old girls school, may have something to do with it. Even at our school kids can be very cruel to each other. Especially online.
If things get out if hand online, we sometimes recommend the parents to press charges with the police since our hands are tied. If it happens outside the school system, we have no authority. Kids usually are smart enough to avoid our platform for these things.
There is a very small percentage of kids that are emotionally neglected by their parents by being spoiled. It once resulted in a lengthy newspaper article about how our school "sent the police to the kid's home for something completely innocent" . (daddy had political connections)
My advice with these things? Whenever a sensational article about school turns up, tread carefully. It may be a very single sided story.
Profit first, child second (Score:2)
Everyone think of the children, until we blame a child for not making 'adult' decisions. I notice, that now it's loud-mouthed schoolgirls supposedly threatening everyone's safety. The amount of spyware pointed at a vulnerable demographic reveals that, rules are there only to make a profit. (Did Gaggle disconnect the schools that abused schoolgirls?) The adults shouting "think of the children" are enabling knee-jerk punishment and more violence against the very children they pretend to protect.
The evid
Forget the AI! (Score:5, Insightful)
Forget the AI angle.
The real story is the response to an "offensive joke"!
Holy shit!!! Every single adult (including the entire school board) involved should lose their job, be fined $25K with the money going in trust for the girl, and prevented from ever having a govt job again.
HOLY FUCKING SHIT!!!
Re: (Score:3)
Truly, "1984" vibes.
Re: (Score:2)
Not vibes, it is exactly like "1984" where people get arrested for what they let slip in front of a Telescreen.
I am fortunate to have been raised in the 70s and 80s, when mass surveillance like this was only a nightmare brought on by works like "1984". There was no surveillance from the parents even; "Do you know where your children are?" was a question my parents rarely answered in the affirmative. Home for supper, home before dark, or a phone call if you're staying at a friend for dinner or a sleepover
The fascism of school admin + teachers union (Score:2)
> Just fascism on the march, no end in sight, too many people approve, and those that don't can't get their shit together to form coherent opposition. Try to keep a safe distance
Well, that's why people are attracted to home schooling, charter schools, and private schools. To get away from the fascism of the merging the government school administrator with the teacher's union. :-)
Re: (Score:1)
The roots of fascism and authoritarianism in general are in the home. From there arises the fascist government and lifestyle, blooming like roses in the springtime. The government is a mirror image of the home.
Wasn't an offensive joke (Score:1, Troll)
She posted kill all the Mexicans and they freaked out because school shootings are so easy to do because firearms are so readily available and we aren't allowed to take them away from people who are irresponsible or mentally ill.
This is about gun control and school shootings not AI but you're not allowed to talk about gun control so here we are.
Re:Wasn't an offensive joke (Score:4, Interesting)
> we aren't allowed to take them [firearms] away from people who are irresponsible or mentally ill.
That is factually incorrect. Those are both situations where firearms can be taken away. As is a temporary mental crisis.
You seem to be conflating the difficulty in establishing a mental illness. Yes, HIPPA laws have an unintended consequence where reporting a person unfit to possess firearms can be difficult. That can be fixed. As criminal convictions are reported to the government for firearms background check purposes, so to could such a determination by a medical professional. Not a description of the illness, just that a person is considered mentally unfit. A person could challenge that before a judge if they wish to. Severe penalties, including loss of medical license, should exist for reports made in bad faith. The current gov't system, which only has the criminal conviction info (would that included mental commitments ordered by a judge?) is available to fun dealers and yields a near-real time YES/NO to the dealer with regard to whether the sale can proceed of they wish to. The system exists, it just needs a way for doctors to contribute.
Re: (Score:2)
Actually, if you've ever been held for a psychiatric evaluation, you name is on a list of people get get special scrutiny if you try to buy a gun. Not sure of the details, but yes, the government does keep track of such things.
Re: (Score:2)
> Actually, if you've ever been held for a psychiatric evaluation, you name is on a list of people get get special scrutiny if you try to buy a gun. Not sure of the details, but yes, the government does keep track of such things.
"held", that sounds like something ordered by a judge. So it's still something initiated by the criminal justice system, not the medical care system. We need the latter to participate too.
Re: Wasn't an offensive joke (Score:2)
Obsessed, much?
Re: (Score:3)
[1]https://www.npr.org/2025/08/07... [npr.org]
see sec 3.2 [2]https://pmc.ncbi.nlm.nih.gov/a... [nih.gov]
[3]https://giffords.org/lawcenter... [giffords.org]
see "State Implementation"... [4]https://www.rand.org/research/... [rand.org]
[5]https://www.newyorker.com/news... [newyorker.com]
the takeaway: many states do enforce such prohibitions. some really, really don't.
[1] https://www.npr.org/2025/08/07/g-s1-81603/nyc-shooter-mental-health-las-vegas-guns
[2] https://pmc.ncbi.nlm.nih.gov/articles/PMC5784421/
[3] https://giffords.org/lawcenter/gun-laws/policy-areas/background-checks/mental-health-reporting/
[4] https://www.rand.org/research/gun-policy/analysis/mental-illness-prohibitions.html
[5] https://www.newyorker.com/news/news-desk/the-sad-reality-of-trying-to-keep-guns-away-from-mentally-ill-people
Re: Wasn't an offensive joke (Score:2)
The folly is in thinking that school shootings are as common in the rest of the world as they are in the US
Re: (Score:1)
This you?
[1]https://theonion.com/no-way-to... [theonion.com]
[1] https://theonion.com/no-way-to-prevent-this-says-only-nation-where-this-r-1848971668/
Re: (Score:3)
The AP article added the context of the joke, which didn't make it funny but subjectively I would agree that it was non-serious. One does not passively say "kill all x" after accused of being one of x in a group chat with friends because she is serious.
A manual review and some minor counseling would have solved this problem, but apparently nobody in a position of leadership understood the point of this software.
Re: Forget the AI! (Score:1)
I guarantee there's a lot more to the story including the kid probably being a bully to other kids and possibly the offensive joke directed to one of their peers?
Re: (Score:1)
So? There are _educational_ measures for dealing with that. Throwing a kid in jail is never acceptable. In the absolute worst case you can lock them in a closed juvenile mental institution, but that requires at the very least a real psych evaluation that clearly says they are a danger.
Re: (Score:3)
> I guarantee there's a lot more to the story including the kid probably being a bully to other kids and possibly the offensive joke directed to one of their peers?
From TFA:
> Earlier in the day, her friends had teased the teen about her tanned complexion and called her “Mexican,” even though she’s not. When a friend asked what she was planning for Thursday, she wrote: “on Thursday we kill all the Mexico’s.”
> Mathis said the comments were “wrong” and “stupid,” but context showed they were not a threat.
Good thing Bender isn't in eighth grade.
Re: (Score:2)
Better yet throw them in jail without an opportunity to speak to their lawyer or friends/family. It's too optimistic to think that these cockroaches will be removed from positions of authority, but we can at least give them a dose of their own medicine so they can see how harmful it is.
Re: (Score:1)
You are to kind. I would add 60 days behind bars for every adult that was involved im making the decision and every adult that could have stopped this but did nothing. There is evil here and it needs to be stopped before it grows even further.
Re: (Score:2)
American schools react pretty strongly when students talk about killing people because it happens quite a bit.
Re: (Score:2)
> The real story is the response to an "offensive joke"!
Response to free speech.
How can one be arrested for speech, intentional or a joke, with full approval from a school and the courts? This case needs to be thrown out with prejudice.
Re: (Score:2)
I think the real story is that high school students are being surveilled, without apologies, explanations, or pushback.
Surveillance it normal.
2 years ago.
Let that sink in.
Re: (Score:2)
> Holy shit!!! Every single adult (including the entire school board) involved should lose their job, be fined $25K with the money going in trust for the girl, and prevented from ever having a govt job again Do you think I will be investigated if I say that instead they should be f***ed with a broomstick, preferably one with a cactus mounted on the top?
Re: (Score:2)
Hey, "Registrations_suck", you should prepare for the cops to come knocking on -your door- after posting that threatening and obscene comment....
Re: (Score:2)
Not just jokes, but all kinds of behaviour. Most people screw up to some degree as children, and it's often just accepted as part of growing up. If going by the letter of the law, it might be a criminal matter, it might warrant severe punishment. That leeway gives children the chance to make mistakes without it ruining their lives. It's even baked into some legal codes, with childhood convictions being forgotten when they reach adulthood.
Things get bad when people start trying to enforce the rules too stric