News: 1628200807

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

Apple is about to start scanning iPhone users' devices for banned content, warns professor

(2021/08/06)


Apple is about to announce a new technology for scanning individual users' iPhones for banned content. While it will be billed as a tool for detecting child abuse imagery, its potential for misuse is vast based on details entering the public domain.

The neural network-based tool will scan individual users' iDevices for child sexual abuse material (CSAM), respected cryptography professor Matthew Green told The Register today.

Rather than using age-old hash-matching technology, however, Apple's new tool – due to be announced today along with a technical whitepaper, we are told – will use machine learning techniques to identify images of abused children.

[1]

"What I know is that it involves a new 'neural matching function' and this will be trained on [the US National Centre for Missing and Exploited Children]'s corpus of child sexual abuse images. So I was incorrect in saying that it's a hash function. It's much more powerful," said Green, who [2]tweeted at length about the new initiative overnight.

[3]

[4]

"I don't know exactly what the neural network does: can it find entirely new content that "looks" like sexual abuse material, or just recognize exact matches?" the US Johns Hopkins University academic told El Reg .

Indiscriminately scanning end-user devices for CSAM is a new step in the ongoing global fight against this type of criminal content. In the UK the Internet Watch Foundation's [5]hash list of prohibited content is shared with ISPs who then block the material at source. Using machine learning to intrusively scan end user devices is new, however – and may shake public confidence in Apple's privacy-focused marketing.

[6]

Apple [7]infamously refuses to talk to The Register , so asking it to comment on this is a fruitless exercise. Doubtless Cupertino will point to its scanning of ( [8]deliberately ) unencrypted iCloud backups as precedent for this, saying it's just an incremental step in the ongoing fight against the true evil of child sexual exploitation. Nonetheless, we've asked the fruity firm to comment and faithfully promise here to reproduce their response for the delight and delectation of El Reg 's readership.

Governments in the West and authoritarion regions alike will be delighted by this initiative, Green feared. What's to stop China (or some other censorious regime such as Russia or the UK) from feeding images of wanted fugitives into this technology and [9]using that to physically locate them ?

[10]Upcoming Android privacy changes include ability to blank advertising ID, and 'safety section' in Play store

[11]Apple scrambles to quash iOS app sideloading demands with 'think of the children' defense

[12]Apple, you've AirDrop'd the ball: Academics detail ways to leak contact info of nearby iThings for spear-phishing

[13]Brit cybercops issue tender to rip and replace their formerly flaw-ridden CyberAlarm tool

"That is the horror scenario of this technology," said Green. "Apple is the only service that still operates a major E2EE service in China, in iMessage. With this technology public, will China demand that Apple add scanning capability to iMessage? I don't know. But I'm sure a lot more worried about it than I was two days ago."

According to Green, who said he had spoken to people who had been briefed about the scheme, the scanning tech will be implemented in a "two party" design. As he explained it: "Apple will hold the unencrypted database of photos (really the training data for the neural matching function) and your phone will hold the photos themselves. The two will communicate to scan the photos on your phone. Alerts will be sent to Apple if *multiple* photos in your library match, it can't just be a single one."

The privacy-busting scanning tech will be deployed against America-based iThing users first, with the idea being to gradually expand it around the world as time passes. Green said it would be initially deployed against photos backed up in iCloud before expanding to full handset scanning.

[14]

If this is the future of using Apple devices, it might not only be sex offenders who question Apple's previously-stated commitment to protecting user privacy. ®

Get our [15]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YQyz9ziGhmPLFCf@37TmpQAAAIE&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://twitter.com/matthew_d_green/status/1423071186616000513

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YQyz9ziGhmPLFCf@37TmpQAAAIE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YQyz9ziGhmPLFCf@37TmpQAAAIE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[5] https://www.theregister.com/2015/08/11/iwf_child_abuse_hash_list_released/

[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YQyz9ziGhmPLFCf@37TmpQAAAIE&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[7] https://www.theregister.com/2016/09/07/reg_effort_to_attend_iphone_7_launch/

[8] https://www.theregister.com/2020/01/21/apple_encrypt_icloud_backups/

[9] https://www.propublica.org/article/operation-fox-hunt-how-china-exports-repression-using-a-network-of-spies-hidden-in-plain-sight

[10] https://www.theregister.com/2021/07/29/android_privacy_changes/

[11] https://www.theregister.com/2021/06/23/apple_ios_sideloading_apps_congress/

[12] https://www.theregister.com/2021/04/22/airdrop_contact_leaks/

[13] https://www.theregister.com/2021/03/10/police_cyberalarm_pervade_software_new_tender/

[14] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_security/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YQyz9ziGhmPLFCf@37TmpQAAAIE&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[15] https://whitepapers.theregister.com/



Blackjack

If this doesn't make people stop using Iphones... then what?

gandalfcn

Given all others follow what Apple does, and always have done that is rather a silly thing to say.

Blackjack

Apple: We care about your privacy that's why we made Facebook and all those people whose revenue depends on tracking you angry.

Also

Apple: We will scan your photos, but is to look for banned content like people that abuse children. Do not worry, there is absolutely no way your photos that do not have anything that's banned content will be leaked and used against you.

Two possible approaches

doublelayer

There are two methods this could take:

1. A model is created from the photos on Apple's end and the phone uploads its pictures to a server at Apple to do the comparison. This involves a mandatory leak of data which a user can't disable and, as Apple doesn't own the devices themselves, is currently illegal.

2. A model is created by Apple and sent to user devices, which scans the pictures onboard and sends the result to Apple. This is more likely to be legally viable, but it is going to cause a lot of problems as the processors in a mobile device are a lot weaker than a server and most models for picture comparison are likely to be large. There will at a minimum be complaints about the network bandwidth and CPU time needed to run this check, especially as I assume the model will get run every time a user takes new pictures and whenever new source material is added causing a model update. In addition, they are either going to have a lot of false positives or false negatives, meaning they'll need some method of determining whether an image is a false positive. Automatic uploads are still legally questionable, so this might result in a lot of suspicious reports which can't be verified. With the alternative that it is mostly useless though, I don't know whether they will accept a high false negative rate.

Re: Two possible approaches

elsergiovolador

They are just going to change the license where somewhere in small print you'll consent to that.

If you decline, then your phone will stop working.

They are getting away with not allowing other app stores to run on iPhone, I can't see why they wouldn't be able to do this.

They'll give a wink wink to three letter agencies about them being able to scan phones to their heart contents and it will be more legal than scrambled eggs on Sunday morning.

Kudos to Gareth

Steve Aubrey

Subtle, but present, humor. Sly, even.

Apple won't talk to El Reg, "so asking it to comment on this is a fruitless exercise"

A stalking-horse for copyright protection

Peter Prof Fox

I have a copy (of uncertain provenance) of that iconic 1977s session by the Five-aside-archers with Beetle Wulvis on Bass Guitar. Sony Music just happen to have signed a hoover-up contract with somebody who claims to have some derived title to the tracks. In England they might sue me under copyright law but if it wasn't currently for sale they can't show a loss so that's two-fingers to them. (They have to demonstrate a loss.) But with other (c) enforcement regimes there are other consequences. Furthermore, having and playing (in England) is neither a crime or a civil tort unless they can prove I'm not entitled. (For example my copy came from Big Joey Frobisher himself.) But in this new world YOU HAVE COPYRIGHTED MATERIAL ON YOUR DEVICE! GO TO JAIL. DO NOT PASS GO. is the default position these mega corporations expect us to accept.

So Apple have solved the problem of what to do with that pile of cash?

Ian Mason

That'll be handing it over to all the people who sue them for this. If the police in most civilised countries need a warrant to search your possessions for unlawful material, what authority do Apple claim for this gross abuse of civil liberties? What theory of law do they have that they think they have carte blanche to start searching through people's phones?

What do their marketing department think about all that money that they've wasted touting Apple's privacy credentials now that another part of Apple has just completely trashed those credentials overnight.

Really Apple? With all your shouting about privacy I really expected better from you.

Anyone who says "Think of the kiddies" and there's no doubt going to be some here: It's always used as the excuse for inserting the thin end of the wedge, and then at the first excuse whacking the other end with a bloody great mallet. A recitation of the evils that follow bending the rules of civil hygiene for some "special case" ought not to be necessary. But for those thinking "Well, it's only going to affect paedophiles" anyone with one jot of sense knows that if this is permitted then there will be another "good and worthy" case permitted, then another less worthy until it trickles down to the point where your phone's camera will feed the onboard AI, which will note the double yellow line you just parked on and immediately debit your bank account the parking fine and put points on your digital driving license.

Won't Someone Think of the Children?

Boo Radley

It's nearly always the go to excuse for something that eventually ends up taking away people's rights.

Re: Won't Someone Think of the Children?

David 132

"We're only using it to scan for images of kiddie porn. Are you on the side of the kiddie fiddlers?"

"We're only using it to scan for images glorifying terrorism. Are you a terrorist?"

"We're only using it to scan for images of serious crimes. Why do you sympathize with murderers?"

"We're only using it to scan for images of banned ideologies. What, are you a neo-Nazi?"

"We're only using it to scan for images of racist or TERF nature. Such thought is wrong and is banned."

"We're only using it to scan for images showing support for Antifa."

"We're only using it here in China to scan for images showing you support the Hong Kong democracy movement."

"We're only using it here in Spain to scan for Catalan separatist imagery..."

I won't stoop to repeating the old Pastor Martin Niemöller quote cliché, but there's a definite slippery slope argument to be made here.

Don't use your iPhone in church

Anonymous Coward

You might accidentally take a picture of a couple getting married, but end up with a few baby angels in the background which Apple's AI might class as naked children and get you prosecuted.

Re: Don't use your iPhone in church

elsergiovolador

I think churches will be whitelisted... ;-)

Re: Don't use your iPhone in church

Clausewitz 4.0

The Vatican will move against this tech

Re: Don't use your iPhone in church

gandalfcn

Only the brainwashed go to churches by choice.

People went to digital photography to get AWAY from this

billdehaan

Back in the 1990s and early 2000s, there was a " think of the children " panic in Canada, and crusaders went on the tear to get the police and government to "do something" to stop it.

In the middle of this climate, I know of three cases where people ended up getting visited by police investigating them for alleged child pornography.

One case was a Japanese anime, as in, a cartoon, with no actual humans being filmed, let alone children.

The other two were the result of photo development. Those old enough to remember actual film cameras know that unless you had a darkroom, chemicals, and skill, you needed to go to a photo developer to convert your raw film into actual snapshots. Camera stores did it, of course, as well as specialty outlets like Fotomat, but one of the most common photo development places was, oddly enough, the pharmacy. And it was pharmacies that called the cops on two people getting their photos developed.

The first case showed the shocking picture of a nude 5 year old boy with his pants on the sidewalk with a scantily clad 3 year old girl next to him. In other words, a 3 year old girl snuck up on her big brother and pants his swimsuit on him. Mom happened to be taking pictures of her kids in the pool, and couldn't resist getting a snap of her kids pranking each other.

The second case was similar, with a grown woman in a bathtub with a 2 year old boy, who decided to make an obscene gesture to shock his mommy just as Daddy walked in. In other words, a typical " Jim, get in here and see what your son is doing " family moment.

Fortunately, in both cases, the police officers were parents themselves and not idiots, and when they visited the families and saw that the kids photographed were the children of the photographers, they realized that the photo developers had completely over reacted. But as you can imagine, those families stopped sending their film out to be developed, and went to digital photography.

Now, you don't even have to drop your film off to have busybodies report you to the cops, your camera vendor will do it as soon as you take your picture.

There's no way that this won't be abused, both by companies, and governments.

Re: People went to digital photography to get AWAY from this

JimboSmith

Not forgetting Julia Somerville and her ordeal. [1]https://en.wikipedia.org/wiki/Julia_Somerville#Allegations

[1] https://en.wikipedia.org/wiki/Julia_Somerville#Allegations

It doesn't surprise me

HildyJ

Apple has a history of kowtowing to various country's demands for suppression of or access to data.

This seems like an attempt to avoid a US mandated backdoor with a "limited" front door.

I wonder if the initial research was done to help China monitor their iPhone users.

Re: It doesn't surprise me

Mark 65

In great need of another OS that can run on Apple hardware or great hardware with another OS. No, that simply isn’t Android.

Re: It doesn't surprise me

Tom Chiverton 1

They stood, rightly, firm before.

What's changed for them?

Re: It doesn't surprise me

gandalfcn

:"Apple has a history of kowtowing to various country's demands for suppression of or access to data.:" As do most. Your point being?

karlkarl

It is basically Apple's clone of Microsoft Defender at this point. It only scans for DRM cracks etc. Doesn't give a damn about viruses.

It is called Microsoft Defender because it defends Microsoft's stakeholders. Not you. Apple is merely copying.

Phil Kingston

"Apple will hold the unencrypted database of photos"

That sound dodgy to anyone else? Are they getting some sort of special protection from law enforcement agencies who would otherwise be all over someone holding those images?

gandalfcn

:"That sound dodgy to anyone else?" Only to the usual Apple haters and the terminally paranoid.

Bandwidth!

Chris Gray 1

I don't own any Apple devices (surprisingly, I'm OK with the walled garden, but can't afford them and since I run Linux, which has poor support for getting images out...), and it now looks like I never will.

My current cell-phone is an 8-year old Samsung S4, and my data plan is tiny by most standards. Any attempt by Google to do this sort of thing will result in me going back to a "feature phone", for purely financial reasons.

Double Plus Good

Anonymous Coward

Right Think is about to be Enforced

So the 1984 ad was their mission statement?

sreynolds

It always beings with the "best intentions". Do it for the kiddies and such. Obviously, if you are against this, then you must be a kiddie fiddler - or something like that goes the usual argument. It's the thin end of the wedge.

A new iPhone meaning for "jailbreak

Kevin McMurtrie

Take 50 photos of family kids playing. Find that they're all blurry and delete them. Get arrested by FBI because AI found illegal content and you deleted both their evidence and your defense.

uncle grumpy

After fighting off Ring and Alexa, my hardware has risen against me. Will there be any place to migrate too? Off course not, the dominoes will fall in rapid succession. Bill Barr must jumping for joy. Imagine the zeal the next regime will use, although I’m sure the current regime is supportive as well. I’m so pissed at the betrayal of these weasels I can barely see straight.

Solution

Clausewitz 4.0

Just like what Signal Private Messenger did with "aesthetic" messages for Israeli-Cellebrite (this chapter isn't finished, expect more news), the solution here is also simple.

Phone-Devs can embed hundreds of digitally-created-naked-children-fake-photos (not real ones) into files not viewed by the user, including fake geotags like for example FBI offices, Apple offices, or even the Pentagon.

I do not endorse adult games with children, but this tech must go.

A small country has fewer people.
Though there are machines that can work ten to a hundred times faster than man, they are not needed.
The people take death seriously and do not travel far.
Though they have boats and carriages, no one uses them.
Though they have armor and weapons, no one displays them.
Men return to the knotting of rope in place of writing.
Their food is plain and good, their clothes fine but simple, their homes secure;
They are happy in their ways.
Though they live within sight of their neighbors,
And crowing cocks and barking dogs are heard across the way,
Yet they leave each other in peace while they grow old and die.