News: 1681382232

  ARM Give a man a fire and he's warm for a day, but set fire to him and he's warm for the rest of his life (Terry Pratchett, Jingo)

It's time to reveal all recommendation algorithms – by law if necessary

(2023/04/13)


Column As it’s been about forty years since I’ve had pimples, it astounds me that YouTube’s recommendation engine recently served me videos of people with some really severe skin problems - generally on their noses. The preview images themselves are horrifying, and should really come with some sort of content warning. I immediately tell YouTube “I don’t want this” and “never recommend this channel.”

Yet it persists.

How? Why?

[1]

I’ve never watched a dermatology video in all of my years using YouTube, nor do I or anyone I know have an adjacent skin condition that might at some point suggest to the algorithm that I needed to be exposed to these truly can-not-be-unseen video images.

[2]

[3]

The essential nature of a recommendation algorithm is that it’s doing its best to anticipate your desires from whatever bits of data it can gather about you.

I defend myself from arbitrary data collection that fuels the algorithms using PiHole, the tracker-blocking Disconnect plugin, and Firefox, plus a few other tricks. In theory recommendation algorithms therefore have less to work with than if I simply journaled my every activity via some sort of oh-so-friendly-and-rapacious Android smartphone.

[4]

I make an active effort to resist data collection - and perhaps these horrors are one consequence.

At best that’s a guess, because I have no way to know what goes on at the heart of YouTube’s recommendation algorithm. If ever exposed, that closely guarded algorithm could be gamed - in a manner similar to the way so many marketing bottom feeders continually test and game search engine results.

There’s a profound commercial disincentive for YouTube to become transparent about how its algorithm works.

[5]

That leaves me and other privacy conscious folk with just one lever to pull offer - disliking a video and a channel and hoping that - somehow - the algorithm might be able to intuit a generalized case from a specific instance.

In the one example where we’ve been exposed to the inner workings of a recommendation engine - Twitter [6]went public with theirs at the end of March - we got an eyeful of a different kind of horror: an algorithm that promoted owner Elon Musk’s tweets above all others, promoted specific political interests - and that specifically [7]will not promote tweets directly pointing to LGBTIQ+ words, concerns, or themes.

While we can all have a nice laugh at Elon’s profoundly public narcissism, the silencing of an entire community at a moment in history when forces around the world seek to roll back recent gains in civil rights for LGBTIQ+ individuals is not funny. Where it becomes harder to share one’s own story, that story can be framed as marginal, unimportant - even dangerous. It’s a profound ‘othering’ that can, thanks to an algorithm, be greatly amplified.

[8]Once AI can create endless viral videos, good luck switching off social media

[9]Conversational AI tells us what we want to hear – a fib that the Web is reliable and friendly

[10]AI may finally cure us of our data fetish

[11]Voice assistants failed because they serve their makers more than they help users

The solution to both issues is obvious, technically easy, and yet commercially a nearly impossible proposition: open up all recommendation algorithms.

Make them completely transparent, and, for the individual being targeted by the recommendation engine, completely programmable.

I should not only be able to interrogate how I got a horrifying video of a very bad case of pimples, I should be able to get in there and tune things so that the algorithm no longer needs to guess my needs, because I have had the opportunity to make those needs clear.

Every algorithm that recommends things to us - music or movies or podcasts or stories or news reports - should be completely visible. There must be nothing secret behind the scenes, because we know now from countless examples - the biggest and ugliest being [12]Cambridge Analytica - how recommendations can be used to drive us to extremes of belief, emotion – even action.

That’s too much power to leave with an algorithm, and too much control to cede to those who tend those algorithms.

If recommendation algos aren’t shared then we need - by legislation, if necessary - a switch that turns the recommendation engine off.

That might leave us floating in a vast and unknowable sea of content, but it’s better to know you’re nowhere than to be led down a garden path. ®

Get our [13]Tech Resources



[1] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZDgnJQt8x102pkD9WS8dOAAAAFg&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0

[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZDgnJQt8x102pkD9WS8dOAAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZDgnJQt8x102pkD9WS8dOAAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZDgnJQt8x102pkD9WS8dOAAAAFg&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0

[5] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/front&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZDgnJQt8x102pkD9WS8dOAAAAFg&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0

[6] https://www.theregister.com/2023/04/07/twitter_code_cve_substack/

[7] https://www.axios.com/2023/04/03/elon-musks-twitter-free-speech-checkmarks-ranking

[8] https://www.theregister.com/2023/03/08/generative_ai_viral_video/

[9] https://www.theregister.com/2023/02/08/ai_battle_microsoft_google/

[10] https://www.theregister.com/2023/01/18/chatgpt_vs_paperless_office/

[11] https://www.theregister.com/2022/12/14/voice_assistants_failed/

[12] https://www.theregister.com/2018/03/28/cambridge_analytica_not_legitimate_business_says_whistleblower/

[13] https://whitepapers.theregister.com/



You need more than an algo-switch

b0llchit

If recommendation algos aren’t shared then we need - by legislation, if necessary - a switch that turns the recommendation engine off.

And still, this will only solve part of the problem. Your data is still (legally and illegally) collected, shared, sold and analysed.

You must be able to say no to any data collection and analysis for an effective remedy. Actually, no should be the default and an explicit opt-in with warnings for your health and well-being should be mandatory.

Too little grist for the mill?

Anonymous Coward

I’ve formed the opinion (totally without evidence, as suits the Zeitgeist), that you get served this sort of low-rent ad when you’ve been successful in keeping useful targetting information from the algorithm. Do you get women's fast fashion, too?

Re: Too little grist for the mill?

Captain Hogwash

I think you're right. I have no Youtube/Google account and actively resist data collection generally. This means the Youtube algorithm probably has only the VPN provided IP address and current session history on which to base any recommendations. This can produce odd results at the start of a session but recommendations tend towards similar subjects as the session goes on.

Re: Too little grist for the mill?

Mike 137

" I have no Youtube/Google account and actively resist data collection generally. "

Unless you block javascript, all those silent trackers on the web sites you choose to visit provide the data feed for the ad brokers. I recently downloaded a professional security report and found that the download page included an obfuscated script bot called directly from Farcebook. As the page (actually unlawfully) demanded an email address to allow the download, it's quite possible that Farcebook now have my email address.

Once upon a time.....

Anonymous Coward

.....primitive "expert systems" were based on sets of programmatic rules. To a greater or lesser extent, the rules were written in pseudo-English, and could probably be understood by a non-expert.

That was then......now we have neural networks providing the "expertise".......and even the creators of these devices have NO INSIGHT AT ALL to the logic which the neural network uses:

(1) Train the neural network on millions of examples ("big data") of the sorts of things you want the thing to do -- this example is "good", that example is "bad" -- over and over

(2) Note that this good/bad classification is arbitrary and may not even be fact checked

(3) Test the "training" to see if the neural network responds "correctly" to inputs which it has not seen before

(4) Note that this testing uses a volume of data MUCH MUCH smaller than the training database

(5) Put the neural network into production

(6) Note that a neural network CANNOT REPORT ON THE INTERNAL LOGIC USED to come to any conclusion

(7) Worse......if the production system is learning as it goes, even the people running the system have no audit ability on what is "learned"

Quote: "...I should not only be able to interrogate how I got a horrifying video of a very bad case of pimples, I should be able to get in there and tune things...."

Dream on!!!! Even the owners/builders of neural networks cannot do that once the neural network is running!!

Isn't progress a wonderful thing.....at least according to the people pushing this technology!!!

re. It's time to reveal all recommendation algorithms

Anonymous Coward

trouble is, there is no one 'algorithm'. Or rather, the algorithms are such a cluster-fuck of algo-cut-and-paste-mix-and-shake-stolen-copied-misapplied-genius-crap-proprietary-algorithm compilation that nobody's able to make any sense of it. Which is pretty convenient ;)

Re: re. It's time to reveal all recommendation algorithms

Steve Button

Well yeah. Indeed.

Because of blue/green deployments and DevOps practices companies like Google can push out an experiment to a small number of people (only a few million) and see what keeps them engaged. They can do this many times per day, and the algorithm will get tweaked based on what's best (for Google, not what's best for you, duh!)

Therefore "the algorithm" could be substantially different at 9am, 12pm and 3pm even for the same person, and they could be running several different experiments simultaneously.

Cluster-fuck is the word.

My advice, stay off social media and try to avoid the recommended videos on YouTube if you can. It's designed to suck as much of your time as possible.

"trouble is, there is no one 'algorithm"

Jedit

I'm fairly sure that there is exactly one algorithm: the adverts promoted to you are the ones from companies who paid the platform to spam their shit.

I'm in full agreement with the person who said that refusing permissions and hiding data leads to you getting random ads, though. It's hardly a shock - the opt outs on platforms like Facebook straight up say "this won't change the number of ads you see, they just may not be relevant to you". To which all I can say is: if you're willing to pay Facebook to have them show irrelevant ads to me, how desperate are you?

I will never do business with a company that spams me on social media. Even if I want what they're offering, I'll go elsewhere. More people need to make this plain, because they're not going to stop hassling us as long as someone sees their ad and goes "Ooo".

Options

Headley_Grange

"That leaves me and other privacy conscious folk with just one lever to pull "

No it doesn't. You could just stop using it.

With "AI" there is no algorithm

Johnb89

To clarify what some other commenters have said, with "AI", or machine learning, there isn't an algorithm. It really isn't a huge number of if/else statements. Even better, even the people that create/own/manage it have no idea why it does what it does, how it works, or indeed how changing something affects the outcomes.

That doesn't stop modifying the AI's output with things like 'anything Elon mutters is gold', of course.

Well

Julz

Perhaps you could just decide for yourself what you might like to watch. There is a difficulty in knowing what is available but this is true of all big data repositories and is usually solved in some way by knowing what it is your interested in or just browsing. Kinda like libraries or book shops, although you do have to walk past the algorithmically chosen books on display at the front of the shop.

As for inappropriate pimple related suggestions; could I interest you in:

https://www.youtube.com/user/drsandralee

Are these much hyped algorithms really that complicated?

Howard Sway

They seem little more than the following :

1) If you search for / post about thing x, we'll recommend other things that people who also searched for / posted about thing x were interested in, or things that link to thing x.

2) If you search for / post about thing x, we'll recommend other things that advertisers have paid to promote to people who are interested in thing x.

3) What Elon Musk wants promoting (mainly himself, feature exclusive to Twitter).

I've always thought that the reason these algorithms are such closely guarded secrets is not because they are so miraculously clever, but more because they are so embarrassingly simple. It's the sheer volume of linked data that makes them look impressive.

Go with Invidious meanwhile

CommonBloke

Since we're dealing with a pipedream where, even if the algorithm became visible, there's nothing ensuring anyone would have the slightest idea of what's going on, it's better to avoid YT.

Invidious lets you do that. Watch YT videos without being tracked, without being served insufferable ads and, depending on instance, download the video. All video recommendations will be based on whatever you're watching, never on your browsing history (since instances don't keep any). On android phones, you can use the NewPipe to the same effect.

Doctor Syntax

I suppose the American Way would be to sue Youtube for the trauma you suffered.

John Lilburne

Don't let YT (or anything else) give you recommendations. If I use YT I come to it via a specific search query.

none_of_your_business

Maybe I've misinterpreted you but you seem to be a bit upset about being shown pimples. Might I suggest that people suffering from skin conditions have as much right to be represented on the internet and share their experiences of their 'own story' as you put it without framing that experience as 'marginal or unimportant' or to suggest that they are so unpleasant that you should be protected from them. Just because you don't like it doesn't mean you can demand the right to be shielded from it.

I don't like the mass surveillance any more than most people in the know, and whilst I don't really use any social media, I do spend time on YouTube and find it quite good at matching what I'm interested in viewing. At least where it works in my favour I'm happy to tolerate the intrusion, perhaps you should try not interfering with it and it might work better at helping you avoid 'unpleasant skin conditions'.

SundogUK

Idiots like you are why this shit happens.

YouTube’s recommendation algorithm

Pascal Monett

I have another solution : don't use it.

Instead, there is the old-school method. It's called bookmarks. In my browser bar, I have a Youtube folder. All the channels I follow are in that folder. At the bottom of the bookmark bar folder, there is an option "Open All in Tabs" (at least, there is in any proper browser). So, when I feel like exploring my channels, all I have to do is go to my bookmark folder, open everything I have chosen, and wait for the refresh (on a 1Gbps fiber line, it's okay).

YouTube recommendations ? I don't need stinkin' YouTube recommendations. If one of the hosts on a channel I follow recommend that I go look at another channel, then I go check it out.

Bah, humbug.

List was current at time of printing.