If you think AI labs wouldn't stoop to using scraped YouTube subtitles for training, think again
(2024/07/17)
- Reference: 1721180525
- News link: https://www.theregister.co.uk/2024/07/17/youtube_video_subtitles_ai/
- Source link:
Comment FYI: It's not just Reddit posts, books, articles, webpages, code, music, images, and so forth being used by multi-billion-dollar businesses for training neural networks. AI labs have been teaching models using subtitles scraped from at least tens of thousands of YouTube videos, much to the surprise of the footage creators.
Those transcripts were compiled into what is termed the YouTube Subtitles dataset and incorporated into a larger repository of training material called [1]the Pile , nonprofit nu-journo outfit Proof News [2]highlighted this week. The YouTube Subtitles collection contains information from 173,536 YouTube videos including those of channels operated by Harvard University, the BBC, and web-celebs like Jimmy "MrBeast" Donaldson.
The dataset is a 5.7GB slice of Pile, a larger 825GB silo created by nonprofit outfit EleutherAI. The Pile [3]includes data pulled from GitHub, Wikipedia, Ubuntu IRC, Stack Exchange, bio-medical and other scientific papers, internal Enron emails, and many other sources. Overall, the YouTube Subtitles dataset is one of the smallest collections in the Pile.
[4]
Big names such as Apple, Salesforce, and Nvidia have incorporated the Pile, including the video transcripts, into their AI models during training. We're told the makers of those YouTube videos weren't aware this was happening. (There's also nothing stopping tech giants from using YouTube data in other dataset collections; the Pile is just one possible source.)
Not hidden
It wasn't a secret EleutherAI had gathered up subtitles from YouTube videos, as the organization not only made the Pile publicly available, it detailed the thing in a [5]research paper in 2020. The code that scraped the YouTube Subtitles dataset is on [6]GitHub for all to see. The script can be told to pull in subtitles for videos that match certain search terms; in the Pile's case, those terms ranged from things like "quantum chromodynamics" to "flat earth."
The actual videos used to form the dataset aren't mentioned in either the 2020 paper nor on GitHub. Only now are people looking through the training data, since superseded by other collections, identifying the videos that were scraped, and tipping off YouTube creators. An online search tool for inspecting the subtitle training material has been offered [7]here .
[8]
[9]
What's interesting is that Google-owned YouTube's [10]terms of service , today at least, explicitly ban the use of scrapers and other automated systems unless they are public search engines that obey YT's [11]robots.txt rules or have specific permission from YouTube.
The terms also seemingly prohibit the downloading and use of things like subtitles in AI training unless, again, YouTube and applicable rights holders give permission. So on the one hand, there is the potential for the automated scraping of subtitles to be against YouTube's rules, but there's also wiggle room for it to be totally fine. Well, as far as YouTube is concerned; creators feeling their work is being unethically exploited, however legal, by rich companies is another thing.
[12]
It's something that everyone is dancing around. The PR folks at Google – which [13]is itself in the AI game – have simply said, in response to this week's reporting, that the internet giant puts a lot of effort into thwarting unauthorized scraping, and declined to talk about individual organizations' use of its YouTube data.
AI labs that used the Pile to build their models argued they simply incorporated a broad public dataset and that they weren't the ones doing any scraping; the training database conveniently acts as rocket fuel and a legal blast shield for their machine learning activities, in their view.
[14]OpenAI: 'Impossible to train today’s leading AI models without using copyrighted materials'
[15]Microsoft CEO of AI: Your online content is 'freeware' fodder for training models
[16]Ex-Amazon exec claims she was asked to ignore copyright law in race to AI
[17]US House mulls forcing AI makers to reveal use of copyrighted training data
"Apple has sourced data for their AI from several companies," tech reviewer Marques Brownlee [18]said on Xitter in light of the findings. "One of them scraped tons of data/transcripts from YouTube videos, including mine."
Brownlee noted that "Apple technically avoids 'fault' here because they're not the ones scraping."
The Register has asked EleutherAI, Apple, Nvidia, and others named in the report for further details and explanations.
[19]
Using people's work to train AI without explicit permission has sparked big lawsuits. Microsoft and OpenAI were sued in April [20]by a cohort of US newspapers, and [21]two AI music generators got complaints from Sony, Warner Brothers, and Universal.
A few things seem certain. Artificial intelligence developers can and will get their hands on all manner of information for training – as training data drives their neural networks' performance – and they don't always need explicit permission from creative types to do it, as permission may already have been quietly granted through platform T&Cs.
And at least some of these development labs are highly reluctant to reveal where exactly they get their training data, for various reasons as you can imagine, including commercial secrecy.
This is something we expect to see rumble on and on, with more and more revelations of info being exploited, no matter how legal or ethical, much to the exasperation of the people creating that material in the first place and being displaced by this technological work. ®
Get our [22]Tech Resources
[1] https://pile.eleuther.ai/
[2] https://www.proofnews.org/apple-nvidia-anthropic-used-thousands-of-swiped-youtube-videos-to-train-ai/
[3] https://github.com/EleutherAI/the-pile
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://arxiv.org/abs/2101.00027
[6] https://github.com/sdtblck/youtube_subtitle_dataset
[7] https://www.proofnews.org/youtube-ai-search/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.youtube.com/t/terms
[11] https://www.youtube.com/robots.txt
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://www.theregister.com/2024/05/14/google_io_is_google_ai/
[14] https://www.theregister.com/2024/01/08/midjourney_openai_copyright/
[15] https://www.theregister.com/2024/06/28/microsoft_ceo_ai/
[16] https://www.theregister.com/2024/04/22/ghaderi_v_amazon/
[17] https://www.theregister.com/2024/04/10/congressional_bill_would_require_ai/
[18] https://x.com/MKBHD/status/1813206956716212511
[19] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[20] https://www.theregister.com/2024/04/30/newspapers_microsoft_openai/
[21] https://www.theregister.com/2024/06/24/udio_suno_riaa/
[22] https://whitepapers.theregister.com/
Those transcripts were compiled into what is termed the YouTube Subtitles dataset and incorporated into a larger repository of training material called [1]the Pile , nonprofit nu-journo outfit Proof News [2]highlighted this week. The YouTube Subtitles collection contains information from 173,536 YouTube videos including those of channels operated by Harvard University, the BBC, and web-celebs like Jimmy "MrBeast" Donaldson.
The dataset is a 5.7GB slice of Pile, a larger 825GB silo created by nonprofit outfit EleutherAI. The Pile [3]includes data pulled from GitHub, Wikipedia, Ubuntu IRC, Stack Exchange, bio-medical and other scientific papers, internal Enron emails, and many other sources. Overall, the YouTube Subtitles dataset is one of the smallest collections in the Pile.
[4]
Big names such as Apple, Salesforce, and Nvidia have incorporated the Pile, including the video transcripts, into their AI models during training. We're told the makers of those YouTube videos weren't aware this was happening. (There's also nothing stopping tech giants from using YouTube data in other dataset collections; the Pile is just one possible source.)
Not hidden
It wasn't a secret EleutherAI had gathered up subtitles from YouTube videos, as the organization not only made the Pile publicly available, it detailed the thing in a [5]research paper in 2020. The code that scraped the YouTube Subtitles dataset is on [6]GitHub for all to see. The script can be told to pull in subtitles for videos that match certain search terms; in the Pile's case, those terms ranged from things like "quantum chromodynamics" to "flat earth."
The actual videos used to form the dataset aren't mentioned in either the 2020 paper nor on GitHub. Only now are people looking through the training data, since superseded by other collections, identifying the videos that were scraped, and tipping off YouTube creators. An online search tool for inspecting the subtitle training material has been offered [7]here .
[8]
[9]
What's interesting is that Google-owned YouTube's [10]terms of service , today at least, explicitly ban the use of scrapers and other automated systems unless they are public search engines that obey YT's [11]robots.txt rules or have specific permission from YouTube.
The terms also seemingly prohibit the downloading and use of things like subtitles in AI training unless, again, YouTube and applicable rights holders give permission. So on the one hand, there is the potential for the automated scraping of subtitles to be against YouTube's rules, but there's also wiggle room for it to be totally fine. Well, as far as YouTube is concerned; creators feeling their work is being unethically exploited, however legal, by rich companies is another thing.
[12]
It's something that everyone is dancing around. The PR folks at Google – which [13]is itself in the AI game – have simply said, in response to this week's reporting, that the internet giant puts a lot of effort into thwarting unauthorized scraping, and declined to talk about individual organizations' use of its YouTube data.
AI labs that used the Pile to build their models argued they simply incorporated a broad public dataset and that they weren't the ones doing any scraping; the training database conveniently acts as rocket fuel and a legal blast shield for their machine learning activities, in their view.
[14]OpenAI: 'Impossible to train today’s leading AI models without using copyrighted materials'
[15]Microsoft CEO of AI: Your online content is 'freeware' fodder for training models
[16]Ex-Amazon exec claims she was asked to ignore copyright law in race to AI
[17]US House mulls forcing AI makers to reveal use of copyrighted training data
"Apple has sourced data for their AI from several companies," tech reviewer Marques Brownlee [18]said on Xitter in light of the findings. "One of them scraped tons of data/transcripts from YouTube videos, including mine."
Brownlee noted that "Apple technically avoids 'fault' here because they're not the ones scraping."
The Register has asked EleutherAI, Apple, Nvidia, and others named in the report for further details and explanations.
[19]
Using people's work to train AI without explicit permission has sparked big lawsuits. Microsoft and OpenAI were sued in April [20]by a cohort of US newspapers, and [21]two AI music generators got complaints from Sony, Warner Brothers, and Universal.
A few things seem certain. Artificial intelligence developers can and will get their hands on all manner of information for training – as training data drives their neural networks' performance – and they don't always need explicit permission from creative types to do it, as permission may already have been quietly granted through platform T&Cs.
And at least some of these development labs are highly reluctant to reveal where exactly they get their training data, for various reasons as you can imagine, including commercial secrecy.
This is something we expect to see rumble on and on, with more and more revelations of info being exploited, no matter how legal or ethical, much to the exasperation of the people creating that material in the first place and being displaced by this technological work. ®
Get our [22]Tech Resources
[1] https://pile.eleuther.ai/
[2] https://www.proofnews.org/apple-nvidia-anthropic-used-thousands-of-swiped-youtube-videos-to-train-ai/
[3] https://github.com/EleutherAI/the-pile
[4] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[5] https://arxiv.org/abs/2101.00027
[6] https://github.com/sdtblck/youtube_subtitle_dataset
[7] https://www.proofnews.org/youtube-ai-search/
[8] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[9] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[10] https://www.youtube.com/t/terms
[11] https://www.youtube.com/robots.txt
[12] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[13] https://www.theregister.com/2024/05/14/google_io_is_google_ai/
[14] https://www.theregister.com/2024/01/08/midjourney_openai_copyright/
[15] https://www.theregister.com/2024/06/28/microsoft_ceo_ai/
[16] https://www.theregister.com/2024/04/22/ghaderi_v_amazon/
[17] https://www.theregister.com/2024/04/10/congressional_bill_would_require_ai/
[18] https://x.com/MKBHD/status/1813206956716212511
[19] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_software/aiml&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33ZpdB2lEhEippJ-3NH2z0CwAAA00&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[20] https://www.theregister.com/2024/04/30/newspapers_microsoft_openai/
[21] https://www.theregister.com/2024/06/24/udio_suno_riaa/
[22] https://whitepapers.theregister.com/
Quality data
Bendacious
As mentioned above, anyone who has tried to use the subtitles on YT will know that they are almost useless. In my experience they have a 100% record for getting technical terms wrong and cannot cope with anything outside simple words spoken with a mild US accent.
As these subtitles are automatically generated, this is a situation discussed often in Reg comments - AI training data produced by AI. I can’t believe the data is not poisoned in many places, which kinda serves them right.
On a separate topic, no one abuses YT creators as much as YT. They recently enabled adverts on people’s channels where the creators had chosen not to advertise, without any option to avoid it. Big channels used by children and even babies (Miss Rachel, for example) now show pre-roll and mid-roll adverts against the wishes of the channels. Evil bastards.
Re: Quality data
Anonymous Coward
I could be wrong, but I'm *pretty* sure this is talking about human-written subtitles, not the auto-generated ones. Everyone knows the auto subtitles are trash, even YouTube, that's why you can still make your own. Some YouTubers even hire dedicated transcribers for their videos. Besides that, generated subtitles would just be a synthetic dataset anyway, so you may as well generate a real synthetic dataset of videos with subtitles instead of a half-real/half-wrong one.
Models trained on transcripts is going to be very interesting, because I'm not sure if I've ever seen a transcript that aligns 1:1 with what the speaker is saying. Every transcript has at least one word or phrase used differently than what was said. I'm sure with enough data that the subtle differences won't amount to much deviation, but it'll be curious if any of it manages to throw a wrench in a model, since apparently you can just straight-up perfectly replicate things models are trained on.