News: 1593600948

Special report MIT has taken offline its highly cited dataset that trained AI systems to potentially describe people using racist, misogynistic, and other problematic terms.

The database was removed this week after The Register alerted the American super-college. And MIT urged researchers and developers to stop using the training library, and to delete any copies. "We sincerely apologize," a professor told us.

The training set, built by the university, has been used to teach machine-learning models to automatically identify and list the people and objects depicted in still images. For example, if you show one of these systems a photo of a park, it might tell you about the children, adults, pets, picnic spreads, grass, and trees present in the snap. Thanks to MIT's cavalier approach when assembling its training set, though, these systems may also label women as whores or bitches, and Black and Asian people with derogatory language. The database also contained close-up pictures of female genitalia labeled with the C-word.

Applications, websites, and other products relying on neural networks trained using MIT's dataset may therefore end up using these terms when analyzing photographs and camera footage.

The problematic training library in question is 80 Million Tiny Images, which was created in 2008 to help produce advanced object detection techniques. It is, essentially, a huge collection of photos with labels describing what's in the pics, all of which can be fed into neural networks to teach them to associate patterns in photos with the descriptive labels. So when a trained neural network is shown a bike, it can accurately predict a bike is present in the snap. It's called Tiny Images because the pictures in library are small enough for computer-vision algorithms in the late-2000s and early-2010s to digest.

Today, the Tiny Images dataset is used to [1]benchmark computer-vision algorithms along with the better-known ImageNet training collection. Unlike [2]ImageNet , though, no one, until now, has scrutinized Tiny Images for problematic content.

Vinay Prabhu, chief scientist at UnifyID, a privacy startup in Silicon Valley, and Abeba Birhane, a PhD candidate at University College Dublin in Ireland, pored over the MIT database and discovered thousands of images labelled with racist slurs for Black and Asian people, and derogatory terms used to describe women. They revealed their findings in a [3]paper undergoing peer review for the [4]2021 Workshop on Applications of Computer Vision conference.

[5]

Graph showing the number of pictures in the MIT dataset labeled with selected problematic words ... Source: Prabhu and Birhane

The dataset holds more than 79,300,000 images, scraped from Google Images, arranged in 75,000-odd categories. A smaller version, with 2.2 million images, could be searched and perused online from the website of MIT’s Computer Science and Artificial Intelligence Lab (CSAIL). This visualization, along with the full downloadable database, were removed on Monday from the CSAIL website after El Reg alerted the dataset's creators to the work done by Prabhu and Birhane.

The key problem is that the dataset includes, for example, pictures of Black people and monkeys labeled with the N-word; women in bikinis, or holding their children, labeled whores; parts of the anatomy labeled with crude terms; and so on – needlessly linking everyday imagery to slurs and offensive language, and baking prejudice and bias into future AI models.

[6]

A screenshot of the 2.2m dataset visualization before it was taken offline this week. It shows some of the dataset's examples for the label 'whore', which we've pixelated for legal and decency reasons. The images ranged from a headshot photo of woman and a mother holding her baby with Santa to porn actresses and a woman in a bikini ... Click to enlarge

Antonio Torralba, a professor of electrical engineering and computer science at CSAIL, said the lab wasn't aware these offensive images and labels were present within the dataset at all. “It is clear that we should have manually screened them,” he told The Register . “For this, we sincerely apologize. Indeed, we have taken the dataset offline so that the offending images and categories can be removed.”

In a [7]statement on its website, however, CSAIL said the dataset will be permanently pulled offline because the images were too small for manual inspection and filtering by hand. The lab also admitted it automatically obtained the images from the internet without checking whether any offensive pics or language were ingested into the library, and it urged people to delete their copies of the data:

It has been brought to our attention that the Tiny Images dataset contains some derogatory terms as categories and offensive images. This was a consequence of the automated data collection procedure that relied on nouns from WordNet. We are greatly concerned by this and apologize to those who may have been affected.

The dataset is too large (80 million images) and the images are so small (32 x 32 pixels) that it can be difficult for people to visually recognize its content. Therefore, manual inspection, even if feasible, will not guarantee that offensive images can be completely removed.

We therefore have decided to formally withdraw the dataset. It has been taken offline and it will not be put back online. We ask the community to refrain from using it in future and also delete any existing copies of the dataset that may have been downloaded.

Prof Torralba told us a little more on how the library was constructed: a huge list of words – including derogatory terms – was obtained and then code was written to search the web for images using these words, and combine the lot. The result was a dataset containing raw internet material.

“The dataset contains 53,464 different nouns, directly copied over from WordNet," Prof Torralba said referring to Princeton University's database of English words grouped into related sets. "These were then used to automatically download images of the corresponding noun from internet search engines at the time, using the available filters at the time, to collect the 80 million images.”

[8]WordNet was built in the mid-1980s at Princeton's Cognitive Science Laboratory under George Armitage Miller, one of the founders of cognitive psychology. “Miller was obsessed with the relationships between words,” Prabhu told us. “The database essentially maps how words are associated with one another.”

For example, the words cat and dog are more closely related than cat and umbrella. Unfortunately, some of the nouns in WordNet are racist slang and insults. Now, decades later, with academics and developers using the database as a convenient silo of English words, those terms haunt modern machine learning.

“When you are building huge datasets, you need some sort of structure,” Birhane told El Reg . “That’s why WordNet is effective. It provides a way for computer-vision researchers to categorize and label their images. Why do that yourself when you could just use WordNet?”

WordNet may not be so harmful on its own, as a list of words, though when combined with images and AI algorithms, it can have upsetting consequences. “The very aim of that [WordNet] project was to map words that are close to each other," said Birhane. "But when you begin associating images with those words, you are putting a photograph of a real actual person and associating them with harmful words that perpetuate stereotypes.”

ImageNet has the same problems, too, as it was also annotated using WordNet. An experiment dubbed [9]ImageNet Roulette allowed people to submit photos to a neural network trained from ImageNet that would describe the images using labels from the dataset. Unsurprisingly, people fed the system snaps that fascinated them the most: their selfies. Some were shocked when the software described them using racist and offensive labels.

This image-recognition roulette is all fun and games... until it labels you a rape suspect, divorcee, or a racial slur [10]READ MORE

The fraction of problematic images and labels in these giant datasets is small, and it’s easy to brush them off as anomalies. Yet this material can lead to real harm if they’re used to train machine-learning models that are used in the real world, Prabhu and Birhane argued.

“The absence of critical engagement with canonical datasets disproportionately negatively impacts women, racial and ethnic minorities, and vulnerable individuals and communities at the margins of society,” they wrote in their paper.

These groups are often not well represented in AI training datasets; it’s the reason why facial-recognition algorithms struggle with identifying women and people of darker skin. A Black man in Detroit was [11]wrongfully arrested by cops after being mistaken for a suspected thief by facial-recognition software earlier this year. It’s also the reason why a [12]controversial AI algorithm that generates high-resolution images from low-resolution snaps turned a blurry photo of Barack Obama into someone more Caucasian than Black.

“People don’t think about how these models are going to applied or what it could be used for," said Birhane. "They just think ‘oh, here’s this cool thing I can do’. But when you start thinking deeper, you will start to find all these insidious purposes and see how these harms manifest.”

Giant datasets like ImageNet and 80 Million Tiny Images are also often collected by scraping photos from Flickr or Google Images without people’s explicit consent. Meanwhile, Facebook [13]hired actors who agreed to have their faces used in a dataset designed to teach software to detect computer-generated faked images.

Prabhu and Birhane said the social network's approach was a good idea, though they noted academic studies are unlikely to have the funding to pay actors to star in training sets. “We acknowledge that there is no perfect solution to create an ideal dataset, but that doesn’t mean people shouldn’t try and create better ones,” they said.

The duo suggested blurring people’s faces in datasets focused on object recognition, carefully screening the images and labels to remove any offensive material, and even training systems using realistic synthetic data. “You don’t need to include racial slurs, pornographic images, or pictures of children," they said. "Doing good science and keeping ethical standards is not mutually exclusive.” ®

Get our [14]Tech Resources



[1] https://www.cs.toronto.edu/~kriz/cifar.html

[2] https://www.theregister.com/2019/10/23/ai_dataset_imagenet_consent/

[3] https://openreview.net/forum?id=s-e2zaAlG3I

[4] http://wacv2021.thecvf.com/home

[5] https://regmedia.co.uk/2020/06/30/graph.jpg

[6] https://regmedia.co.uk/2020/06/30/screenshot_mit_dataset.jpg

[7] https://groups.csail.mit.edu/vision/TinyImages/

[8] https://wordnet.princeton.edu/

[9] https://www.theregister.com/2019/09/18/imagenet_roulette/

[10] https://www.theregister.com/2019/09/18/imagenet_roulette/

[11] https://www.theregister.com/2020/06/24/facial_recognition_arrest/

[12] https://www.theregister.com/2020/06/24/ai_image_tool/

[13] https://ai.facebook.com/blog/deepfake-detection-challenge/

[14] https://whitepapers.theregister.com/