Fixing public datasets — Re-annotating Animals 10

Public datasets are surely handy, but are they always dandy? We found that many large public datasets contain some amount of erroneous data, however, scrolling through thousands of images manually is quite time-consuming. Luckily, we can parallelize this task by having thousands of people go through a dataset with Rapidata.

Dataset

For this post, we will have a look at Animals-10. The description says:

All the images have been collected from "google images" and have been checked by human. There is some erroneous data to simulate real conditions (eg. images taken by users of your app).

Well, I'd rather have a clean dataset and inject the noise only when I need it. Let's fix that.

Essentially, all we need is people to look at an image and specify whether it is one of the ten classes or something else. Rapidata's classification job is perfect for this task. Below we will show you how to set up a project to re-annotate the dataset. But first, let's look at the outcome.

Results

In total, out of the 23,999 images, 483 had an incorrect label. That's about 2% of the dataset! Errors fall into two buckets: images that should have been another of the 10 classes and images that should not be in this dataset altogether.

Here are a few examples for image labels that were fixed:

Understandable mistake (original: sheep, new: dog)

Fluffy like a sheep (original: sheep, new: cow)

C'mon, this one is easy (original: cat, new: dog)

A doggo roll (original: squirrel, new: dog)

Not a dogfight (original: dog, new: chicken)

You can ride on more than horses. (original: horse, new: cow)

Other categories

Here are the some images that don't fit into any of the 10 classes, which was a much bigger bucket:

Colorful as a butterfly, but actually a Eurasian Hoopoe (original: butterfly, new: something else)

This might be one in the future (original: butterfly, new: something else)

What's up? (original: butterfly, new: something else)

Homo Sapiens (original: cow, new: something else)

Fancy speakers (original: chicken, new: something else)

Maybe it has few horses under the hood (original: horse, new: something else)

Evidently, we successfully identified quite a few mislabelings. The corrected dataset is now available to be explored and downloaded on Hugging Face.

Setting up the project

We will use this data in the following example, but feel free to use your own.

First, we need to install the Rapidata package:

pip install -U rapidata

Then we can start setting up the project. We can start by importing the necessary packages, creating the Rapidata client object and picking an audience. The global audience is the broadest pool of labelers and is ready to work immediately:

from rapidata import RapidataClient
 
rapi = RapidataClient()
audience = rapi.audience.get_audience_by_id("global")

From there we can define the categories and the path to the images. Presenting the user with too many categories can lead to an overload, so it's best to use a maximum of 8 categories. Since we have 10 classes at hand, we first label 6 categories and specify an addional 7th category that we call "Something else". All the categories that are not part of the 6 categories are then labeled as "Something else". In a second step we can then label those with the remaining 4 categories.

CATEGORIES_BATCH_1 = ["Cat", "Dog", "Cow", "Spider", "Butterfly", "Elephant"]
CATEGORIES_BATCH_2 = ["Horse", "Chicken", "Sheep", "Squirrel"]

Starting the job

Setting up the actual labeling job is now relatively simple. First we get all the image paths:

import os
 
MEDIA_FOLDER = "path/to/image/folder" # make sure it's unzipped
 
file_paths = [os.path.join(MEDIA_FOLDER, file) for file in os.listdir(MEDIA_FOLDER)]

Then we can define the job and assign it to the audience, which starts the labeling:

job_definition = rapi.job.create_classification_job_definition(
    name="Animal Classification Batch 1",
    instruction="What is shown in the image?",
    answer_options=CATEGORIES_BATCH_1 + ["Something else"],
    datapoints=file_paths,
    responses_per_datapoint=20,
    confidence_threshold=0.97,
)
 
job = audience.assign_job(job_definition)

The labeling stops for a datapoint when the confidence level for a category exceeds 97% (typically achieved with 3-4 human labels), even if it hasn't reached the full 20 responses. This threshold is controlled by the confidence_threshold parameter. In case these labels are ambiguous, more labels will be collected, but never more than 20 for a single image.

For higher quality, you can replace the global audience with a custom audience that only admits labelers who pass qualification examples for your task. For more info, check out our documentation.

Obtaining the results

We have two ways to monitor the progress of the job, either we head over to app.rapidata.ai or we can also use the SDK to show a progress bar using job.display_progress_bar()

Once the job is finished, we can download the results, either using the UI or the SDK with job.get_results().

That's it!

Next steps

Now it's up to you to label the rest of the data by creating a 2nd job with all the images that were labeled as "Something else" in the first job. You can use the same code as above, just with the second batch of categories. Finally you can merge the results of both jobs to get the full dataset.