Freetext: How to label unknown classes
In the last post, we re-annotated the Animals-10 dataset using the given 10 categories, plus Something Else. It turned out, quite a few images ended up in this mystery bucket. Today we are going to label them!
What to do with our bucket of mystery images?
We only know that the Something Else images are unlikely to be one of the original 10 categories, but it is near impossible to set up a classification task to give users the option to annotate arbitrary animals.
But don't despair, Freetext is here to the rescue! It has the power to label anything.
What? How is that possible? From the user's perspective, it looks like this:
They can simply type any animal into the textbox, hence giving infinite possiblities.
Let's go
Setting this one up is super easy. First we create a client to interact with the Rapidata API and find all images in the directory:
import os
import rapidata
rapi = rapidata.RapidataClient()
# Define the directory path
directory_path = "path_to_your_mystery_folder"
image_paths = []
# Walk through the directory and collect all file paths
for root, dirs, files in os.walk(directory_path):
for file in files:
image_paths.append(os.path.join(root, file))
You might be prompted to login if it's the first time you are calling rapidata.RapidataClient(). After executing this cell, we have all images stored in image_paths.
Now, we can simply run a Free Text Order on these images like this:
order = (
rapi.create_free_text_order("Animals Free Text test")
.question("What is the animal in the image?")
.media(paths_list)
.responses(5)
.minimum_characters(2)
.submit()
)We collect five responses image by setting .responses(5). Also, the shortest animal name I can think of is ox, so I set the minimum characters to three with .minimum_characters(2).
Depending on how many images we gave, this might take a few minutes. We can check the progress with
order.display_progress_bar()Once the order has completed, we can fetch the results:
results = order.get_results()Results
Here are 10 sample responses (from various iamges):
- Es un lobo
- Passaro
- Slepi miš
- Donkey
- León
- ديك رومى
- ماعز
- 灰色的仓鼠
- Pavo real
- Schwein
Perhaps this is not the result you expected. As our platform operates globally, we get global responses. Our end goal is to know which image shows what animal. Hence we want for every image, a single english corresponding label. Luckily, OpenAI's ChatGPT knows plenty of vocabulary and can even correct small typos.
Here's how we can get english labels:
from openai import OpenAI
from concurrent.futures import ThreadPoolExecutor
client = OpenAI(
api_key="your-openai-api-key"
)
def translate_to_english(row):
try:
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": "You are a given responses from users answering what animal is shown in the image in different languages. You are given the language that the device is set to as a hint. Ouptut only the english name of the animal, all lowercase without any punctuation. If the input text does not mention an animal, output 'none'.",
},
{
"role": "user",
"content": f"Input text: '{row['response']}'. The user's device was set to the language code '{row['language']}'.",
},
],
)
return response.choices[0].message.content
except Exception as e:
return f"Error: {e}"
def parallel_translate(df, max_workers=10):
with ThreadPoolExecutor(max_workers=max_workers) as executor:
results = list(
executor.map(translate_to_english, [row for _, row in df.iterrows()])
)
return results
df["animal"] = parallel_translate(df, max_workers=5)This gives us:
- Es un lobo | wolf
- Passaro | bird
- Slepi miš | dormouse
- Donkey | donkey
- León | lion
- ديك رومى | turkey
- ماعز | goat
- 灰色的仓鼠 | hamster
- Pavo real | peacock
- Schwein | pig
Much better!
Lastly, I grouped the translated responses by image and only took the ones where we had at least 3 voters in agreement. This resulted in 103 images categorized into 34 different classes. The labeled images can be found on HuggingFace.
A few examples of the labeled images:
