Can VLMs Judge Animal Cuteness?

Why we ran this

In the image above, we show a simple example of where VLMs can fall short in representing human preferences. We asked 30 people which kitten was cuter, and the votes split 15-15. We then asked a VLM the same question 30 times, and it always picked the same kitten. VLMs can of course give different answers across runs, but that doesn't mean their answers reflect the distribution of human preferences.

There's a growing belief that VLMs can replace humans in annotation tasks. While this makes sense for many objective evaluations, it doesn’t hold up as well for subjective questions, where human opinions are ultimately what we're trying to measure.

The small experiment described in this blog post was motivated by a recent LinkedIn discussion following a post by our Chief Scientist, Mads, about the importance of human feedback when rewards cannot be automatically verified. Some argued that LLMs could largely replace human judgment. So we decided to test how well a VLM actually represents human preferences.

While there are certainly domains where LLMs/VLMs can be effectively used as a judge, in the breadth of all possible tasks, there exist many domains where this does not hold. As a toy example we proposed the task of rating cuteness in pictures of animals.

We ran the study ourselves as a quick experiment. We asked people and a VLM the same question about the same images and compared their answers. All data from the experiment is publicly available on Hugging Face.

The setup

We had six image models make pictures for 22 prompts about cats, dogs and rabbits, like "a cute kitten curled up in a basket." The models were gpt-image-2, seedream-5-lite, p-image, nano-banana-2, flux-2-flex and krea-2-medium. Pairing up each prompt's images gave us 325 pairs.

We took those image pairs and added a question “which animal is cuter?”, and then ran those pairs through the Rapidata API to collect 30 responses per pair, totalling 9,750 votes from people in 103 countries.

Then we gave the same pairs and the same question to a VLM, we chose google/gemini-3.8-flash as the model, through OpenRouter. It also voted 30 times per pair.

Here is what each side saw for the same pair:

What people saw

What the VLM saw

[system]
You are shown two images, A and B. Answer the question by picking one of them. You must always pick A or B; ties are not an allowed answer.
 
[user]
Image A:
<first image>
Image B:
<second image>
Which animal is cuter?
 
Answer with the label of the image you prefer, a confidence between 0 and 1, and a reason of at most 15 words. Reply with JSON only.

We balanced the order of the images, too. If a pair had images X and Y, X came first in 15 of the 30 requests and Y came first in the other 15. This is to account for potential position bias [1], a documented phenomenon where the order in which the images are supplied to the VLM will influence the decision outcome. With the order balanced, that position bias is mitigated.*

What we found

People split 15-15. The VLM voted 30-0.

A key insight that we found is that the VLM in general is super deterministic, and even overly confident in cases where the human opinion is split.

The above pair is a prime example of what we found. Thirty people looked at these two kittens. Fifteen picked the sleeping one and fifteen picked the one that's awake. People were subjectively split on which kitten is cuter, which should be considered as the ground truth.

The VLM picked the sleeping kitten all 30 times. This shows how decisive the VLM is compared to humans and misaligned it is with human opinions.

It happened on other pairs too

People split 16-14 or closer on 53 of the 325 pairs. The VLM voted 30-0 on 30 out of those 53 pairs.

Here are five of the pairs where the VLM voted 30-0 on 216 pairs (about two thirds of all the pairs). Here are five of those pairs, next to how people voted:

The VLM's answer is the same in every row, even though people's answers aren't. The first two examples the VLM was very closely aligned with humans. On the third pair people were split, and the VLM was very biased towards the first pair. The last two examples, the VLM was very misaligned with people’s opinions.

Overall, nothing in the results suggests the VLM reliably reproduces how people judge cuteness. On the 216 pairs where it voted 30-0, it agreed with the human majority on 163 (75%). On 23 (11%), it unanimously picked the image most people didn't choose. On the other 30 (14%), people were split close to even, like with the kittens, and the VLM hid that split completely.

We think the VLM may be falling back on a more general sense of image quality or aesthetic preference, rather than answering the actual question of which animal is cuter.

What this means

"Which animal is cuter?" has no single correct answer. What matters is how people's preferences are distributed. Yet, on 30 of 53 nearly evenly split pairs, the VLM voted 30-0. On another 23 pairs, it picked the opposite of the clear human favorite.

Now why does this matter? If we use such labels to train or evaluate models, we are optimizing for one model's preferences rather than what people actually like.

Next week, we're releasing a benchmark on mode collapse in text-to-image models. There we see this happening in practice, some of the best-performing models on diversity are older versions, rather than their newer successors.

Our little study doesn't explain this, but it raises an interesting question more labs should reflect on: Why are we relying so heavily on VLMs and reward models to approximate human preferences, potentially narrowing the diversity of what models generate, when direct human feedback is available at scale and high throughput?

Of course, this experiment covers just one VLM, one subjective task, and 325 image pairs. It's not a general conclusion, but I think it's a question worth exploring.

PS: Many VLMs can’t be used to train your own model

Google, OpenAI and Anthropic all restrict using their models' outputs to train models that compete with their own. [2] So if you label data with a VLM and train an image or video model on those labels, you may be breaking the VLM's terms. Even in this experiment, where we trained nothing, we got flagged as inconsistent with Google's Usage Policy.

Notes

* The VLM picked the image it saw first 52.7% of the time. Flipping the order changed its pick on 31 of the 325 pairs. We can't test people this way, because our data doesn't record which side they saw each image on. Rapidata automatically shows each person the two images in a random order, so a lean toward one side averages out.

We are aware that asking the same VLM the same question 30 times likely produces a more deterministic result than asking 30 different people, since every answer comes from the same model with the same biases. A more rigorous comparison would poll several different VLMs and compare their opinions.

[1] Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena", NeurIPS 2023 (arxiv.org/abs/2306.05685).

[2] Gemini API Additional Terms of Service (ai.google.dev/gemini-api/terms); Google Cloud Service Specific Terms, section 17, "Use Restrictions for AI/ML Services" (cloud.google.com/terms/service-terms); OpenAI Services Agreement, section 3 and the definition of "Permitted Exception" (openai.com/policies/services-agreement); Anthropic, "Can I use my Outputs to train an AI model?" (support.claude.com). Checked October 14, 2026. Terms change often, and this isn't legal advice.