A frontier image lab used live human rankings as the reward for reinforcement learning.
Instead of training a reward model, a frontier image-generation lab put real people directly into its RLHF post-training loop, and needed them to keep up with thousands of GPUs.
- 200M+
- human judgments
- 6M+
- different people
- 60–90 s
- per ranking cycle
- 6.6M
- judgments in a single day
The challenge
Most labs post-train with RLHF by first fitting a reward model on a fixed preference dataset, then optimizing against that proxy. This lab wanted fresh human judgments on its own model's outputs as the reward itself, so the model aligns with what people actually prefer rather than with a model of it.
Its training loop ran on thousands of GPUs. For every prompt the model generated a group of six outputs, GRPO-style, and people had to rank all six within 60 to 90 seconds, around the clock, or the hardware would sit idle. That is millions of judgments per day, where a staffed annotation team delivers thousands per week.
How the lab uses Rapidata
Each group goes to Rapidata through the API and comes back ranked; the ranking within the group becomes the relative reward for that training step. Many prompts are ranked in parallel, so the GPUs keep generating while people judge the previous batch.
For calls that need a trained eye, such as anatomy errors, garbled text, images that look AI-generated and crop quality, the lab built custom audiences: people first qualify on the lab's own examples before their answers count.
The result
Human rankings kept pace with the GPUs for six months across ~150 collection days, peaking at more than 350,000 judgments in a single hour. The busiest month passed 100 million responses, counting the quality-control questions that keep the signal clean.
With millions of different people behind the rankings, the model is aligned to broad human preference rather than to any single annotator's taste.
“[…] you are so precious to us that we can't share it with our competitors.”
Source: Rapidata platform data, aggregated and anonymized