# Rapidata

> Rapidata delivers human feedback at scale: crowd-sourced labeling, model
> evaluation, ranking, and preference data (RLHF-style) collected from real
> people around the world — typically in days rather than months. This site
> (rapidata.ai) is the company's public website.

Developers integrate through the Rapidata Python SDK; see the documentation and
source links below. API access uses a token created in the Rapidata app.

## When to use Rapidata

Reach for Rapidata when a task needs **real human judgment at scale** rather than
a model's own output:

- Evaluate or compare model outputs (image, video, audio, text) with human preferences — pairwise A/B comparisons, ranking, leaderboards.
- Build RLHF / preference datasets and reward-model training data.
- Label data: classification, comparison, ranking and free-text answers, at over 300,000 responses per hour.
- Run work on a custom audience trained on your own examples with known answers, optionally narrowed by country or language, with confidence-based early stopping.

It is **not** the right tool for fully automated labeling with no human in the
loop, sub-second real-time inference, or expert-written and long-form data.

How an agent calls it: use the Python SDK (`pip install rapidata`) or the OAuth 2.0
REST API — see the [SDK documentation](https://docs.rapidata.ai) and
[developer portal](https://docs.rapidata.ai/developers/). Coding agents can install
the [official skill](https://docs.rapidata.ai/ai_agents/) to have
integrations written for them.

## Product

- [Model Rank Insights](https://rapidata.ai/model-rank-insights): benchmark and rank models with human preference data
- [Ranking Flows](https://rapidata.ai/flows): continuous human ranking inside a training loop
- [Custom Audiences](https://rapidata.ai/audiences): qualify annotators on your task before they work on it
- [Pricing](https://rapidata.ai/pricing): start free with $20 in credit, then pay as you go

## Solutions

- [Human evaluation for AI models](https://rapidata.ai/solutions/model-evaluation)
- [RLHF and DPO preference data](https://rapidata.ai/solutions/reinforcement-learning)
- [Online RLHF](https://rapidata.ai/solutions/online-rlhf): human preferences streamed into the optimizer during training
- [Human evaluation as a service](https://rapidata.ai/solutions/human-evaluation): comparisons, ratings and rankings for image, video and audio models
- [Data labeling](https://rapidata.ai/solutions/data-labeling): human annotation through one API
- [Audio and TTS evaluation](https://rapidata.ai/solutions/audio-evaluation): naturalness and MOS ratings from native speakers
- [Multilingual and demographic evaluation](https://rapidata.ai/solutions/multilingual-and-demographic): feedback from 190+ countries, by language
- [Use-case explorer](https://rapidata.ai/use-cases): real open-source human-feedback examples by modality and annotation type

## How Rapidata compares

Each page has a "best for / not for" table; Rapidata is not the right choice for
expert-written or long-form data (SFT demonstrations, coding or domain-expert work).

- [Amazon Mechanical Turk alternatives](https://rapidata.ai/alternatives/mturk): MTurk closed on 2026-09-30; HIT-to-Rapidata migration guide with Python code
- [Rapidata vs Scale AI](https://rapidata.ai/alternatives/scale-ai): self-serve alternative to a managed data partner
- [Toloka alternatives](https://rapidata.ai/alternatives/toloka)
- [Surge AI alternatives](https://rapidata.ai/alternatives/surge-ai)
- [Prolific alternatives](https://rapidata.ai/alternatives/prolific): same 300 tasks on both crowds: $4 vs $76 per 1,000 answers

## Developers

- [SDK documentation](https://docs.rapidata.ai): the Rapidata Python SDK
- [Quick Start](https://docs.rapidata.ai/quickstart/): install, authenticate, run your first job
- Install: `pip install rapidata`
- Source: https://github.com/RapidataAI/rapidata-python-sdk
- PyPI: https://pypi.org/project/rapidata/
- Public benchmarks and leaderboards: https://www.benchmark.ai

## Research

- [Can VLMs Judge Animal Cuteness?](https://rapidata.ai/blog/vlm-vs-crowd): VLMs are often used to annotate images, but that might misrepresent how people feel. We compared a VLM with people on 325 pairs of pet images.
- [Ding-dong, the glitch is dead](https://rapidata.ai/blog/the-glitch-is-gone): AI images used to give themselves away through mistakes. The best models have stopped making them. People can still tell, but not the way they used to.
- [Live Human Feedback in the Training Loop: Aligning Diffusion Models with Real People](https://rapidata.ai/blog/live-human-feedback-rlhf): Online RLHF for image and video models usually relies on a learned reward model as a stand-in for human judgement. This post shows how Rapidata Flows bring real human preferences into the post-training loop instead, fast enough to keep up with training and in just a few lines of code.
- [Rapidata vs. Prolific - Cost and Quality on the Same 300 Tasks](https://rapidata.ai/blog/rapidata-vs-prolific): Prolific pays its crowd by the hour. Rapidata reaches thousands of people within seconds through ads. We asked both crowds the same 300 questions and counted who got them right, how fast, and at what cost.
- [Fine-Tuning Stable Diffusion to Generate Full Wine Glasses](https://rapidata.ai/blog/wine-glasses): Image models can't draw a completely full wine glass. We fine-tuned Stable Diffusion with D3PO on human preferences from Rapidata, and it beat GPT-4o feedback.
- [Landing the Lunar Lander with Human Feedback](https://rapidata.ai/blog/lunar-lander): Training a PPO agent to land the Lunar Lander from human comparisons of short clips, collected with Rapidata, instead of the environment's built-in reward.
- [Beyond Image Preferences - Rich Human Feedback](https://rapidata.ai/blog/rich-human-feedback): We collected 1.5M annotations from 150k+ people for a rich text-to-image feedback dataset: Likert ratings, misaligned prompt words and heatmaps of problem areas.
- [Finding the subjective truth - Collecting 2M Votes for GenAI model evaluation](https://rapidata.ai/blog/genai-blog-post): We collected over 2 million human annotations on 4,512 images to rank DALL-E 3, Flux.1, MidJourney and Stable Diffusion on style, coherence and prompt alignment.
- [Object Detection in Computer Vision](https://rapidata.ai/blog/object-detection): Explore the evolution and applications of object detection in computer vision, from early algorithms like Viola-Jones to modern models like YOLO. Learn about the importance of quality labels and popular datasets such as COCO and Pascal VOC.

## Guides

- [Freetext: How to label unknown classes](https://rapidata.ai/guides/animal-free-text): How to label images that fit none of your classes: we used Rapidata's free-text tasks to name the animals in the 'Something Else' bucket of Animals-10.
- [Fixing public datasets — Re-annotating Animals 10](https://rapidata.ai/guides/animal-classification): Public datasets contain label errors. We had thousands of people re-annotate the 24,000 images of Animals-10 with Rapidata and found 483 wrong labels, about 2%.
- [On-Demand Human Preference Data for AI Training](https://rapidata.ai/guides/preference-dataset-demo): Collect pairwise human preference data on Flux.1 [pro] vs Stable Diffusion 3 images with the Rapidata Python SDK, the same method behind our 2M-response benchmark.
- [Bored or Intrigued - Setting Up a Classification Annotation Project](https://rapidata.ai/guides/classification-guide): Learn how to set up a classification annotation project using DALL-E 3 and Rapidata for bored vs intrigued emotion detection in images. Follow step-by-step instructions to replicate the experiment.

## Company

- [Papers using Rapidata](https://rapidata.ai/papers)
- [Research grants](https://rapidata.ai/research-grants): up to $50,000 in annotation credits per research project
- [Press](https://rapidata.ai/press)
- [Security](https://rapidata.ai/security): how Rapidata handles the data you send and the people who label it
- [Team](https://rapidata.ai/team)
- [Careers](https://rapidata.ai/join)
- [Contact](https://rapidata.ai/contact)