Physics-IQ Verified tests whether video models understand physics. It builds on Google DeepMind's Physics-IQ: 66 real experiments, each filmed from three camera angles, so 198 scenes.
A model gets one frame of a scene and a short description, and continues it for five seconds. For example, it gets a frame of two tennis balls on a table and the text "A light-colored wooden tabletop with two pipes at the edges. The blue and the yellow tennis ball roll out of the pipes and collide with the same speed." Physics-IQ then scores the video without looking at what's in it: it takes pixel differences between frames to find where and when things change, and compares that with the real recording at fixed time intervals. [1]
We had people judge the same videos instead: 26 models on all 198 scenes. [2] They saw two videos at a time and answered "Which video is more realistic?" They never saw the prompt or the real recording.
Where the metric prefers the worse one
Two balls collide
In Wan 2.5 Preview's video the balls speed up when they collide and roll off the table. In the real recording they come to a stop. In both, by the end of the video nothing on the table is moving, so Physics-IQ Verified ranks it high. Wan 3.0's balls slow down on the collision and then keep rolling slowly, the way real balls do.
A ball behind a turning card
MiniMax H3's grabber and ball glitch through the card, and still score higher than FLUX 3 Video, whose grabber simply puts the ball down behind it.
Marble run
Neither marble moves quite right. The one people preferred is unnaturally bouncy, but that's a much smaller violation of physics than Wan 2.7's marble, which rolls up the wall out of the frame, comes back and glitches through a ramp.
Different, but not wrong
In some of these scenes, almost nothing changes over the whole recording. Nothing in the frame or the description says it has to stay that way, but anything that interacts with the scene costs a lot of Physics-IQ Verified score, because it doesn't match the recording.
A tap nobody opens
A hand opens the tap and pours, which takes more physics to get right than a still dispenser. But it strays further from the recording, so it gets punished.
A second candle
A hand lights the second candle, which again takes more understanding of physics. Veo 3.1, where nothing happens, scores 100.
Burning paper
The paper catches fire. The model can't know this piece burns poorly, so catching fire is a valid outcome.
A still image beats four of the 26 models
Take the first frame, hold it still for five seconds, and score that with Physics-IQ. No model, nothing moves. Wherever the real scene is still, a frozen image is a perfect match, so it scores 28.2.
| Physics-IQ Verified score | |
|---|---|
| HappyHorse 1.0 | 28.4 |
| First frame, held still | 28.2 |
| Vidu Q3 Pro | 25.7 |
| Seedance V1.5 Pro | 22.0 |
| SANA-WM | 21.7 |
| Dream-X-World | 17.6 |
Four of the video models we tested score lower than doing nothing at all.
Prompt adherence isn't physical correctness
A brush through paint
FLUX turns the brush counterclockwise; the prompt says clockwise. Otherwise it's fine. SANA-WM's video glitches completely, but the pixels change in the right area, so it scores high.
Where people get it wrong
A kettlebell on a mug
The mug shatters. It looks natural, but the real mug holds. People tend to prefer videos where something happens.
It depends on the question
We also asked a second group, who saw the real recording next to the two videos, which one looks more like it. That's Physics-IQ's question, and their answers come out much closer to the metric's.
Per matchup, the blind group picks Physics-IQ's favorite 62% of the time, where 50% would be chance. The group that saw the recording picks it 71% of the time, and 85% when Physics-IQ's scores for the two videos are at least 20 points apart.
The takeaway: Physics-IQ Verified measures which video is closer to the recording. But a scene doesn't have just one physically valid future. People can judge whether a video could have happened, without punishing futures that are different but still plausible.
Notes
[1] Physics-IQ combines four scores: how well the areas where things change overlap with the real recording's, a weighted version of that, how well they overlap in each time interval, and how different the pixels are. Each score is compared with how much two separate real recordings of the same experiment differ, so a model only has to match as closely as the experiment repeats itself. In 36 of the 198 scenes, including the tap and the card, the dataset's two recordings are the same file. With no difference between them to compare against, the pixel score is 0 for almost every model in those scenes.
[2] Physics-IQ Verified's prompts, unchanged. One video per model per scene, 5,103 in total, scored with the official Physics-IQ code. Ranks under each video compare it with the other videos of that scene. Seedance 2.x on fal zooms the start image by about 6%, which costs it a few points.
All videos and rankings are on the Physics-IQ Verified benchmark.
