Accepted at BMVC 2026

Off-Manifold Refinement

Guiding Video Generators with a Frozen World Model

Hai Nguyen-Truong¹ Tuan-Anh Vu² Dang Huynh¹

¹ Fulbright University Vietnam² University of California, Los Angeles

Read the paper Explore comparisons Cite OMR

Use the world-model signal before the video is finished.

OMR uses a frozen world model to correct a frozen video generator during a single sampling trajectory. A small adapter keeps guidance in latent space—without in-loop VAE decoding or V-JEPA encoding.

Teaser

A backflip onto an inflatable cushion.

Download GIF ↓Full comparison ↗

Inspect a stop

During landing, the body should move continuously and the cushion should deform under contact. Red circles mark abrupt leg–torso changes in Wan2.2; green circles follow OMR’s landing and cushion response.

Both clips slow and pause together over 0–3.5 s. OMR still shows body–cushion overlap in the full comparison.

Method

World-model guidance in generator-latent space.

Video generators can produce convincing clips that violate basic cause and effect. OMR turns a frozen world model’s predictive temporal prior into a local correction, connecting the two models only through a separately trained representation adapter.

OMR method overview: a video generator predicts a clean latent, an adapter maps it to world-model features, and the predictive-surprise gradient guides the sampling velocity. — OMR corrects provisional clean estimates during successive sampling steps. The adapter replaces in-loop VAE decoding and video encoding.

Frozen generatorFrozen world model33.3M adapterNo in-loop decode–encode

z_t+Δt ← z_t + Δt · u_θ(z_t, t) − η(t) · ∇_ẑ₁ E_V-JEPA(A_ψ(ẑ₁))

The frozen generator supplies the ODE velocity; the frozen predictor supplies a local surprise correction through the adapter.

01
Predict
At a guided ODE step, estimate the provisional clean video latent from the current noisy state.
02
Measure surprise
Map that latent into V-JEPA feature space, mask future tokens within a 48-frame window, and compare them with the predictor’s forecast.
03
Refine
Backpropagate predictive surprise through the adapter and add a local correction to the sampling update.

Qualitative comparisons

Look closely at contact, support, and material response.

Start with the annotated loops, then step through five comparisons with P&P. More baseline–OMR examples are available at the end of the page.

Selected differences, up close

Does contact lead to recovery?

0.5× speed

OMR shows the rod approaching, pressing into the bag, and withdrawing as the bag recovers. Wan2.2 shows a rotating prop with no visible hand support.

Download GIF ↓

Both clips misrepresent the requested nunchucks. Inspect contact and recovery, not correct execution of the prompt.

Baseline, P&P, and OMR

Five examples · full-length comparisons

Impact and contact

Baseball–wall impact

Baseline · P&P · OMR

Wan2.2Base generator

P&PSelf-refining sampling

OMR OursWorld-model guidance

Same prompt · same seed · one synchronized comparison

Watch for

The wall just before the ball arrives: does the damage follow contact? Inspect 0.5–3 seconds.

What changes

In OMR, the visible wall response more closely coincides with ball contact. Compare with premature or weakly coupled damage in the baselines.

Prompt & original clips

“In a dynamic sports action sequence, a baseball soars through the air, propelled by a powerful throw from an unseen player. The camera captures the ball's trajectory in a stunning close-up, revealing its textured surface and the blurred background, which hints at a bustling stadium. As the ball approaches a weathered brick wall, its speed intensifies, creating a dramatic tension. The moment of impact is captured with striking clarity, as the ball smashes into the wall, leaving a visible dent and a cloud of dust. The camera remains static, allowing the viewer to fully absorb the force of the collision and the resulting mark, a testament to the power of the throw. This cinematic moment encapsulates the raw energy and excitement of baseball, showcasing the sport's visceral impact.”

Original clips: Wan2.2 ↗P&P ↗OMR ↗

Selected qualitative comparisons, not a win-rate estimate. Inspect the remaining artifacts alongside the visible differences.

Primary results

Better physical commonsense, with unchanged prompt-adherence rate.

We fixed 400 prompt IDs from the released VideoPhy-2 detailed-prompt pool before generation or scoring, and use the identical unfiltered list for every method.

VideoPhy-2 · fixed 400 prompts

Wan2.2-T2V-A14B

Higher is better ↑

Method	SA ≥ 4 (%)	PC ≥ 4 (%)	Joint (%)
Wan2.2	68.0	57.0	47.00
+ CFG-Zero*	68.0	56.0	48.25
+ P&P	70.0	55.0	48.75
+ OMR (ours)	68.0	60.0	52.00

One generation per prompt. SA: semantic adherence. PC: physical commonsense. Joint requires both scores to be at least 4 on the 1–5 scale.

Why it is efficient

A shorter gradient route.

WMReward-∇ differentiates through a V-JEPA encoder and VAE decoder. OMR instead uses the predictor and generator-specific adapter, avoiding those two additional Jacobians and their compute.

The comparison supports a more direct correction route; showing that gradients are larger or better preserved would require gradient-norm measurements.

WMReward-∇ decode–encode route

Backward route predictor + encoder + decoder

OMR-∇ ours adapter-space route

Backward route predictor + adapter No in-loop pixel decoding

Backward gradient: OMR bypasses the in-loop VAE decoder and V-JEPA encoder.

The final output video is still decoded once after sampling.

Separate efficiency protocol

Fixed 50-prompt Wan2.2 subset

Method	Relative time	Peak VRAM	Joint (%)
Wan2.2	1.00×	65.4 GB	48.0
+ P&P	1.6×	72.7 GB	50.6
+ BoN-2	2.00×	65.4 GB	50.4
+ WMReward-∇ + BoN-4	19.88×	134 GB	53.6
+ OMR (ours)	1.71×	74.1 GB	54.0

Wan2.2 and OMR scores are three-seed means; the other rows are point estimates. Time is relative to the base model.

Blind human study

OMR is preferred for physical plausibility.

Thirteen annotators compared OMR and P&P on 100 matched prompts, yielding 1,300 judgments for this question.

OMR 49.69%Tie 30.08%P&P 20.23%

Physical-plausibility preference against P&P; presentation order was randomized.

Prompt adherence: ties are the largest outcome (45.00%); P&P wins 29.92% and OMR wins 25.08%.

Transfer

Beyond one generator and one benchmark.

These fixed 50-prompt pilots are transfer evidence and point estimates, not full-benchmark results.

Text to video · VideoPhy-2

CogVideoX-5B

32.0%42.0%

Joint success with a separately trained CogVideoX adapter. OMR runs at 1.67× relative time and 30.4 GB peak VRAM.

Image to video · Physics-IQ

Wan2.2-I2V-A14B

26.4431.34

OMR provides a +4.90 single-trajectory gain; combining it with BoN-4 reaches 34.71 (+8.27).

Physics-IQ score gains for OMR and selection methods, including published MAGI-1 video-to-video and vLDM image-to-video context. — Gray rows are published, unmatched mixed-modality WMReward context: MAGI-1 (V2V) and vLDM (I2V). Blue rows are our matched Wan2.2-I2V pilot.

Representation alignment

Why can an adapter connect independently trained models?

The generator latent and world-model target describe the same video under different objectives: one preserves content and motion for synthesis; the other preserves structure for future prediction. OMR learns a task-specific transformation between them without claiming that the two spaces are identical or globally aligned.

Training. The adapter learns V-JEPA token geometry from real-video latents.

During sampling. We test whether it preserves feature geometry and surprise rankings on provisional clean predictions—not raw noisy states—against the exact decode–encode path.

Adapter vs. exact decode–encode path

Representation and surprise-ranking fidelity

Higher is better ↑

Evaluation point	N	Token cosine	Spearman ρ	Kendall τ
Held-out real-video clips	500	0.7506	0.668	0.477
Mid-trajectory clean predictions	200	0.762	0.617	0.446
Matched finished controls	50	0.769	0.624	0.451

This supports adapter use on OMR’s operating distribution. It does not prove exact adapter–encoder equivalence, raw-noise robustness, or causal attribution of the downstream gain.

Scope and outlook

A practical interface, not a guarantee of physical correctness.

Current scope

OMR inherits the blind spots and data biases of its world model. It adds runtime and memory over base sampling, and each new VAE latent space currently requires adapter retraining. The transfer pilots are small and do not include paired prompt-level confidence intervals.

Next questions

Can one adapter serve multiple generators? Can the guidance extend to longer videos? Larger repeated-seed studies and gradient diagnostics would help establish when and why the correction is effective.

Resources

Paper and citation.

Manuscript

Read the complete paper

The PDF includes the main paper and linked supplementary material.

Open PDF

Artifacts

Code and adapter weights

Implementation, configurations, prompt lists, and trained adapters will be linked here.

Coming soon

BibTeX

@inproceedings{nguyentruong2026omr,
  title     = {Off-Manifold Refinement: Guiding Video Generators
               with a Frozen World Model},
  author    = {Hai Nguyen-Truong and Tuan-Anh Vu and Dang Huynh},
  booktitle = {British Machine Vision Conference},
  year      = {2026}
}