An Effective World Action Model
with One-Step Action Generation
Structured imagination. One-step action.
Future-aware control for autonomous driving and embodied intelligence.
The Hong Kong Polytechnic University
Inference uses the current observation, task conditioning, and ego state for driving. No future video rollout.
NAVSIM PDMS
NAVSIM · compiled · RTX 5090
RoboTwin 2.0 average success
RoboTwin · compiled · RTX 5090
01 / THE IDEA
A better starting point
for action.
Can a world action model produce high-quality actions in a single denoising step?
Direct-policy world action models avoid generating future video at inference, yet still rely on iterative action denoising. Simply reducing the step count leaves too little opportunity to recover coherent actions from unstructured noise.
OSWAM changes the initialization. Compact imagination tokens extract an observation-conditioned action prior from a pretrained Video DiT. A single Action DiT step refines that structured prior into an action sequence. During training, privileged-information distillation transfers future context from a teacher while keeping inference conditioned on the current observation.
Read the full abstract +
Direct-policy World Action Models (WAM) have demonstrated the benefits of decoupling action generation from explicit future visual prediction for efficient inference in autonomous driving and embodied control. However, they still require tens of denoising steps to achieve competitive action quality, incurring substantial latency and hindering real-time deployment. Simply reducing the number of denoising steps will collapse the nonlinear multi-step denoising trajectory into coarse updates, substantially degrading action quality. To reduce the denoising steps while enabling real-time action generation, we present an effective world action model with one-step action generation, namely OSWAM, for autonomous driving and embodied control. We argue that it is the absence of structured action priors in random noise initialization that forces direct-policy WAMs to recover coherent action sequences through iterative denoising. Hence, we construct imagination tokens to initialize the diffusion process, providing physical priors and action-conditioned representations. We further introduce privileged information distillation with future conditioning to enhance future-aware action generation. As a result, OSWAM can efficiently and effectively generate high quality action in just one diffusion step. Our experiments demonstrate that OSWAM achieves comparable or even better results, in terms of both autonomous driving and embodied control, than direct-policy WAM and Joint-WAM that require tens of steps. It achieves approximately 3× speedup over direct-policy WAM baselines and supports real-time compiled inference with latency as low as 40ms on an NVIDIA RTX 5090 GPU.
02 / EXPERIMENTS
Less denoising.
Competitive control.
Evaluated on autonomous driving
and bimanual manipulation.
NAVSIM navtest
Matches SimWAM's PDMS with one action step and a 3.01× speedup.
| Method | Action steps ↓ | PDMS ↑ | Latency (ms) ↓ |
|---|---|---|---|
| DriveVA‡ | 2 | 90.5 | 593* |
| DriveWAM | 10 | 90.1 | 1032 |
| SimWAM† | 10 | 90.3 | 410 |
| Metis | 10 | 89.1 | 440* |
| OSWAM Ours | 1 | 90.3 | 136 |
NVIDIA H20 GPU / Intel Xeon Max 9468 CPU. † Without reinforcement learning. ‡ Without CARLA mix training. * Latency reproduced by the authors. PDMS: Predictive Driver Model Score.
RoboTwin 2.0 50 tasks
2.80× faster than Fast-WAM, with a 1.83 percentage-point decrease in average success.
| Method | Action steps ↓ | Clean (%) ↑ | Randomized (%) ↑ | Average (%) ↑ | Latency (ms) ↓ |
|---|---|---|---|---|---|
| Flash-WAM | 1 | 82.56 | 80.26 | 81.41 | 609 |
| Efficient-WAM | 10 | 86.70 | 85.70 | 86.20 | 568 |
| Fast-WAM | 10 | 91.88 | 91.78 | 91.83 | 397 |
| Light-WAM | — | 76.40 | 76.30 | 76.40 | 107 |
| OSWAM Ours | 1 | 90.04 | 89.96 | 90.00 | 142 |
NVIDIA H20 GPU / Intel Xeon Max 9468 CPU. Average success over 100 trials per task in Clean and Randomized settings. Baseline values are reproduced as reported in the supplied manuscript.
RTX 5090 cached language context
3.39× overall speedup on RoboTwin and 3.32× on NAVSIM before compilation.
| Benchmark | Method | Overall latency (ms) ↓ | Compiled latency (ms) ↓ |
|---|---|---|---|
| RoboTwin 2.0 | Fast-WAM | 323.67 | 73.64 |
| RoboTwin 2.0 | OSWAM | 95.35 | 40.81 |
| NAVSIM | SimWAM | 330.80 | 90.53 |
| NAVSIM | OSWAM | 99.70 | 52.64 |
Single NVIDIA RTX 5090 GPU / Intel Xeon Platinum 8470Q CPU, with cached language context. Compiled inference uses torch.compile and CUDA Graph optimization. These timings use a different setup from the H20 benchmark tables.
03 / VISUALIZATIONS & DEMOS
Robot executions.
Driving predictions.
AUTONOMOUS DRIVING / NAVSIM
Lane following. Left turn. Right turn.
01 Lane following
Original speed · 1×02 Left turn
Original speed · 1×03 Right turn
Original speed · 1×Open-loop predictions from one initial observation · Action: 2 steps · Video: 25 steps. BEV uses recorded map and actor states; teal shows the prediction, gray dashed lines the reference. These samples use a different inference configuration from the one-step benchmark above.
04 / CITE THIS WORK
BibTeX