Source linked

QPILOTS Steers Flow Policies to 90% Success Without Retraining

arxiv.org@frontier_wire8 hours ago·Machine Learning·7 comments

New inference-time method projects noisy actions to clean estimates before computing critic gradients, hitting 90% average success on 50 offline-to-online RL tasks and steering frozen VLA models.

qpilotsflow policiesdiffusion policiesreinforcement learningoffline to online rlvision language action model

90% average success across 50 offline-to-online RL tasks: that's what QPILOTS delivers, and it does so without retouching the original policy. The method leaves frozen flow-matching or diffusion policies intact and steers the denoising process at inference time. No distillation, no multi-step backprop through unstable gradients.

Critic Gradients on Clean Action Projections

The core problem with steering diffusion policies is that intermediate noisy actions produce unreliable critic predictions. QPILOTS side-steps this by first projecting the noisy intermediate state to an estimate of the final clean action, then computing the critic gradient on that estimate. This lets the method exploit the critic's action gradient without the numerical instability of backpropagating through the full multi-step denoising chain.

Two variants handle that projection differently. QPILOTS-U uses a fast single-point approximation that trades some accuracy for speed. QPILOTS-M draws differentiable posterior samples via a learned auxiliary network, giving higher fidelity at the cost of a small extra model. Both keep the original policy frozen.

Benchmarks Across 50 Tasks and a Frozen VLA

On a standard offline-to-online RL benchmark suite, QPILOTS achieves the best aggregate performance across 50 tasks with that 90% success rate. That's a direct comparison against methods that discard gradient information, distill into one-step actors, or repeatedly fine-tune the denoising policy. QPILOTS beats all of them without any policy modification.

The same approach also steers a large, frozen, pretrained Vision-Language Action (VLA) foundation model. Across six manipulation tasks in simulation, QPILOTS outperforms or matches prior inference-time steering approaches. For anyone running a large VLA model that's too expensive to fine-tune, this is a practical way to inject task-specific Q-values at test time.

QPILOTS turns the critic from a training-only signal into a runtime controller for diffusion-based policies, opening up test-time adaptation without touching the base model weights.

Source: QPILOTS: Efficient Test-Time Q-Steering for Flow Policies
Domain: arxiv.org

Read original source ->

External source stays available while the OJO article and comment thread stay local.

More in Machine Learning

view topic
New Concentration Bound Cancels Latent Variance in AI Benchmarks Like MMLU

A de Finetti-inspired analysis shows that for zero-sum linear contrasts, the latent mixture term-the main source of uncertainty in exchangeable data-cancels exactly, yielding a tighter Hoeffding-type bound for...

Edge-TSR Exposes 20-30% Benchmark Overstatement for Edge Inference

A continuous roadside perception system on the Jetson Orin Nano shows static benchmarks overstate real-world performance by 20-30%, with temporal stabilization recovering up to 10.16% accuracy.

GroqRack Beats GPUs on Decode Latency but Loses on Batch Scaling

GPUs still own the prefill phase, but GroqRack delivers lower time per output token during decode until batch sizes climb.

SageMaker AI Container Caching Cuts GenAI Scale-Out Latency by Over 50%

Container caching removes the image pull step, cutting end-to-end startup latency from 525 seconds to 258 seconds for a Qwen3-8B model on ml.g6.2xlarge.

Comments load interactively on the live page.