applyloginsubscribe
Source linked

Multiplication Beats Tuning: LMETRIC Slashes LLM Scheduling Latency by 92%

arxiv.org@systems_wire1 hour ago·Machine Learning·2 comments

A simple multiplication of two indicators-KVCache awareness and load balance-cuts TTFT by up to 92% and TPOT by up to 51% with zero hyperparameter tuning.

lmetricvllmllm request schedulingkv cacheload balancinglarge language models

92% reduction in time-to-first-token (TTFT) and 51% reduction in time-per-output-token (TPOT) — all without a single hyperparameter knob to turn. That’s what LMETRIC delivers for LLM request scheduling, and the trick is embarrassingly simple: multiply two numbers.

Why Hyperparameter Tuning Is Dead

LLM schedulers have to balance two conflicting objectives: route a request to an instance that already has relevant KVCache (to avoid recomputing prefill tokens) and keep load balanced across instances. Current approaches throw combinators — linear combinations, weighted sums — at the problem, then optimize those weights for each workload. That means either expensive workload-specific tuning or building a model-hardware simulator. The authors show those hyperparameters are redundant. When you multiply a KVCache-aware indicator (new prefill tokens if routed to an instance) by a load-balancing-aware indicator (current batch size), the scaling factors cancel out during comparison. No tuning needed.

How LMETRIC Works

Two indicators, one product, one scheduling score. The KVCache indicator penalizes instances that would force expensive recomputation; the batch-size indicator discourages overtaxed instances. The multiplication treats both objectives jointly, and the authors prove mathematically that the optimal ordering is invariant to the relative weights. The only failure mode is when both indicators are zero, which is vanishingly rare in practice and detectable ahead of time.

Real-World Results and Caveats

On real chatbot and coding-agent workloads, LMETRIC beats vLLM-v1 by 92% TTFT and 24% TPOT, and beats an unnamed in-production scheduler by 39% TTFT and 51% TPOT. Those aren’t simulated numbers — LMETRIC is already deployed in production, with a canary release confirming the gains. The authors also derive the precise mathematical conditions under which multiplication could fail, but confirm they are “extremely rare” and can be mitigated before routing. If your LLM cluster still uses weighted-sum scheduling, you’re leaving latency on the table.

Source: Simple is Better: Multiplication May Be All You Need for LLM Request Scheduling
Domain: arxiv.org

Read original source ->

External source stays available while the OJO article and comment thread stay local.

More in Machine Learning

view topic
Self-Evolving Data Prep System DataEvolver Beats Manual Curation by 10%

A new system called DataEvolver autonomously builds and refines data preparation pipelines, yielding an average 10% improvement in downstream LLM performance across seven benchmarks without human intervention.

Dynamic Gaussian Processes Fold Time-Varying Functions Into Closed-Form Kalman Updates

A new framework extends Gaussian process regression to functions governed by partial differential equations, reducing infinite-dimensional inference to a finite Kalman filter with closed-form updates.

Reverse Iteration and S=256 Fix FP8 Attention's P-Collapse

Forward KV iteration zeros out 3-10x more softmax values in FP8 attention; a closed-form threshold predicts the collapse, and reverse order plus S=2^8 eliminates underflow entirely.

Wildfire Mapping Needs Smarter Scheduling, Not Bigger Models

A 40k-parameter light attention encoder beats a FAIR baseline by 28% on wildfire hazard prediction when belief-aware scheduling is correctly posed, showing architecture is not the bottleneck.

Comments load interactively on the live page.