Overview
Shipping fewer KV bytes is neither automatically faster nor automatically safe. Fixed codec costs create a link- and size-dependent crossover, and model-dependent KV magnitudes cap the admissible ratio. We turn measured quality, transport, and memory profiles into a deployment verdict: a feasible operating point, fp16, or explicit infeasibility.
Abstract
KV cache compression makes two promises to a distributed LLM serving system: a compressed cache preserves model output, and shipping fewer bytes is faster. This paper shows that both promises fail unless checked against measured quantities. Fewer bytes are not automatically faster: codec, metadata, and consumer overhead can dominate small transfers, so a compressed path beats fp16 transport only above a link- and size-dependent crossover, and on a measured same-GPU path it never pays at all. Fewer bytes are not automatically safe either: aggressive quantization can violate a quality target — uniform INT4 collapses a 3B model while remaining PPL-neutral at 125M scale — so the admissible compression ratio is model-dependent.
We therefore propose a measurement-calibrated runtime framework that decides when compression is worth deploying by accounting quality, latency, and memory jointly: (i) an adaptive INT4/INT8/FP16 codec enforces a writer-verified per-write reconstruction budget, checked on the actual packed representation at every write; (ii) a fixed-cost-aware transport ledger accounts codec, metadata, transfer, and consumer costs and yields a closed-form crossover; (iii) a profile-based deployment resolver returns the highest-compression feasible operating point or explicit infeasibility from measured profiles and operator constraints. The resolver selects certified B=0.2 under a 1% PPL-degradation SLO and B=0.5 under 3%, admits a uniform-INT4 point only for the smallest model at a 5% SLO, and prefers fp16 on the same-GPU path. The strongest result is negative: compression may not pay — its benefit is link- and size-dependent, model-dependent KV magnitudes bound the achievable ratio, and no single compression ratio is universally deployable.
Video
Method
From “pick a ratio” to “resolve a contract”.
Instead of choosing a compression ratio and checking quality afterwards, we start from the operator’s constraints and resolve them into a deployable operating point.
-
Error-bounded adaptive codec
$$\text{mode}(b)=\begin{cases}\text{INT4}, & e_4(b)\le B\\ \text{INT8}, & e_8(b)\le B\\ \text{FP16}, & \text{otherwise}\end{cases}$$Each block takes the cheapest precision whose error fits the budget B. Every write is verified on its actual packed representation.
-
Fixed-cost-aware transport ledger
$$S^{*}_{16} = \frac{\Delta\alpha + \Delta T_c}{\beta_{\text{link}}\,(1 - 1/r)}$$Codec, metadata, transfer, and consumer costs are counted together, which gives a closed-form crossover chunk size per link.
-
Measured-profile deployment resolver
Given the operator’s quality, latency, and memory constraints, the resolver returns the highest-compression feasible point, fp16, or explicit infeasibility. Any bounded-error codec plugs in through three curves: error, latency, and bytes.
Crossover
Fewer bytes ≠ faster.
Fixed codec costs dominate small transfers, so compression pays only above a crossover chunk size that depends on the link. Drag the chunk size and switch links.
Quality
Fewer bytes ≠ safe.
WikiText-2 perplexity degradation per model and codec. Pick a quality SLO: the admissible ratio changes from model to model.
Resolver
Constraints in, operating point out.
Set the quality, latency, and memory constraints for a 2 MiB KV chunk. The resolver returns the highest-compression point that satisfies all three, or tells you not to compress.
Future Work
- More links. Measure the crossover on NVLink, RDMA/InfiniBand, and inter-node networks.
- Pipelined transfers. Extend the ledger to overlapped codec, transfer, and decode.
- Larger models and tasks. Calibrate quality on larger models, long-context workloads, and downstream tasks.
- Native serving path. Run decode on vLLM’s own attention kernels and report TTFT, inter-token latency, and throughput under concurrency.
- From KV error to output quality. Connect the per-write bound to attention-output and task-level quality.
- Online resolver. Adapt the operating point online with INT2/INT3 modes and per-layer budgets.
BibTeX
@inproceedings{fan2026kvpay,
title = {When Does {KV} Cache Compression Pay? A Quality- and Cost-Aware
Runtime Framework for Cloud {LLM} Serving},
author = {Fan, Yuchen},
booktitle = {Proceedings of the International Conference on Green, Pervasive,
and Cloud Computing (GPC)},
year = {2026}
}