KV cache compression · vector quantization · serving diagnostics

TorbuQuant

TorbuQuant is a Python package for LLM KV-cache compression, vector quantization, cache accounting, attention diagnostics, and runtime integration work for HuggingFace and vLLM.

The repository is built around a measured goal: reduce stored KV bytes, preserve model behavior under explicit gates, and improve serving capacity when KV memory, context length, or batch size is the limiting factor.

Quickstart Read the design API reference

TQ4 page tensor [num_blocks, block_size, num_kv_heads, slot_bytes]

slot 17 block = slot // block_size row = slot % block_size

slot row page[block, row, kv_head, :]

K region bytes V region bytes

K packed K indices fp32 norm

V packed V indices fp32 norm

K bytesceil(D * k_bits / 8) + 4B V bytesceil(D * v_bits / 8) + 4B slot_bytesK + V, optional power-of-two padding

slot slot row decode

This TQ4 page path uses the same norm-coded row packer for K and V; other V formats in the library may use scale and zero metadata.

Diagnostic and serving routes Separated by design

Packed K/V accounting Metadata included

Qwen A/B evaluation Measured comparison

What Makes This Repository Different

TorbuQuant is documented as a research implementation and a serving-system prototype at the same time. The documentation therefore records:

• which formulas come from the paper,
• which paths are diagnostic,
• which paths own packed cache storage,
• which byte formulas are layout estimates,
• which measurements have been run,
• which vLLM pieces are still contracts rather than live hooks.

That separation is intentional. It prevents a storage result from being misread as a serving result.

1. Quantize Rotate, normalize, codebook-encode, and pack tensor rows.

2. Store Keep historical K/V in compact uint8 page or row layouts.

3. Attend Use diagnostic or packed-reference attention paths by intent.

4. Measure Report memory, quality, latency, throughput, and raw outputs.

Current Status

Area	Status
Lloyd-Max codebooks	Implemented for exact Beta-sphere and Gaussian approximation.
Rotations	QR and RHT are implemented.
MSE vector quantization	Implemented with packed index storage.
QJL residual	Implemented for vector and inner-product experiments.
KV formats	K16, K8, K4 and V8, V4, V3, V2 helpers exist.
Diagnostic attention	Dense, dequantized, direct-QK, and packed-V reference paths exist.
Triton decode	Format-specific kernels exist for selected non-paged paths.
TQ packed pages	Page storage, PyTorch row packing, and paged decode reference exist.
HuggingFace	Diagnostic DynamicCache wrappers and Qwen capture helpers exist.
vLLM	Metadata, page geometry, registry, runtime selection, and verification helpers exist.
vLLM live backend	Not yet wired as a live vLLM attention backend in this repository.
Qwen A/B	Scripted A/B evaluation exists for HF diagnostic runs.

What This Documentation Covers

The site documents the code currently present in this repository. Where a serving feature is not wired end to end, the page says so explicitly. It does not import or depend on code under other_implemnetations/.

Install

Environment setup, optional packages, and validation commands.

Quickstart

Small commands that exercise quantization, cache accounting, and evaluation paths.

Design

Route ownership, data contracts, diagnostic boundaries, and serving constraints.

API reference

Generated module reference from the repository's Python source.

Reader Paths

Reader	Suggested path
Researcher	Math, Algorithms, Limits.
Runtime engineer	Design, Architecture, vLLM Usage.
Model evaluator	Qwen A/B, Performance Notes, Troubleshooting.
Contributor	Developer Guide, Contributing, API Reference.