WarpQuant: Dual-Domain LLM Quantization via Hadamard Rotation and Output-Fisher Sensitivity

Definition: quantize three memory paths separately

Autoregressive decoding streams weights for every token, accumulates a KV cache with context length, and moves activations through each linear operation. WarpQuant does not force one bit-width onto tensors with different distributions and lifetimes.

Design: compress in the rotated domain, recover in the original domain

A deterministic signed Hadamard transform R=HD rotates each weight matrix W. The rotated weights use 128-column groups, a Gaussian Lloyd-Max 3-bit codebook, FP16 scales, and block-GPTQ error feedback into later groups. [1][4][5]

In R16E4H4, R16 restores selected weak columns in BF16, while E4 and H4 store the token embedding and lm_head in group-128 INT4. This connects the 3.55-bpw projection design to the payload of the complete text model.

Selecting recovery columns with Output-Fisher

Recovery priority is not determined by weight error alone. Input activation energy H_X, quantization residual E, and the squared output NLL gradient H_G estimate how strongly each column's error reaches next-token loss. [8]

Here W_base is the block-GPTQ reconstruction mapped back to the original coordinates. The column score used by the selector is defined by the following PyTorch function.

import torch

def output_fisher_score(W, W_base, X, H_G, index_bits=32):
    assert W.ndim == W_base.ndim == 2
    assert X.shape[-1] == W.shape[1]
    assert H_G.shape == (W.shape[0],)

    E = W.float() - W_base.float()
    H_X = X.float().square().mean(dim=0)
    loss_weighted_error = (
        H_G.float()[:, None] * E.square()
    ).sum(dim=0)
    cost_bits = 16 * W.shape[0] + index_bits
    return H_X * loss_weighted_error / cost_bits

Because output directions are not treated uniformly, the budget shifts toward loss-sensitive regions such as v_proj and down_proj. At the same Llama 3 8B payload, WikiText-2 perplexity moves from 7.3953 to 7.3446.

Results: Qwen3.8-27B text backbone

The denominator is 26,895,998,464 text-generation parameters, excluding the vision tower and MTP. Commonsense is the macro average of fixed 1,000-item samples from HellaSwag, WinoGrande, and PIQA.

PPL, ARC, and MMLU use the same llama.cpp evaluation path. GSM8K uses the same first 500 examples and 5-shot prompts for all four models; the table reports lm-evaluation-harness flexible-extract accuracy.

The same format comparison on other models

On Qwen3.5-4B, WarpQuant is 243.6 MB smaller than IQ3_M while improving PPL, ARC, and MMLU. On Llama 3 8B, it is 43.0 MB smaller than IQ3_S and scores higher on ARC, while IQ3_S retains better PPL and MMLU.

KV cache and activations

Across Qwen3.8-27B's 16 full-attention layers, the newest 128 tokens stay in BF16. Older keys use a 3-bit MSE code plus a 1-bit QJL residual after random rotation; values use per-token group-32 INT4. This applies TurboQuant's online inner-product quantization to the KV cache. [9][10][11][12]

Activations use a per-token absolute-maximum scale for inputs to 496 decoder linear modules. The table compares weight-only, KV, A8, and combined paths on the same Qwen3.8-27B R16E4H4 checkpoint and 4,088 WikiText-2 validation tokens. [13][14]

Measurement and reproduction

GGUF baselines use their physical text-model rate and payload. WarpQuant sums packed codes, FP16 scales, weak-column indices and residuals, plus embedding/head group scales. Its evaluation checkpoint is a BF16 carrier containing the same quantized values.

Relationship to prior work