Outlier-Free 4-Bit & FP8 LLM Systems: Hadamard Invariance (QuaRot), Stiefel Rotations (SpinQuant) & Blockwise FP8
In LLM Inference & Speculative Decoding and Sparse Mixture-of-Experts (MoE), we saw how modern serving and training pipelines are governed by the Roofline ratio of HBM bandwidth to Tensor Core compute. Dropping weights, activations, and the KV-cache from BF16 down to INT4 or FP8 immediately quadruples or doubles arithmetic intensity – yet naive low-bit quantization collapses above $6.7\text{B}$ parameters due to emergent activation outliers where a handful of hidden channels spike to $20\text{–}100\times$ the median magnitude.
QuaRot (Ashkboos et al., 2024, arXiv:2404.00456; GitHub) and SpinQuant (Liu et al., 2024, arXiv:2405.16406; GitHub) exploit a clean linear-algebraic symmetry of RMSNorm Transformers: multiplying the residual stream by an orthogonal rotation matrix $H \in \mathrm{O}(d)$ ($H H^\top = I$) and counter-rotating every adjacent weight matrix ($W_{\text{in}}' = W_{\text{in}} H$, $W_{\text{out}}' = H^\top W_{\text{out}}$) leaves the full-precision forward pass mathematically identical while smearing channel-aligned outliers uniformly across all $d$ dimensions.
0. Computational Invariance of Orthogonal Residual Rotations
Recall that RMSNorm normalizes a token vector $x \in \mathbb{R}^d$ strictly by its Euclidean $\ell_2$ norm $\|x\|_2 / \sqrt{d}$ before applying a diagonal elementwise scale $\mathrm{diag}(\gamma)$. Because orthogonal transformations preserve $\ell_2$ norms ($\|x H\|_2 \equiv \|x\|_2$), absorbing $\mathrm{diag}(\gamma)$ into the subsequent linear projection $W$ makes the entire block rotation-invariant:
Because every entry of a normalized Walsh–Hadamard matrix has magnitude $1/\sqrt{d}$, a single channel outlier of magnitude $M$ in $x$ is redistributed into $\pm M/\sqrt{d}$ across all channels – collapsing fourth-moment kurtosis from $\kappa \gt 500$ down to the Gaussian baseline $\kappa \approx 3$!
RoPE) applies $2 \times 2$ block rotations $R_{\Theta, m}$ inside $Q$ and $K$, you cannot fuse a global residual rotation $H$ into $W_Q$ and $W_K$ across the RoPE boundary – instead, $H$ is absorbed on the input side ($W_Q' = H^\top W_Q$), and a fast online Walsh–Hadamard transform is applied to $Q$ and $K$ after RoPE to quantize the KV-cache to 4 bits.
(2) Cayley Stiefel Parameterization in SpinQuant: When optimizing learned rotation matrices $R_1, R_2 \in \mathrm{SO}(d)$ via gradient descent (SpinQuant), projecting onto $\mathrm{O}(d)$ via SVD every step triggers an expensive host-device sync; use the skew-symmetric Cayley map $R = (I - A)(I + A)^{-1}$ ($A = -A^\top$) instead.
1. Runnable PyTorch Reference: Fast Walsh–Hadamard Transform (FWHT) & W4A4 Error Verification
The self-contained PyTorch script below implements the $O(d \log d)$ in-place butterfly Fast Walsh–Hadamard Transform, verifies exact FP64 invariance, and measures the $2.3\times$ MSE reduction under W4A4 symmetric quantization:
💻 Python: Fast Walsh–Hadamard Transform (FWHT) & Outlier-Free W4A4 QuantizationPyTorch
import math
import torch
def fwht_pow2(x: torch.Tensor) -> torch.Tensor:
"""O(d log d) orthogonal Fast Walsh-Hadamard Transform along last dim (Ashkboos et al., arXiv:2404.00456)."""
# x: [..., D] where D is a power of 2
d = x.size(-1)
h = x.clone()
step = 1
while step < d:
h = h.view(*x.shape[:-1], d // (2 * step), 2, step)
a, b = h[..., 0, :], h[..., 1, :]
h = torch.stack((a + b, a - b), dim=-2).reshape(x.shape)
step *= 2
return h / math.sqrt(d)
def quantize_sym_int4(t: torch.Tensor) -> torch.Tensor:
"""Per-token / per-channel symmetric 4-bit quantization to [-8, 7]."""
scale = t.abs().amax(dim=-1, keepdim=True) / 7.0 + 1e-12
return torch.clamp(torch.round(t / scale), -8.0, 7.0) * scale
if __name__ == "__main__":
torch.manual_seed(42)
X = torch.randn(32, 64, dtype=torch.float64) * 0.5
X[:, 7] *= 40.0 # Inject 40x activation outlier spike in channel 7
W = torch.randn(32, 64, dtype=torch.float64) * 0.2
Y_exact = X @ W.T
X_rot, W_rot = fwht_pow2(X), fwht_pow2(W)
Y_rot_exact = X_rot @ W_rot.T
mse_naive = (Y_exact - (quantize_sym_int4(X) @ quantize_sym_int4(W).T)).pow(2).mean().item()
mse_quarot = (Y_exact - (quantize_sym_int4(X_rot) @ quantize_sym_int4(W_rot).T)).pow(2).mean().item()
kurt_before = (X.pow(4).mean() / X.pow(2).mean().pow(2)).item()
kurt_after = (X_rot.pow(4).mean() / X_rot.pow(2).mean().pow(2)).item()
print(f"[PASS] Full-Precision Hadamard Invariance Diff: {(Y_exact - Y_rot_exact).abs().max().item():.2e}")
print(f"[PASS] Activation Kurtosis (Before -> After) : {kurt_before:.1f} -> {kurt_after:.2f}")
print(f"[PASS] W4A4 GEMM MSE (Naive -> QuaRot) : {mse_naive:.4f} -> {mse_quarot:.4f} ({mse_naive / mse_quarot:.1f}x lower error!)")
# Output:
# [PASS] Full-Precision Hadamard Invariance Diff: 1.07e-14
# [PASS] Activation Kurtosis (Before -> After) : 152.8 -> 2.73
# [PASS] W4A4 GEMM MSE (Naive -> QuaRot) : 0.4083 -> 0.1741 (2.3x lower error!)
2. Low-Bit LLM Quantization Schemes: Head-to-Head
| Scheme | SmoothQuant (W8A8) | QuaRot (arXiv:2404.00456, W4A4KV4) | SpinQuant (arXiv:2405.16406) + Block FP8 |
|---|---|---|---|
| Outlier Mitigation Mechanism | Diagonal channel migration $s_j = \max|X_j|^\alpha / \max|W_j|^{1-\alpha}$ | Random orthogonal Walsh–Hadamard $H \in \mathrm{O}(d)$ | Cayley-learned Stiefel rotations $R_1, R_2 \in \mathrm{SO}(d)$ + $1 \times 128$ tile scaling |
| 4-Bit Activation & KV Accuracy | Collapses at 4-bit (outliers migrate to weights) | Retains 99% zero-shot accuracy on 70B models | Closes W4A4KV4 gap to within 2.9 pts of BF16 on Llama-3 |
| Runtime Overhead | Zero (fused into LayerNorm) | 2 online FWHT kernels per block ($O(d \log d)$) | Zero extra overhead for $R_1$ (fused offline into weights) |