LOOPQ: QUANTIZATION FOR LOOPED LANGUAGE MODELS
Overview
Looped language models increase effective depth by repeatedly applying shared Transformer blocks. This reuse saves parameters but makes post-training quantization sensitive to how computation evolves across loops. The same block encounters different hidden-state distributions, quantized states are passed across loop boundaries, and errors introduced early can accumulate during later iterations. LOOPQ studies these three sources of degradation and develops a quantization framework around them.
The method retains a shared quantized backbone. Loop-aware activation scaling accommodates changes in activation magnitude, while a sharing-sensitivity score identifies a small subset of transformation groups that benefit from loop-specific treatment. Lightweight transition adapters correct hidden states at loop boundaries. Trajectory-aware calibration then coordinates these components by matching full-precision boundary states and final output distributions, addressing error propagation over the complete recurrent computation.
With 4-bit weights and 4-bit activations, the manuscript reports a 67.8% average relative improvement in five-task mean accuracy across four backbones, compared with the strongest static quantization baseline for each comparison. Perplexity experiments further assess language-modeling quality across eight backbone–dataset pairs. The results examine how targeted loop-dependent adaptation can improve low-precision inference while retaining the parameter-sharing advantage of looped models.
Citation
Rui Fang, Hsi-Wen Chen, and Ming-Syan Chen. (2026). "LOOPQ: QUANTIZATION FOR LOOPED LANGUAGE MODELS." Submitted to ICLR 2027; under review.