Retrofitting Recurrent Depth into a Pretrained Language Model: Installation, Extrapolation, Transfer, and Retention at Two Parameter Budgets
New research shows how retrofitting recurrent blocks into pretrained models unlocks deeper reasoning and efficiency.
A dense, pretrained language model can be retrofitted with recurrent depth and learn an iterative latent transition that persists after outcome-only annealing. Qwen2.5-0.5B-Instruct is split into a Prelude, a weight-tied Recurrent Block, and a Coda, with an identity-preserving on...
ArXiv CS.CL
Source
Generated insight by OnAirToday AI
Retrofitting recurrent depth into existing language models enables a more efficient way to perform deep reasoning without the massive computational overhead of increasing static model depth. This approach demonstrates that models can learn iterative, reusable logic paths that significantly outperform traditional scratchpad-trained counterparts at extended depths.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.