AdaRoPE: Not All Attention Heads Should Rotate and Scale Equally
Improving long-context language modeling by personalizing rotary position embeddings for every attention head.
Rotary Position Embedding (RoPE) is widely adopted in Transformers to encode positional information, yet standard implementations enforce a uniform frequency schedule and scaling across all attention heads. Using simplified retrieval tasks and length generalization scenarios, we ...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
AdaRoPE shifts transformer architecture toward head-specific positional encoding, revealing that a uniform approach to rotary embeddings is an unnecessary constraint. By enabling learnable frequencies for each attention head, this method significantly boosts performance in long-context tasks while preserving critical short-term retrieval accuracy.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.