AI Research
Chorus: Shared N-gram Embeddings with Layer-wise Adaptation
Learned vectors for short token sequences can add capacity at low computational cost.
Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers.
AC
ArXiv CS.CL
Source
2 min
ArXiv CS.CL
AI Analysis
Generated insight by OnAirToday AI
N-gram embedding tables offer a way to increase a language model’s representational capacity without much added computation. Adding their learned vectors at the input or intermediate layers gives models another mechanism to use short token sequences.
Powered by AI • Not a substitute for human analysisMore Insights →
Share
Filed under:AI Research

Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.