AI Research

    Chorus: Shared N-gram Embeddings with Layer-wise Adaptation

    Learned vectors for short token sequences can add capacity at low computational cost.

    Language models can gain capacity at little computational cost from n-gram embedding tables: large tables of learned vectors for short sequences of consecutive tokens, which are looked up and added to the model's input or to its intermediate layers.

    AC

    ArXiv CS.CL

    Source

    2 min
    ArXiv CS.CL
    AI Analysis

    Generated insight by OnAirToday AI

    N-gram embedding tables offer a way to increase a language model’s representational capacity without much added computation. Adding their learned vectors at the input or intermediate layers gives models another mechanism to use short token sequences.

    Powered by AI • Not a substitute for human analysisMore Insights →
    Read the full story at ArXiv CS.CL →
    Share
    Filed under:AI Research

    Comments

    Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.

    0/1000