AI Research

    Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese

    Byte-Pair Encoding tokenization is statistically efficient for vocabulary compression, but semantically blind to structured technical entities, fragmenting physical quantities, numbers, units, and symbolic expressions into lexically arbitrary subwords. We present TOTEN, a knowled...

    AC

    ArXiv CS.AI

    Source

    2 min
    ArXiv CS.AI
    AI Analysis

    Generated insight by OnAirToday AI

    This paper introduces TOTEN, a novel tokenization framework that addresses the limitations of statistical methods by incorporating a formal ontology of engineering entities. By grounding tokenization in knowledge, TOTEN achieves superior accuracy in representing physical quantities and technical notations in Brazilian Portuguese, paving the way for more precise AI applications in scientific and engineering domains.

    Powered by AI • Not a substitute for human analysisMore Insights →
    Read the full story at ArXiv CS.AI →
    Share
    Filed under:AI Research

    Comments

    Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.

    0/1000