Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese
Byte-Pair Encoding tokenization is statistically efficient for vocabulary compression, but semantically blind to structured technical entities, fragmenting physical quantities, numbers, units, and symbolic expressions into lexically arbitrary subwords. We present TOTEN, a knowled...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
This paper introduces TOTEN, a novel tokenization framework that addresses the limitations of statistical methods by incorporating a formal ontology of engineering entities. By grounding tokenization in knowledge, TOTEN achieves superior accuracy in representing physical quantities and technical notations in Brazilian Portuguese, paving the way for more precise AI applications in scientific and engineering domains.

Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.