Nexus: Depth-Adaptive KV-Cache Splicing and Retrieval-Decoupled Tool Routing for Agentic LLMs on Unified Memory
Reducing tool-use latency in agentic LLMs through decoupled routing and adaptive KV-cache splicing.
Agentic large language models (LLMs) on the Model Context Protocol (MCP) re-encode verbose tool schemas every turn, so prefill - quadratic in sequence length - dominates time-to-first-token (TTFT) as the tool registry grows. Nexus's primary lever is to decouple routing from the s...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
Nexus addresses the critical bottleneck of tool-use latency in agentic LLMs by decoupling schema-heavy prefill from token generation. This efficiency gain enables significantly faster responses in complex environments without sacrificing the structural integrity of attention mechanisms.
Get the day's AI news in one short email. Free.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.