Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
A novel training-free method to compress long prompts and accelerate inference via spectral token aggregation.
Long-prompt inference remains expensive because prefill attention scales quadratically with sequence length. We propose Spectral-LSH, a training-free prompt compression method that operates before the prompt enters the language model. Spectral-LSH approximates the dominant compon...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
Spectral-LSH effectively bypasses the quadratic scaling bottleneck of long-context LLMs by using Krylov subspace projection to compress prompts without training. This approach significantly improves performance at high compression ratios, offering a viable path for handling massive datasets with reduced computational overhead.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.