RoCo-ACE: Rollout-Conditioned Online Distillation for Retention-Aware Knowledge Injection
A new distillation objective that balances domain-specific learning with the preservation of core model knowledge.
Knowledge injection updates pretrained MLLMs with new factual or domain-specific knowledge, but fitting full authoritative answers can cause drift in non-updated behavior. Online distillation mitigates this drift by training on model-generated rollouts, yet uniform reference-cond...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
RoCo-ACE addresses the persistent challenge of catastrophic forgetting in large language models by refining how new information is integrated without overwriting existing capabilities. This method provides a more surgical approach to knowledge injection, ensuring that models remain accurate while preserving their foundational intelligence.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.