S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF
A new hierarchical reward framework enhances alignment stability by balancing token-level precision with sentence-based structure.
Reinforcement learning from human feedback (RLHF) with preference-based reward models often exhibits unstable training dynamics. A key contributing factor is that standard RLHF relies on a single sequence-level scalar reward, which is propagated to token-level policy updates and ...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
S2T-RLHF addresses the instability of existing models by treating sentences as the primary unit of reward allocation to mitigate noise in token-level updates. This hierarchical approach offers a more robust framework for preference alignment, effectively balancing granular supervision with the inherent uncertainty of human feedback.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.