AI Research
Mitigating Cognitive Bias in RLHF by Altering Rationality
Adapting reward models to account for human cognitive biases through dynamic rationality weighting
How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise co...
AC
ArXiv CS.AI
Source
2 min
ArXiv CS.AIRead the full story at ArXiv CS.AI
AI Analysis
Generated insight by OnAirToday AI
Current RLHF methods treat human feedback as inherently rational, which ignores the systemic cognitive biases that contaminate training data. By dynamically adjusting the rationality parameter, this research enables models to effectively filter out flawed human judgments and improve overall alignment.
Powered by AI • Not a substitute for human analysisMore Insights →
Share
Filed under:AI Research
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.