AI Research

    Mitigating Cognitive Bias in RLHF by Altering Rationality

    Adapting reward models to account for human cognitive biases through dynamic rationality weighting

    How can we make models robust to even imperfect human feedback? In reinforcement learning from human feedback (RLHF), human preferences over model outputs are used to train a reward model that assigns scalar values to responses. Because these rewards are inferred from pairwise co...

    AC

    ArXiv CS.AI

    Source

    2 min
    AI Analysis

    Generated insight by OnAirToday AI

    Current RLHF methods treat human feedback as inherently rational, which ignores the systemic cognitive biases that contaminate training data. By dynamically adjusting the rationality parameter, this research enables models to effectively filter out flawed human judgments and improve overall alignment.

    Powered by AI • Not a substitute for human analysisMore Insights →
    Read the full story at ArXiv CS.AI →
    Share
    Filed under:AI Research

    Comments

    Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.

    0/1000