Enhancing Human Feedback Reliability in Preference-Based Reinforcement Learning
Date:
Ni Mu’s PhD proposal defense. The research vision is to build preference-based reinforcement learning (PbRL) agents that reliably align with human intentions, by systematically addressing the problem that human preference feedback are often inaccurate, ambiguous, or inconsistent in practice.
The proposal decomposes the unreliability of human feedback into three hierarchical causes, and proposes corresponding solutions:
- Behavioral Indistinguishability (Bottom Layer): When candidate trajectory segments are too similar, annotators resort to random guessing. Solution: Skill-driven PbRL (S-EPOA), which uses unsupervised skill discovery to pretrain diverse, distinguishable behaviors and actively selects high-contrast queries for human feedback.
- Judgment Ambiguity (Middle Layer): Even when behaviors differ, humans may struggle to judge which is better. Solution: Contrastive learning-based PbRL (CLARIFY), which learns a structured embedding space to quantify query clarity, and employs a rejection-sampling mechanism to filter out ambiguous comparisons and prioritize crisp ones.
- Multi-Objective Conflicts (Top Layer): Real-world tasks require trade-offs (e.g., safety vs. efficiency) that cannot be captured by a single scalar preference. Solution: Preference-based Multi-Objective RL (PbMORL), which extends the Bradley-Terry model with weight vectors to represent multi-objective preferences and learns conditional policies that cover the Pareto frontier.
Preliminary results for all three directions have been published or accepted at IJCAI 2025, ICML 2025, NeurIPS 2025, and IEEE T-ASE, with applications to robotics, autonomous driving, and energy systems.