CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries
Date:
This seminar presents CLARIFY (ICML 2025), which identified a critical bottleneck in PbRL: when two trajectory segments are behaviorally similar, human annotators not only struggle to judge but also produce noisy labels that corrupt the reward model.
Our solution is a contrastive learning framework that learns a trajectory embedding space, where distance directly reflects human judgment clarity. We designed two contrastive losses: an ambiguity loss that clusters ambiguous queries and separates clear ones, and a quadrilateral loss that enforces geometric consistency of preference rankings across multiple queries. Once the embedding space is learned, we use rejection sampling over embedding distances to actively select queries that are crisp for humans and informative for the model.
On MetaWorld and DMControl benchmarks, and with both synthetic teachers and real human annotators, CLARIFY outperforms prior offline PbRL methods in annotation clarity, label accuracy, and final policy performance.
Related paper: CLARIFY (ICML 2025) · 📄 Slides (PDF)