Ni Mu

Ni Mu

Ph.D. student, Dept. of Automation, Tsinghua University.

Posts by Collection

portfolio

publications

Integrating mechanism and data: Reinforcement learning based on multi-fidelity model for data center cooling control

Published in CAC 2023 (2023 China Automation Congress), 2023

Combines mechanism models with data via a multi-fidelity model for reinforcement-learning-based data-center cooling control.

Recommended citation: Ni Mu, Xiao Hu, Qing-Shan Jia. (2023). "Integrating mechanism and data: Reinforcement learning based on multi-fidelity model for data center cooling control." 2023 China Automation Congress (CAC).
Download Paper

S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning

Published in IJCAI 2025, 2025

To address the indistinguishability of behaviorally similar segments in preference-based RL, S-EPOA uses unsupervised skill discovery to pick segments that differ widely in skill, improving human label efficiency.

Recommended citation: Ni Mu*, Yao Luan*, Yiqin Yang, Bo Xu, Qing-Shan Jia. (2025). "S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning." IJCAI 2025.
Download Paper

CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries

Published in ICML 2025, 2025

In preference-based RL, humans struggle to compare behaviorally similar trajectory segments, which hurts label efficiency. CLARIFY learns a contrastive objective in the trajectory embedding space to select segment pairs that humans can actually compare.

Recommended citation: Ni Mu*, Hao Hu*, Xiao Hu, Yiqin Yang, Bo Xu, Qing-Shan Jia. (2025). "CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries." ICML 2025.
Download Paper

Preference-based Multi-Objective Reinforcement Learning

Published in IEEE Transactions on Automation Science and Engineering (T-ASE), 2025

A theoretically grounded multi-objective preference learning paradigm: optimizing the multi-objective preference reward model is equivalent to solving the convex Pareto front in policy space.

Recommended citation: Ni Mu*, Yao Luan*, Qing-Shan Jia. (2025). "Preference-based Multi-Objective Reinforcement Learning." IEEE Transactions on Automation Science and Engineering.
Download Paper

Safety-Guaranteed Policy Composition Via Generalized Policy Improvement for Autonomous Vehicles

Published in IEEE CASE 2025 (Outstanding WiRA Student Paper Award), 2025

A Generalized-Policy-Improvement-based policy composition with safety and performance guarantees that resolves the performance–safety–sample-efficiency “impossible triangle” for autonomous vehicles. Outstanding WiRA Student Paper Award.

Recommended citation: Ni Mu, Yao Luan, Qing-Shan Jia. (2025). "Safety-Guaranteed Policy Composition Via Generalized Policy Improvement for Autonomous Vehicles." IEEE 21st International Conference on Automation Science and Engineering (CASE 2025).
Download Paper

SunTzu: A Long-Horizon Decision Making Benchmark for Large Language Models

Published in Preprint, 2025

A benchmark for evaluating LLMs long-horizon decision making in StarCraft II, together with a hierarchical agent baseline that self-corrects via expert knowledge and self-improves via supervised fine-tuning.

Recommended citation: Pengbo Shen*, Yaqing Wang*, Ni Mu*, Yao Luan, Runpeng Xie, Senhao Yang, Lexiang Wang, Hao Hu, Shuang Xu, Yiqin Yang, Bo Xu. (2025). "SunTzu: A Long-Horizon Decision Making Benchmark for Large Language Models." Submitted to NeurIPS 2026.
Download Paper

STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning

Published in NeurIPS 2025, 2025

In multi-stage tasks, preference-based RL suffers from stage misalignment: comparing behaviors from different stages is hard for humans and uninformative for policy optimization. STAIR minimizes the temporal distance between segment pairs so comparisons stay within a stage.

Recommended citation: Yao Luan*, Ni Mu*, Yiqin Yang, Bo Xu, Qing-Shan Jia. (2025). "STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning." NeurIPS 2025.
Download Paper

LLM-Empowered Knowledge Graph Construction for Optimization Formalization in Smart Manufacturing

Published in CAC 2025 (2025 China Automation Congress), 2025

Uses a large language model to construct knowledge graphs that formalize optimization problems in smart manufacturing.

Recommended citation: Ni Mu, Zeyuan Liu, Yuhang Zhu, Qing-Shan Jia. (2025). "LLM-Empowered Knowledge Graph Construction for Optimization Formalization in Smart Manufacturing." 2025 China Automation Congress (CAC).
Download Paper

MrCoM: A Meta-Regularized World-Model Generalizing Across Multi-Scenarios

Published in AAAI 2026, 2026

Model-based RL world models generalize poorly across scenarios. MrCoM extracts scenario information via meta-state regularization and aligns model optimization with multi-scenario policy learning via meta-value regularization.

Recommended citation: Xuantang Xiong*, Ni Mu*, Runpeng Xie, Senhao Yang, Yaqing Wang, Lexiang Wang, Yao Luan, Siyuan Li, Shuang Xu, Yiqin Yang, Bo Xu. (2026). "MrCoM: A Meta-Regularized World-Model Generalizing Across Multi-Scenarios." AAAI 2026.
Download Paper

COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space

Published in ICML 2026, 2026

Unsupervised skill discovery often yields task-irrelevant or hazardous behaviors. COLLIE leverages dense unsupervised data to construct a semantically coherent skill latent space, enabling reliable, training-free guidance from sparse online human feedback.

Recommended citation: Yao Luan*, Ni Mu*, Hanfei Ge, Yiqin Yang, Bo Xu, Qing-Shan Jia. (2026). "COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space." ICML 2026.
Download Paper

talks

CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries Permalink

Published:

Seminar on CLARIFY (ICML 2025). We introduced a contrastive learning framework that learns a structured trajectory embedding space to quantify human judgment clarity, coupled with a rejection-sampling query mechanism that actively filters ambiguous comparisons. The method significantly improves both human annotation reliability and offline preference-based RL performance.

Enhancing Human Feedback Reliability in Preference-Based Reinforcement Learning Permalink

Published:

PhD proposal defense on enhancing the reliability of human feedback in Preference-Based RL. The work identifies three fundamental causes of unreliable feedback: behavioral indistinguishability, judgment ambiguity, and multi-objective conflicts; and proposes a three-part technical framework: skill-driven PbRL (S-EPOA), contrastive learning-based PbRL (CLARIFY), and preference-based multi-objective RL (Pb-MORL).

Covering the Pareto Frontier with LLM-Coordinated Interpretable Policy Library Permalink

Published:

Conference talk at IFAC 2026, which proposes a vision of self-designing industrial autonomy, and a paradigm shift from tuning opaque neural weights to searching over explicit code logic. We provide a proof-of-concept: a dual-agent LLM loop to autonomously build an interpretable policy library. On industrial multi-objective tasks, it matches deep RL in 30 iterations (<90 min) with fully transparent, deployment-ready code.

teaching