Diffusion Models for Reinforcement Learning: A Survey

This repository contains a collection of resources and papers on Diffusion Models for RL.

🚀 Please check out our survey paper Diffusion Models for Reinforcement Learning: A Survey

Table of Contents

• Diffusion Models for Reinforcement Learning: A Survey
  • Table of Contents
  • Papers
    • Offline Reinforcement Learning
    • Online Reinforcement Learning
    • Imitation Learning
    • Trajectory Generation
    • Data Augmentation
  • Citation

Papers

Offline Reinforcement Learning

• Planning with Diffusion for Flexible Behavior Synthesis, ICML 2022. [paper] [code]

• Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning, ICLR 2023. [paper] [code]

• Offline Reinforcement Learning via High-fidelity Generative Behavior Modeling, ICLR 2023. [paper] [code]

• Is Conditional Generative Modeling all you need for Decision-Making?, ICLR 2023. [paper] [code]

• AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners, ICML 2023. [paper] [code]

• Metadiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL, ICML 2023. [paper]

• Hierarchical Diffusion for Offline Decision Making, ICML 2023. [paper] [code]

• Contrastive Energy Prediction for Exact Energy-guided Diffusion Sampling in Offline Reinforcement Learning, ICML 2023. [paper] [code]

• Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks, arXiv 2023. [paper] [code]

• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies, arXiv 2023. [paper] [code]

• Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning, NeurIPS 2023. [paper] [code]

• EDGI: Equivariant Diffusion for Planning with Embodied Agents, NeurIPS 2023. [paper]

• Extracting Reward Functions from Diffusion Models, NeurIPS 2023. [paper]

• Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?, NeurIPS 2023. [paper]

• Reward-Directed Conditional Diffusion: Provable Distribution Estimation and Reward Improvement, NeurIPS 2023. [paper] [code]

• Refining Diffusion Planner for Reliable Behavior Synthesis by Automatic Detection of Infeasible Plans, NeurIPS 2023. [paper] [code]

• SafeDiffuser: Safe Planning with Diffusion Probabilistic Models, arXiv 2023. [paper]

• Efficient Diffusion Policies for Offline Reinforcement Learning, arXiv 2023. [paper] [code]

• MADiff: Offline Multi-agent Learning with Diffusion Models, arXiv 2023. [paper] [code]

• Beyond Conservatism: Diffusion Policies in Offline Multi-agent Reinforcement Learning, arXiv 2023. [paper]

• Fighting Uncertainty with Gradients: Offline Reinforcement Learning via Diffusion Score Matching, CoRL 2023. [paper] [code]

• Value function estimation using conditional diffusion models for control, arXiv 2023. [paper]

• Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learning, arXiv 2023. [paper]

• Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning, arXiv 2023. [paper]

• Diffusion Policies as Multi-Agent Reinforcement Learning Strategies, ICANN 2023. [paper]

• DiffCPS: Diffusion Model based Constrained Policy Search for Offline Reinforcement Learning, arXiv 2023. [paper] [code]

• Score Regularized Policy Optimization through Diffusion Behavior, ICLR 2024. [paper] [code]

• Adaptive Online Replanning with Diffusion Models, arXiv 2023. [paper]

• AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model, arXiv 2023. [paper] [code]

• SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Execution, CVPR 2024. [paper] [website]

• Learning a Diffusion Model Policy from Rewards vis Q-score Matching, arXiv 2023. [paper]

• Simple Hierarchical Planning with Diffusion, ICLR 2024. [paper]

• Reasoning with Latent Diffusion in Offline Reinforcement Learning, ICLR 2024. [paper]

• Efficient Planning with Latent Diffusion, ICLR 2024. [paper]

• Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning, arXiv 2024. [paper]

• DMBP: Diffusion model-based predictor for robust offline reinforcement learning against state observation perturbations, ICLR 2024. [paper] [code]

• Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning, arXiv 2024. [paper] [code]

• Diffusion World Model, arXiv 2024. [paper]

• Diffusion World Models, OpenReview 2024. [paper]

• Policy-Guided Diffusion, arXiv 2024. [paper] [code]

Online Reinforcement Learning

• Policy Representation via Diffusion Probability Model for Reinforcement Learning, arXiv 2023. [paper]

• Boosting Continuous Control with Consistency Policy, arXiv 2023. [paper]

• Diffusion Reward: Learning Rewards via Conditional Video Diffusion, arXiv 2023. [paper] [website] [code]

• ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories, OpenReview 2024. [paper]

Imitation Learning

• Imitating Human Behaviour with Diffusion Models, ICLR 2023. [paper] [code]

• Diffusion Policy: Visuomotor Policy Learning via Action Diffusion, RSS 2023. [paper] [code]

• Goal-Conditioned Imitation Learning using Score-based Diffusion Policies, RSS 2023. [paper] [code]

• To the Noise and Back: Diffusion for Shared Autonomy, RSS 2023. [paper] [code]

• DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics, RAL 2023. [paper]

• Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition, CoRL 2023. [paper] [code]

• XSkill: Cross Embodiment Skill Discovery, CoRL 2023. [paper]

• ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation, CoRL 2023. [paper] [code]

• PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play, CoRL 2023. [paper]

• Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models, CoRL 2023. [paper] [code]

• Multimodal Diffusion Transformer for Learning from Play, CoRL 2023. [paper]

• GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields, CoRL 2023. [paper] [code]

• Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning, arXiv 2023. [paper] [code]

• Diffusion Co-Policy for Synergistic Human-Robot Collaborative Tasks, arXiv 2023. [paper] [code]

• Compositional Foundation Models for Hierarchical Planning, NeurIPS 2023. [paper] [code]

• Generating Behaviorally Diverse Policies with Latent Diffusion Models, NeurIPS 2023. [paper]

• NoMaD: Goal Masking Diffusion Policies for Navigation and Exploration, arXiv 2023. [paper] [code]

• Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models, arXiv 2023. [paper]

• Imitation Learning from Purified Demonstrations, arXiv 2023. [paper]

• Planning as In-Painting: A Diffusion-Based Embodied Task Planning Framework for Environments under Uncertainty, arXiv 2023. [paper]

• Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learning, arXiv 2024. [paper]

• 3D Diffusion Policy, arXiv 2024. [paper] [website] [code]

• Large-Scale Actionless Video Pre-Training via Discrete Diffusion for Efficient Policy Learning, arxiv 2024. [paper] [website]

• SculptDiff: Learning Robotic Clay Sculpting from Humans with Goal Conditioned Diffusion Policy, arXiv 2024. [paper] [website] [code]

• Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation, ICRA 2024. [paper] [website]

Trajectory Generation

• MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model, arXiv 2022. [paper] [code]

• Human Motion Diffusion Model, ICLR 2023. [paper] [code]

• Executing your Commands via Motion Diffusion in Latent Space, CVPR 2023. [paper] [code]

• MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesis, CVPR 2023. [paper] [code]

• ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model, ICCV 2023. [paper] [code]

• MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion, CVPR 2023. [paper]

• Learning Universal Policies via Text-Guided Video Generation, NeurIPS 2023. [paper]

• EquiDiff: A Conditional Equivariant Diffusion Model For Trajectory Prediction, arXiv 2023. [paper]

• Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models, IROS 2023. [paper] [code]

• EDMP: Ensemble-of-costs-guided Diffusion for Motion Planning, arXiv 2023. [paper] [code]

• Sampling Constrained Trajectories Using Composable Diffusion Models, IROS 2023. [paper]

• DiMSam: Diffusion Models as Samplers for Task and Motion Planning under Partial Observability, arXiv 2023. [paper]

• Conditioned Score-Based Models for Learning Collision-Free Trajectory Generation, NeurIPSW 2022. [paper]

• Video Language Planning, arXiv 2023. [paper] [code]

• Learning to Act from Actionless Video through Dense Correspondences, arXiv 2023. [paper] [code]

• Learning Interactive Real-World Simulators, arXiv 2023. [paper]

• DNAct: Diffusion Guided Multi-Task 3D Policy Learning, arXiv 2024. [paper] [website]

Data Augmentation

• Scaling Robot Learning with Semantically Imagined Experience, RSS 2023. [paper]

• GenAug: Retargeting behaviors to unseen situations via Generative Augmentation, RSS 2023. [paper] [code]

• Synthetic Experience Replay, NeurIPS 2023. [paper] [code]

• World Models via Policy-Guided Trajectory Diffusion, arXiv 2023. [paper]

• Distilling Conditional Diffusion Models for Offline Reinforcement Learning through Trajectory Stitching, arXiv 2024. [paper]

• DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching, arXiv 2024. [paper]

• Flow to Better: Offline Preference-based Reinforcement Learning via Preferred Trajectory Generation, ICLR 2024. [paper] [code]

Citation

@article{zhu2023diffusion,
  title={Diffusion Models for Reinforcement Learning: A Survey},
  author={Zhu, Zhengbang and Zhao, Hanye and He, Haoran and Zhong, Yichao and Zhang, Shenyu and Yu, Yong and Zhang, Weinan},
  journal={arXiv preprint arXiv:2311.01223},
  year={2023}
}