在機器人運動規劃任務中，提供多種不同的軌跡來引導機器人到達給定目標對於其對環境變化的適應能力是很重要的。近期在強化學習領域中的一些研究已提出了可以學習多樣化策略的方法，並在運動規劃任務展示了好的學習成果。這些方法使用了由兩部分獎勵組成的獎勵函數，分別為 goal-reaching reward 以及 diversity reward。由於 distance reward 能提供能在訓練過程中提供豐富的獎勵資訊驅使 agent 前往任務目標，因此經常被用來當作 goal-reaching reward 的部分。但 distance reward 會隨著 agent 跟目標之間的距離不同而有所變化，因此其在跟 diversity reward 結合時，可能會與後者相斥，影響學習表現。在這篇論文中，我們提出使用二元獎勵 (binary reward) 來當作 goal-reaching reward。Binary reward 的穩定及可預測性使我們可以協調 goal-reaching reward 和 diversity reward。我們進而在方法中加入 Hindsight Experience Replay (HER) 技巧來避免稀疏獎勵問題。實驗結果顯示我們的方法可以在運動規劃任務中學習高效的目標可達策略，與先前的多樣化策略學習方法相比，所學到的策略有更高的多樣性及對環境變化的穩健性。

In robot motion planning, providing multiple diverse trajectories to guide the robot to reach a given target is important for its resilience to environmental changes. Recent advances in reinforcement learning (RL) have shown promising results in enabling the agents to learn diverse trajectories in motion planning. The majority of such works employed a two-part reward system consisting of a goal-reaching reward and a diversity reward. For the former, distance rewards are normally adopted for their rich signals to guide the agents closer to the target. The problem is that distance rewards change constantly as the agents progress towards the target and, when combined with the diversity rewards, may conflict with the latter, causing poor learning results. In this paper, we propose to use binary rewards as the goal-reaching rewards. Binary rewards are static and predictable, allowing the diversity rewards to remain stable. The sparsity of binary rewards is resolved with Hindsight Experience Replay (HER). Experimental results show that the proposed method enables the agents to learn efficient goal-reaching policies that are more diverse and robust to environmental changes compared to the prior diverse-policies-learning approaches.

References
[1] C. Voss, M. Moll, and L. E. Kavraki, “A heuristic approach to finding diverse short paths,”
International Conference on Robotics and Automation (ICRA), 2015.
[2] A. Upadhyay, B. Goldfarb, and C. Ekenna, “A topological approach to finding coarsely
diverse paths,” International Conference on Intelligent Robots and Systems (IROS), 2021.
[3] V. Vonásek, R. Pěnička, and B. Kozlíková, “Computing multiple guiding paths for
sampling-based motion planning,” International Conference on Advanced Robotics
(ICAR), 2019.
[4] V. Vonasek and M. Saska, “Increasing diversity of solutions in sampling-based path planning,” Proceedings of the 2018 4th International Conference on Robotics and Artificial
Intelligence, 2018.
[5] Y. Lee, J. Yang, and J. J. Lim, “Learning to coordinate manipulation skills via skill behavior diversification,” International conference on learning representations, 2019.
[6] S. Kumar, A. Kumar, S. Levine, and C. Finn, “One solution is not all you need: Fewshot extrapolation via structured maxent rl,” Advances in Neural Information Processing
Systems, 2020.
[7] H. Ha, J. Xu, and S. Song, “Learning a decentralized multi-arm motion planner,”
arXiv:2011.02608, 2020.
[8] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew,
J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” Advances in
neural information processing systems, 2017.
[9] S. M. LaValle, “Rapidly-exploring random trees: A new tool for path planning,” Technical
report 98-11, 1998.
[10] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars, “Probabilistic roadmaps for
path planning in high-dimensional configuration spaces,” IEEE transactions on Robotics
and Automation, 1996.
[11] C. Florensa, Y. Duan, and P. Abbeel, “Stochastic neural networks for hierarchical reinforcement learning,” arXiv:1704.03012, 2017.
[12] Z.-W. Hong, T.-Y. Shann, S.-Y. Su, Y.-H. Chang, T.-J. Fu, and C.-Y. Lee, “Diversity-driven
exploration strategy for deep reinforcement learning,” Advances in neural information
processing systems, 2018.
[13] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller, “Learning an embedding space for transferable robot skills,” International Conference on Learning Representations, 2018.
[14] S. Mohamed and D. Jimenez Rezende, “Variational information maximisation for intrinsically motivated reinforcement learning,” Advances in neural information processing systems, 2015.
[15] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learning
skills without a reward function,” International Conference on Learning Representations,
2018.
[16] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum
entropy deep reinforcement learning with a stochastic actor,” International conference on
machine learning, 2018.
[17] T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal value function approximators,”
International conference on machine learning, 2015.
[18] O. Michel, “Cyberbotics ltd. webots™: professional mobile robot simulation,” International Journal of Advanced Robotic Systems, 2004.