利用不同的長度的步數返還值已經被廣泛運用在深度增強式學習，不同長度在更新價值函數上提供了不同的好處，包含了影響價值估計的偏差和變異、收斂速度、代理人的探索行為等。傳統的方式像是TD-lambda透過指數平均方式去結合長度不同的步數返還值，把平均後的結果當作價值更新的目標。然而結合成單一個目標會犧牲了各自的優點。
為了解決這個問題，我們提出了Mixture Bootstrapped DQN (MB-DQN)，MB-DQN基於Bootstrapped DQN的架構上而在每個Bootstrapped head使用了各自長度的步數返還值。MB-DQN能保持不同目標值之間的異質性而這是只有單一個目標值的方法所無法達到的。我們首先在迷宮環境上面呈現出我們的動機，接著我們把MB-DQN訓練在Atari上展現出相比於原本方式，他優化的幅度，同時也對於不同配置的MB-DQN做更進一步的分析。

The concept of utilizing multi-step returns for updating value functions has been adopted in deep reinforcement learning (DRL) for a number of years. Updating value functions with different backup lengths provides advantages in different aspects, including bias and variance of value estimates, convergence speed, and exploration behavior of the agent. Conventional methods such as TD~($\lambda$) leverage these advantages by using a target value equivalent to an exponential average of different step returns. Nevertheless, integrating step returns into a single target sacrifices the diversity of the advantages offered by different step return targets. To address this issue, we propose Mixture Bootstrapped DQN (MB-DQN) built on top of bootstrapped DQN, and uses different backup lengths for different bootstrapped heads. MB-DQN enables heterogeneity of the target values that is unavailable in approaches relying only on a single target value. Hence, it is able to maintain the advantages offered by different backup lengths. In this paper, we first discuss the motivational insights through a simple maze environment. To validate the effectiveness of MB-DQN, we perform experiments on the Atari 2600, and demonstrate the performance improvement of MB-DQN over several baseline methods. We further provide ablation studies to examine the impacts of different design configurations of MB-DQN.

[1] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. In Proceedings of International Conference on Machine Learning (ICML), volume 80, pages 1787–1796, 2018.
[2] Brett Daley and Christopher Amato. Reconciling λ – returns with experience replay. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 1133–1142. 2019.
[3] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, pages 529–533, 2015.
[4] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In Proceedings of International Conference on Machine Learning (ICML), pages 1995–2003, 2016.
[5] HadovanHasselt,ArthurGuez,andDavidSilver.Deepreinforcementlearningwith double q-learning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), page 2094–2100, 2016.
[6] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 4026–4034. 2016.
[7] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Gheshlaghi Azar, and David Sil- ver. Rainbow: Combining improvements in deep reinforcement learning. In Pro- ceedings of AAAI Conference on Artificial Intelligence (AAAI), pages 3215–3222, 2018.
[8] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998.
[9] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous meth- ods for deep reinforcement learning. In Proceedings of International Conference on Machine Learning (ICML), pages 1928–1937, 2016.
[10] GabrielBarth-Maron,MatthewW.Hoffman,DavidBudden,WillDabney,DanHor- gan, Dhruva TB, Alistair Muldal, Nicolas Heess, and Timothy P. Lillicrap. Dis- tributed distributional deterministic policy gradients. In Proceedings of Interna- tional Conference on Learning Representations (ICLR), 2018.
[11] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep rein- forcement learning. In Proceedings of International Conference on Learning Rep- resentations (ICLR), 2016.
[12] Kristopher De Asis, J. Fernando Hernandez-Garcia, G. Zacharias Holland, and Richard S. Sutton. Multi-step reinforcement learning: A unifying algorithm. In Proceedings of AAAI Conference on Artificial Intelligence, pages 2902–2909, 2018.
[13] ArtemijAmiranashvili,AlexeyDosovitskiy,VladlenKoltun,andThomasBrox.An- alyzing the role of temporal differencing in deep reinforcement learning. In Proceed- ings of International Conference on Learning Representations (ICLR), 2018.
[14] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), pages 4148–4152, 2015.
[15] Tommi S. Jaakkola, Michael I. Jordan, and Satinder P. Singh. On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation, 6(6): 1185–1201, 1994.
[16] MarcG.Bellemare,SriramSrinivasan,GeorgOstrovski,TomSchaul,DavidSaxton, and Re ́mi Munos. Unifying count-based exploration and intrinsic motivation. In Proceedings of Advances in Neural Information Processing Systems (NIPS), pages 1471–1479, Dec. 2016.