在第五代行動通訊網絡（5G）興起之後，越來越多的論文專注於軟體定義網絡（SDN），此外，對於即時應用程式需求的快速增長也加深了人們對於具服務品質感知能力 (QoS-aware) 繞送機制的興趣。在本文中，我們提出了一多重應用且具服務品質感知能力的智慧繞送機制。有別於大部分的繞送機制，使用了深度強化學習（DRL）並且搭配具服務品質感知能的多應用獎勵函數。 此方法不僅使決策更加靈活，而且也使訓練出來的模型能更接近真實的用戶體驗。 根據實驗結果，與傳統的最短路徑算法和基於單參數的強化學習式智慧繞送方法相比，我們的方法都可以更有效地優化繞送路徑。

After the rise of the fifth-generation (5G) mobile network, more and more papers focus on software-defined networks (SDN). Moreover, the rapid growth in demand for real-time applications has aroused the interest of QoS-aware routing. In our thesis, we proposed a QoS-aware intelligent routing scheme to support multi-application meanwhile. Different from other works, we use deep reinforcement learning (DRL) with a QoS-aware multi-application reward function. Our method not only makes decision-making more flexible but also trained model closer to the real user experience. According to the experimental results, our method can optimize the routing path more effectively than the traditional shortest path algorithm and single-parameter RL-based routing method.

1. R. Braden, D. Clark, and S. Shenker, “Rfc1633: Integrated services in the internet architecture: an overview,” 1994.
2. S. Blake, D. Black, M. Carlson, E. Davies, Z. Wang, and W. Weiss, “An architecture for differentiated services,” 1998.
3. M. Karakus and A. Durresi, “Quality of service (qos) in software defined networking (sdn): A survey,” Journal of Network and Computer Applications, vol. 80, pp. 200– 218, 2017.
4. [4] O. M. Mon and M. T. Mon, “Quality of service sensitive routing for software defined network using segment routing,” in 2018 18th International Symposium on Commu- nications and Information Technologies (ISCIT), pp. 180–185, IEEE, 2018.
5. H. E. Egilmez and A. M. Tekalp, “Distributed qos architectures for multime- dia streaming over software defined networks,” IEEE Transactions on Multimedia, vol. 16, no. 6, pp. 1597–1609, 2014.
6. H. E. Egilmez, S. T. Dane, K. T. Bagci, and A. M. Tekalp, “Openqos: An open- flow controller design for multimedia delivery with end-to-end quality of service over software-defined networks,” in Proceedings of the 2012 Asia Pacific signal and infor- mation processing association annual summit and conference, pp. 1–8, IEEE, 2012.
7. L. Yanjun, L. Xiaobo, and Y. Osamu, “Traffic engineering framework with machine learning based meta-layer in software-defined networks,” in 2014 4th IEEE Inter- national Conference on Network Infrastructure and Digital Content, pp. 121–125, IEEE, 2014.
8. A. Azzouni, R. Boutaba, and G. Pujolle, “Neuroute: Predictive dynamic routing for software-defined networks,” in 2017 13th International Conference on Network and Service Management (CNSM), pp. 1–6, IEEE, 2017.
9. S.-C. Lin, I. F. Akyildiz, P. Wang, and M. Luo, “Qos-aware adaptive routing in multi- layer hierarchical software defined networks: A reinforcement learning approach,” in 2016 IEEE International Conference on Services Computing (SCC), pp. 25–33, IEEE, 2016.
10. F. Francois and E. Gelenbe, “Optimizing secure sdn-enabled inter-data centre overlay networks through cognitive routing,” in 2016 IEEE 24th International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pp. 283–288, IEEE, 2016.
11. S. Sendra, A. Rego, J. Lloret, J. M. Jimenez, and O. Romero, “Including artificial intelligence in a routing protocol using software defined networks,” in 2017 IEEE International Conference on Communications Workshops (ICC Workshops), pp. 670– 674, IEEE, 2017.
12. G. Stampa, M. Arias, D. S´anchez-Charles, V. Munt´es-Mulero, and A. Cabellos, “A deep-reinforcement learning approach for software-defined networking routing opti- mization,” arXiv preprint arXiv:1709.07080, 2017.
13. C. Yu, J. Lan, Z. Guo, and Y. Hu, “Drom: Optimizing the routing in software-defined networks with deep reinforcement learning,” IEEE Access, vol. 6, pp. 64533–64539, 2018.
14. H. Yao, T. Mai, X. Xu, P. Zhang, M. Li, and Y. Liu, “Networkai: An intelligent network architecture for self-learning control strategies in software defined networks,” IEEE Internet of Things Journal, vol. 5, no. 6, pp. 4319–4327, 2018.
15. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” 2015.
16. M. R. Parsaei, R. Mohammadi, and R. Javidan, “A new adaptive traffic engineering method for telesurgery using aco algorithm over software defined networks,” European Research in Telemedicine/La Recherche Europeenne en Telemedecine, vol. 6, no. 3-4, pp. 173–180, 2017.
17. J. Wang, C. de Laat, and Z. Zhao, “Qos-aware virtual sdn network planning,” in 2017 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), pp. 644–647, IEEE, 2017.
18. J. A. Boyan and M. L. Littman, “Packet routing in dynamically changing networks: A reinforcement learning approach,” in Advances in neural information processing systems, pp. 671–678, 1994.
19. Z. Mammeri, “Reinforcement learning based routing in networks: Review and classification of approaches,” IEEE Access, vol. 7, pp. 55916–55950, 2019.
20. D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “Determinzstic policy gradient algorithms,” 2014.
21. J. Su´arez-Varela, A. Mestres, J. Yu, L. Kuang, H. Feng, A. Cabellos-Aparicio, and P. Barlet-Ros, “Routing in optical transport networks with deep reinforcement learn- ing,” IEEE/OSA Journal of Optical Communications and Networking, vol. 11, no. 11, pp. 547–558, 2019.
22. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in International conference on machine learning, pp. 1889–1897, 2015.
23. M. B. Hossain and J. Wei, “Reinforcement learning-driven qos-aware intelligent rout- ing for software-defined networks,” in 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5, IEEE, 2019.
24. X. You, X. Li, Y. Xu, H. Feng, J. Zhao, and H. Yan, “Toward packet routing with fully-distributed multi-agent deep reinforcement learning,” arXiv preprint arXiv:1905.03494, 2019.
25. S. Fang, Y. Yu, C. H. Foh, and K. M. M. Aung, “A loss-free multipathing solution for data center network using software-defined networking approach,” IEEE transactions on magnetics, vol. 49, no. 6, pp. 2723–2730, 2013.
26. A. Basta, A. Blenk, K. Hoffmann, H. J. Morper, M. Hoffmann, and W. Kellerer, “Towards a cost optimal design for a 5g mobile core network based on sdn and nfv,” IEEE Transactions on Network and Service Management, vol. 14, no. 4, pp. 1061– 1075, 2017.
27. A. Nygren, B. Pfaff, B. Lantz, B. Heller, C. Barker, C. Beckmann, D. Cohn, D. Malek, D. Talayco, D. Erickson, et al., “Openflow switch specification version 1.5. 1,” Open Networking Foundation, Tech. Rep., 2015.
28. J. Moy et al., “Ospf version 2,” 1998.
29. D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. A. Riedmiller, “Deterministic policy gradient algorithms,” in ICML, pp. 387–395, 2014.
30. M. et al., “Human-level control through deep reinforcement learning,” Nature, vol. 518, pp. 529–533, Feb 2015.
31. C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992.
32. C. Ye, H. Ma, X. Zhang, K. Zhang, and S. You, “Survival-oriented reinforcement learning model: An effcient and robust deep reinforcement learning algorithm for autonomous driving problem,” in International Conference on Image and Graphics, pp. 417–429, Springer, 2017.
33. A. Varga, “Omnet++,” in Modeling and tools for network simulation, pp. 35–59, Springer, 2010.
34. P. Sun, J. Li, J. Lan, Y. Hu, and X. Lu, “Rnn deep reinforcement learning for routing optimization,” in 2018 IEEE 4th International Conference on Computer and Communications (ICCC), pp. 285–289, IEEE, 2018.
35. A. Varga, “Inet framework for the omnet++ discrete event simulator,” 2012.
36. N. Brownlee and K. C. Claffy, “Understanding internet traffic streams: dragonflies and tortoises,” IEEE Communications magazine, vol. 40, no. 10, pp. 110–117, 2002.