隨著5G網路快速的發展，為了達到更好的可靠性及表現，發展低延遲及高頻寬
為其中的目標，而用戶以及基地台的數量在未來的發展自然也大量的增加，這
便導致了儘管在跟以前網路在相同的範圍之下，遭受到的干擾以及對Quality of
Experience (QoE)的影響相較之前更加劇烈，更加難以保持甚至提升用戶的要求。
再來，小覆蓋範圍的基地台自然導致了更多的換手發生，頻繁以及不佳的換手選擇
更影響了用戶的體驗嚴重甚至斷線，因此，考慮一個對用戶更有效率的換手方式至
關重要。本文中，由於無法獲得完整的環境資訊，不少資訊為未知，例如：排程
器，所以透過使用RL來跟環境互動使幫助用戶做出換手決定，實驗結果也表明在
平均QoE的部分我們的方法可以比傳統的方法更好。

With the rapid development of 5G networks, to improve better reliability and performance, low latency and high bandwidth are the key features of 5G, and the number
of user equipmetes (UEs) and base stations (BSs) increase many times. Compared
to previous networks, although a UE moves in the same area, the interference and
the impact on Quality of Experience (QoE) are more severe than before. It is more
challenging to maintain or even improve QoE for UEs. Furthermore, BSs with a small
coverage area leads to more handovers. Frequent and unnecessary handovers affect
the experience even cause radio link failure. Therefore, it is crucial to consider a more
efficient handover for users. In this thesis, since we do not know all information about
the environment, such as the scheduler, we attempt to propose a handover algorithm
using reinforcement learning to interact with the environment. The simulation result
shows our method outperforms two baselines owes in terms of the average QoE of
UEs.

[1] 3GPP, “Evolved Universal Terrestrial Radio Access (E-UTRA); Radio
Resource Control (RRC); Protocol specification ,” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 36.331, 3 2021, version 16.4.0. [Online]. Available: https://portal.3gpp.org/desktopmodules/Specifications/
SpecificationDetails.aspx?specificationId=2440

[2] K. Da Costa Silva, Z. Becvar, and C. R. L. Frances, “Adaptive hysteresis margin based on fuzzy logic for handover in mobile networks with dense small cells,” IEEE Access, vol. 6, pp. 17 178–17 189, 2018.

[3] K. C. Silva, Z. Becvar, E. H. S. Cardoso, and C. R. Francˆes, “Self-tuning handover algorithm based on fuzzy logic in mobile networks with dense small cells,” in 2018 IEEE Wireless Communications and Networking Conference (WCNC), 2018, pp. 1–6.

[4] Z.-H. Huang, Y.-L. Hsu, P.-K. Chang, and M.-J. Tsai, “Efficient handover algorithm in 5g networks using deep learning,” in GLOBECOM 2020 - 2020 IEEE Global Communications Conference, 2020, pp. 1–6.

[5] Y. Zhao and X. Luo, “Handover mitigation in dense hetnets via bandit arm elimination,” in 2019 IEEE Global Communications Conference (GLOBECOM), 2019, pp. 1–6.

[6] Z. Wang, L. Li, Y. Xu, H. Tian, and S. Cui, “Handover optimization via asynchronous multi-user deep reinforcement learning,” in 2018 IEEE International Conference on Communications (ICC), 2018, pp. 1–6.

[7] Y. Chen, X. Lin, T. Khan, and M. Mozaffari, “Efficient drone mobility support using reinforcement learning,” in 2020 IEEE wireless communications and networking conference (WCNC). IEEE, 2020, pp. 1–6.

[8] M. M. U. Chowdhury, W. Saad, and I. G¨uven¸c, “Mobility management for cellular-connected uavs: A learning-based approach,” in 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020, pp. 1–6.

[9] Y. Koda, K. Nakashima, K. Yamamoto, T. Nishio, and M. Morikura, “Handover management for mmwave networks with proactive performance prediction using camera images and deep reinforcement learning,” IEEE Transactions on Cognitive Communications and Networking, vol. 6, no. 2, pp. 802–816, 2019.

[10] M. Sana, A. De Domenico, E. C. Strinati, and A. Clemente, “Multi-agent deep reinforcement learning for distributed handover management in dense mmwave networks,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 8976–8980.

[11] M. S. Mollel, S. Kaijage, M. Kisangiri, M. A. Imran, and Q. H. Abbasi, “Multiuser position based on trajectories-aware handover strategy for base station selection with multi-agent learning,” in 2020 IEEE International Conference on Communications Workshops (ICC Workshops). IEEE, 2020, pp. 1–6.

[12] X. Bao, W. Adjardjah, A. Okine, W. Zhang, and N. Bao, “Vertical handover scheme for enhancing the qoe in vlc heterogeneous networks,” in 2018 IEEE/CIC International Conference on Communications in China (ICCC), 2018, pp. 437–442.

[13] C. J. Watkins and P. Dayan, “Q-learning,” Machine learning, vol. 8, no. 3-4, pp. 279–292, 1992.

[14] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,
A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.

[15] ITU-T, “The E-model: a computational model for use in transmission planning,” International Telecommunication Union (ITU-T), Recommendation G.107, 6 2015. [Online]. Available: https://www.itu.int/rec/T-REC-G.107

[16] S. Sengupta, M. Chatterjee, and S. Ganguly, “Improving quality of voip streams over wimax,” IEEE transactions on computers, vol. 57, no. 2, pp. 145–156, 2008.

[17] T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband et al., “Deep q-learning from demonstrations,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018.

[18] 3GPP, “3rd Generation Partnership Project; Technical Specification Group Radio Access Network; Study on 3D channel model for LTE (Release 12),” 3rd Generation Partnership Project (3GPP), Technical Specification (TS) 36.873, 12 2017, version 12.7.0. [Online]. Available:
https://portal.3gpp.org/desktopmodules/Specifications/SpecificationDetails.aspx?specificationId=2574