於行動邊際運算（MEC, mobile edge computing）系統中，行動用戶可藉由將待計算的任務（task）卸載（offload）予鄰近的伺服器來達到節省功耗與延遲的的目標。然而，卸載時的傳輸也會增加額外的功耗與延遲。當有多個使用者共享伺服器的資源時，資源分配的方式尤為重要。本論文中，我們考慮了一個有多個運算存取點（CAP, computation access points）與多個用戶裝置（UE, user equipment）的無線邊際卸載系統。UE的卸載決策、傳輸功率、與CAP的運算資源都需要被適當分配以降低系統成本（cost）。為此，我們列出了一個卸載決策問題以及一個資源分配問題，並提出多種以不同方法設計的演算法求解。

　　我們首先提出了以深度Q網路（DQN, deep Q network）設計的方法。UE端以DQN學習最佳期望決策（expected best offloading decision），在CAP端使用DQN學習傳輸功率與運算資源的最佳分配方式。同時，UE決策時不需要知道通道狀態資訊（CSI, channel state information），且個別CAP只需要知道選擇卸載給它的UE的通道係數。因此，可以大量減少資訊交換所需的通訊負擔（overhead）。藉由設計適當的回報值（reward）與更新係數，我們的方法可以利用賽局理論（game theory）證明其能收斂至卸載決策的局部最佳點（local optimal）。然而，由於DQN的輸出維度有限，在越大的通訊系統中，取樣過後的資源變數將會造成演算法嚴重的效能減退。因此使用DQN的做法擴展性（scalability）將較為受限。

　　為了克服DQN輸出維度對演算法效能所造成的限制，我們用凸優化（convex optimization）演算法取代處理資源分配的DQN，並提出一個混合式方法（hybrid approach）。在UE端以DQN學習卸載決策，藉此保留以DQN方法收斂到卸載決策局部最佳點的優勢，CAP端則以凸優化求解最佳資源分配以避免對資源變數取樣所造成的效能損失。根據模擬結果，我們所提出的混合式方法在數種不同大小的網路中，相比完全以DQN設計的方法，在降低成本的效果上有相當大幅的改善。除此之外，由於只使用DQN的方法需要等待CAP與UE的DQN係數都達到收斂，而混合式方法只需要等待UE端的DQN係數收斂，所以後者也能有較短的收斂時間。然而此兩種方法都是讓UE的DQN學習在特定網路拓樸（network topology）之下的局部最佳卸載決策。因此在UE的位置以及數量都會隨著時間改變的時候，只要網路拓樸改變，局部最佳卸載決策就須重新被學習。雖然系統成本依然可以隨著卸載決策的收斂而逐漸降低，但在未達收斂以前的成本高峰（peak）仍會使長期平均成本提高，這也因此是一個仍待解決的問題。

　　為了要改善混合式方法在動態網路中（dynamic network）的效果，我們提出了一個強健階層式學習方法（robust hierarchical learning approach）。我們在CAP端新增了一個深度神經網路（DNN, deep neural network），並以UE的相關資訊來預測此UE的DQN在收斂之後的輸出，此輸出我們亦稱為Q值向量（Q-value vector）。在網路拓樸改變時，各UE可以從鄰近的CAP端下載一個稱為Q偏向量（Q-bias vector）的Q值向量預測值。同時，UE會以自己的DQN學習Q偏向量與實際Q值向量之間的差異。UE會以補償過後的Q偏向量，亦即Q偏向量與其DQN輸出的和來選擇卸載決策。藉由從更接近區域最佳卸載測的次佳點開始收斂，卸載決策可以更快達到收斂，系統成本高峰也可以因此降低。模擬結果證明此兩點優勢，系統的長期累積成本也因為此兩項優勢而被大幅改善。

In mobile edge computing (MEC) networks, the mobile users save the local computational energy and latency via offloading their tasks to the computational servers in their proximity. Nonetheless, the task transmission also leads to additional energy consumption and latency. Particularly, proper resource management is necessary when multiple users share the servers' resources. In this work, we consider a MEC network where multiple user equipments (UEs) are served by multiple computation access points (CAPs). The offloading decision, transmission power, and computational resource at each CAP should be allocated to minimize the system cost. Specifically, we formulate an offloading decision-making problem and a resource allocation problem. Several distributed approaches that implement different methodology are proposed to address the two problems.

Firstly, we propose a deep-Q-network-based (DQN-based) approach. The DQN at each UE learns the expected best offloading decision, while the DQN at each CAP learns the allocation of the transmission power and computational resource. Since the output dimension of a DQN should be finite, the continuous resource variables are discretized to form finite combinations of resource allocation. Notably, the channel state information (CSI) is not required at the UEs for the decision-making, while each CAP allocates resources with the channel coefficients of merely the UEs that offload to it. Therefore, heavy communication overhead for the information exchange is avoided. Moreover, we prove the convergence of the DQN approach to a local optimal with the designed award and updating principles. Nonetheless, the discretization of the resource variables severely deteriorates the performance of the DQN-based approach when the size of the network increases. In other words, the scalability of the DQN-based approach is limited.

To overcome the DQN's output dimension limitation, we re-design the resource allocation method with convex optimization methodology and propose a hybrid approach. Specifically, the discrete offloading decisions are learned with the DQNs at the UEs, while each CAP allocates the continuous resource variables with a convex optimizer. By this means, we guarantee the convergence of the joint offloading decision and avoid the performance loss due to the discretization. According to the simulation result, the hybrid approach outperforms the pure DQN-based approach significantly in different network settings. Furthermore, the hybrid approach converges faster since the pure DQN-based approach requires extra time for the DQNs at both the UEs and CAPs to converge. Notably, both the hybrid and pure DQN-based approaches implement DQNs at the UEs to learn the best offloading decision for specific network topologies. In other words, while considering a more practical scenario where the locations and the number of UEs vary with time, the DQNs have to re-learn from scratch whenever the topology changes. Although the cost still reduces eventually along with the convergence of the DQNs, the painful recurring cost peaks are a potential issue for dynamic networks.

To improve the performance of the hybrid approach when considering dynamic networks, we further propose a robust hierarchical learning approach. An additional deep neural network (DNN) is applied at each CAP to predict the output of a UE's DQN with the UE information. When the topology changes, each UE could download an approximated Q-value vector, which is referred to as the Q-bias vector, from a nearby CAP. Meanwhile, each UE also learns the difference between the Q-bias vector and the actual Q-value vector with its DQN. Hence, its offloading decisions will be made according to the compensated biased Q-value vector, which is the sum of the Q-bias vector and the outputs of the UE's DQN. In this way, the offloading decisions could converge from a near-local optimal point faster and thus suppress the cost peaks. The simulation results demonstrate that the hierarchical learning approach reduces the magnitude of the cost peaks and enhances the convergence of the algorithm, thus offering lower accumulated cost.

[1]“Cisco Annual Internet Report - Cisco Annual Internet Report (2018–2023) WhitePaper.” [Online]. Available: https://www.cisco.com/c/en/us/solutions/collateral/executive-perspectives/annual-internet-report/white-paper-c11-741490.html
[2]M. R. Palattella, M. Dohler, A. Grieco, G. Rizzo, J. Torsner, T. Engel, and L. Ladid,“Internet of Things in the 5G era: Enablers, architecture, and business models,”IEEEJournal on Selected Areas in Communications, vol. 34, no. 3, pp. 510–527, Mar. 2016,conference Name: IEEE Journal on Selected Areas in Communications.
[3]D. Agrawal, S. B. Mane, A. Pacharne, and S. Tiwari, “IoT based augmented reality systemof human heart: An android application,” in2018 2nd International Conference on Trendsin Electronics and Informatics (ICOEI), May 2018, pp. 899–902.
[4]Z. Lv, T. Yin, X. Zhang, H. Song, and G. Chen, “Virtual reality smart city based onwebvrgis,”IEEE Internet of Things Journal, vol. 3, no. 6, pp. 1015–1024, 2016.
[5]T. Soyata, R. Muraleedharan, C. Funai, M. Kwon, and W. Heinzelman, “Cloud-Vision: Real-time face recognition using a mobile-cloudlet-cloud acceleration architecture,” in2012 IEEESymposium on Computers and Communications (ISCC), Jul. 2012, pp. 000059–000066.
[6]W. Trappe, R. Howard, and R. S. Moore, “Low-energy security: Limits and opportunitiesin the Internet of Things,”IEEE Security Privacy, vol. 13, no. 1, pp. 14–21, Jan. 2015.87
[7]R. Buyya, C. S. Yeo, and S. Venugopal, “Market-oriented cloud computing: Vision, hype,and reality for delivering IT services as computing utilities,” in2008 10th IEEE InternationalConference on High Performance Computing and Communications, 2008, pp. 5–13.
[8]M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies, “The case for vm-based cloudletsin mobile computing,”IEEE Pervasive Computing, vol. 8, no. 4, pp. 14–23, 2009.
[9]C. You, K. Huang, and H. Chae, “Energy efficient mobile cloud computing powered bywireless energy transfer,”IEEE Journal on Selected Areas in Communications, vol. 34,no. 5, pp. 1757–1771, 2016.
[10]L. Yang, J. Cao, H. Cheng, and Y. Ji, “Multi-user computation partitioning for latencysensitive mobile cloud applications,”IEEE Transactions on Computers, vol. 64, no. 8, pp.2253–2266, 2015.
[11]Z. Sanaei, S. Abolfazli, A. Gani, and R. Buyya, “Heterogeneity in mobile cloud computing:Taxonomy and open challenges,”IEEE Communications Surveys and Tutorials, vol. 16,no. 1, pp. 369–392, 2014.
[12]K. Kumar and Y. Lu, “Cloud computing for mobile users: Can offloading computation saveenergy?”Computer, vol. 43, no. 4, pp. 51–56, 2010.
[13]M. Chiang and T. Zhang, “Fog and iot: An overview of research opportunities,”IEEEInternet of Things Journal, vol. 3, no. 6, pp. 854–864, 2016.
[14]N. Abbas, Y. Zhang, A. Taherkordi, and T. Skeie, “Mobile edge computing: A survey,”IEEE Internet of Things Journal, vol. 5, no. 1, pp. 450–465, Feb. 2018.
[15]Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge comput-ing: The communication perspective,”IEEE Communications Surveys Tutorials, vol. 19,no. 4, pp. 2322–2358, 2017.88
[16]P. Porambage, J. Okwuibe, M. Liyanage, M. Ylianttila, and T. Taleb, “Survey on multi-access edge computing for internet of things realization,”IEEE Communications SurveysTutorials, vol. 20, no. 4, pp. 2961–2991, 2018.
[17]C. You, K. Huang, H. Chae, and B. Kim, “Energy-efficient resource allocation for mobile-edge computation offloading,”IEEE Transactions on Wireless Communications, vol. 16,no. 3, pp. 1397–1411, 2017.
[18]H. Guo, J. Liu, J. Zhang, W. Sun, and N. Kato, “Mobile-edge computation offloading forultradense iot networks,”IEEE Internet of Things Journal, vol. 5, no. 6, pp. 4977–4988,Dec. 2018.
[19]Z. Ning, P. Dong, X. Kong, and F. Xia, “A cooperative partial computation offloadingscheme for mobile edge computing enabled Internet of Things,”IEEE Internet of ThingsJournal, vol. 6, no. 3, pp. 4804–4814, Jun. 2019.
[20]T. Q. Dinh, J. Tang, Q. D. La, and T. Q. S. Quek, “Offloading in mobile edge computing:Task allocation and computational frequency scaling,”IEEE Transactions on Communica-tions, vol. 65, no. 8, pp. 3571–3584, Aug. 2017.
[21]T. X. Tran and D. Pompili, “Joint task offloading and resource allocation for multi-servermobile-edge computing networks,”IEEE Transactions on Vehicular Technology, vol. 68,no. 1, pp. 856–868, Jan. 2019.
[22]V. Francois-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau, “An introduc-tion to deep reinforcement learning,”Foundations and Trends in Machine Learning, vol. 11,no. 3-4, pp. 219–354, 2018.89
[23]K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, “Deep ReinforcementLearning: A Brief Survey,”IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26–38,Nov. 2017. [Online]. Available: http://ieeexplore.ieee.org/document/8103164/
[24]M. Min, L. Xiao, Y. Chen, P. Cheng, D. Wu, and W. Zhuang, “Learning-based computa-tion offloading for IoT devices with energy harvesting,”IEEE Transactions on VehicularTechnology, vol. 68, no. 2, pp. 1930–1941, Feb. 2019.
[25]X. Chen, H. Zhang, C. Wu, S. Mao, Y. Ji, and M. Bennis, “Optimized computation offloadingperformance in virtual edge computing systems via deep reinforcement learning,”IEEEInternet of Things Journal, vol. 6, no. 3, pp. 4005–4018, Jun. 2019.
[26]X. Xiong, K. Zheng, L. Lei, and L. Hou, “Resource allocation based on deep reinforcementlearning in iot edge computing,”IEEE Journal on Selected Areas in Communications, vol. 38,no. 6, pp. 1133–1146, 2020.
[27]Y. Liu, H. Yu, S. Xie, and Y. Zhang, “Deep reinforcement learning for offloading and re-source allocation in vehicle edge computing and networks,”IEEE Transactions on VehicularTechnology, vol. 68, no. 11, pp. 11158–11168, 2019.
[28]H. Guo and J. Liu, “Collaborative computation offloading for multiaccess edge computingover fiber-wireless networks,”IEEE Transactions on Vehicular Technology, vol. 67, no. 5,pp. 4514–4526, May 2018.
[29]T. Q. Dinh, Q. D. La, T. Q. S. Quek, and H. Shin, “Learning for computation offloadingin mobile edge computing,”IEEE Transactions on Communications, vol. 66, no. 12, pp.6353–6367, Dec. 2018.
[30]S. Boyd and M. Grant, “CVX: Matlab software for disciplined convex programming | CVXResearch, Inc.” [Online]. Available: http://cvxr.com/cvx/90
[31]H. Fawzi, J. Saunderson, and P. A. Parrilo, “Semidefinite approximations of the matrixlogarithm,”Foundations of Computational Mathematics, vol. 19, no. 2, p. 259–296, Mar2018.
[32]S. Herbert and D. Marculescu, “Variation-aware dynamic voltage/frequency scaling,” in2009 IEEE 15th International Symposium on High Performance Computer Architecture,Feb. 2009, pp. 301–312.
[33]S. Bi and Y. J. Zhang, “Computation rate maximization for wireless powered mobile-edgecomputing with binary computation offloading,”IEEE Transactions on Wireless Commu-nications, vol. 17, no. 6, pp. 4177–4190, Jun. 2018.
[34]J. Liu and Q. Zhang, “Offloading schemes in mobile edge computing for ultra-reliable lowlatency communications,”IEEE Access, vol. 6, pp. 12825–12837, 2018.
[35]T. Q. Dinh, Q. D. La, T. Q. S. Quek, and H. Shin, “Learning for computation offloadingin mobile edge computing,”IEEE Transactions on Communications, vol. 66, no. 12, pp.6353–6367, Dec. 2018.
[36]J. Li, H. Gao, T. Lv, and Y. Lu, “Deep reinforcement learning based computation offloadingand resource allocation for MEC,” in2018 IEEE Wireless Communications and NetworkingConference (WCNC), Apr. 2018, pp. 1–6.
[37]T. Burd and R. W. Brodersen, “Processor design for portable systems,”Journal of VLSISignal Processing, vol. 13, pp. 203–222, Aug. 1996.
[38]A. P. Chandrakasan, S. Sheng, and R. W. Brodersen, “Low power CMOS digital design,”IEEE Journal of Solid-State Circuits, vol. 27, pp. 473–484, 1995.91
[39]Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, “A survey on mobile edge comput-ing: The communication perspective,”IEEE Communications Surveys Tutorials, vol. 19,no. 4, pp. 2322–2358, 2017.
[40]Q. D. Lã, Y. H. Chew, and B.-H. Soong,Potential Game Theory: Applications in RadioResource Allocation. Springer International Publishing, May 2016.
[41]Z. Zhang, F. R. Yu, F. Fu, Q. Yan, and Z. Wang, “Joint offloading and resource alloca-tion in mobile edge computing systems: An actor-critic approach,” in2018 IEEE GlobalCommunications Conference (GLOBECOM), 2018, pp. 1–6.
[42]K. Shen and W. Yu, “Fractional programming for communication systems–Part I: Powercontrol and beamforming,”IEEE Transactions on Signal Processing, vol. 66, no. 10, pp.2616–2630, May 2018.
[43]F. Palacios-Gomez, L. Lasdon, and M. Engquist, “Nonlinear Optimization by SuccessiveLinear Programming,”Management Science, vol. 28, no. 10, pp. 1106–1120, Oct. 1982.
[44]W. Li, T. Chang, C. Lin, and C. Chi, “Coordinated beamforming for multiuser MISO in-terference channel under rate outage constraints,”IEEE Transactions on Signal Processing,vol. 61, no. 5, pp. 1087–1103, May 2013.