聯邦式學習是一種分散式機器學習模式，參與訓練的用戶端在訓練過程中只提 供訓練模型的更新參數，保障了用戶端的資料隱私性的問題，並且個用戶端能 共同進行模型的訓練，提高了資料的多元性及豐富性。然而，共同學習的模式 固然有好處，在非獨立同分布的資料集和用戶端運算及傳輸資源分配不同的異 質性環境下，容易造成非最佳解的訓練結果。現今主要研究聯邦式學習的論文 多注重在探討如何在非獨立同分布的資料集提高模型準確度，鮮少有討論同時 考量最佳會訓練模型時間集準確度的論文，本論文探討了在聯邦式學習的架構 如何在異質性的環境下得到模型訓練時間集準確度的最佳解。為了達到這個目 標，我們的演算法主要分三個階段；第一步我們挑選對準確度貢獻程度較高的 用戶端參與訓練，並在第二個階段依據用戶端的運算資源將用戶端分成數個階 級，同樣階級的用戶端共同訓練，最後是藉由排除不合條件的用戶端在訓練時 間集準確度做優化的機制。我們的實驗在非獨立同分布的資料集 MNIST 和 CIFAR-10 上比起現存的演算法都有較高效率的表現，在時間上減少了 0.5 倍的 時間同時增加的 10% 的準確度。 關鍵字 : 聯邦式學習、資料異質性、資源異質性、高效能資源運用、邊緣運算

Federated Learning is a distributed learning paradigm that the clients contribute model updates in the training process without compromising data privacy issues. At the same time, elevate the accuracy performance based on a variety of training model updates from different clients. Even though it has benefited from collaborative training, it can be sub-optimal results caused by non-IID datasets and resource inefficient caused by additional computation and communication costs of clients. The existing significant works have focused on achieving statistical performance. However, sacrifice the system efficiency. This work examines the use of federated learning among distributed clients with heterogeneous data and resources. We proposed a federated learning algorithm that achieves optimal training model time and accuracy. Our algorithm consists of three parts: selecting clients depending on the statistical improving to the accuracy performance and dividing clients into different training speed levels according to their resources information; the last part is optimizing the result by dropping clients. Our experiments on the non-IID MNIST and CIFAR-10 dataset have demonstrated the effectiveness of improving the global model’s resource performance and accuracy performance at the same time, the proposed algorithm compared to existing schemes that random selection of clients. Keywords - Federated Learning, Data heterogeneity, Resource efficiency, Edge

[1] C. Tankard, “What the gdpr means for businesses,” Network Security, vol. 2016, no. 6, pp. 5–8, 2016.
[2] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017.
[3] A. Hard, K. Rao, R. Mathews, S. Ramaswamy, F. Beaufays, S. Augenstein, H. Eichner, C. Kiddon, and D. Ramage, “Federated learning for mobile keyboard prediction,” arXiv preprint arXiv:1811.03604, 2018.
[4] I. Castiglioni, L. Rundo, M. Codari, G. Di Leo, C. Salvatore, M. Interlenghi, F. Gallivanone, A. Cozzi, N. C. D’Amico, and F. Sardanelli, “Ai applications to medical images: From machine learning to deep learning,” Physica Medica, vol. 83, pp. 9–24, 2021.
[5] T. Nishio and R. Yonetani, “Client selection for federated learning with heterogeneous resources in mobile edge,” in ICC 2019-2019 IEEE International Conference on Communications (ICC), pp. 1–7, IEEE, 2019.
[6] A. M. Elbir, B. Soner, and S. Coleri, “Federated learning in vehicular networks,” arXiv preprint arXiv:2006.01412, 2020.
[7] S. AbdulRahman, H. Tout, A. Mourad, and C. Talhi, “Fedmccs: multicriteria client selection model for optimal iot federated learning,” IEEE Internet of Things Journal, vol. 8, no. 6, pp. 4723–4735, 2020. 34
[8] G. Wang, C. X. Dang, and Z. Zhou, “Measure contribution of participants in federated learning,” in 2019 IEEE International Conference on Big Data (Big Data), pp. 2597–2604, IEEE, 2019.
[9] R. Yu and P. Li, “Toward resource-efficient federated learning in mobile edge computing,” IEEE Network, vol. 35, no. 1, pp. 148–155, 2021.
[10] S. K. Shyn, D. Kim, and K. Kim, “Fedccea: A practical approach of client contribution evaluation for federated learning,” arXiv preprint arXiv:2106.02310, 2021.
[11] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” arXiv preprint arXiv:1812.06127, 2018.
[12] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” arXiv preprint arXiv:1907.02189, 2019.
[13] M. Mohri, G. Sivek, and A. T. Suresh, “Agnostic federated learning,” in International Conference on Machine Learning, pp. 4615–4625, PMLR, 2019.
[14] A. Katharopoulos and F. Fleuret, “Not all samples are created equal: Deep learning with importance sampling,” in International conference on machine learning, pp. 2525–2534, PMLR, 2018.
[15] Z. Chai, A. Ali, S. Zawad, S. Truex, A. Anwar, N. Baracaldo, Y. Zhou, H. Ludwig, F. Yan, and Y. Cheng, “Tifl: A tier-based federated learning system,” in Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, pp. 125–136, 2020.
[16] Y. Jin, L. Jiao, Z. Qian, S. Zhang, S. Lu, and X. Wang, “Resource-efficient and convergence-preserving online participant selection in federated learning,” in 2020 IEEE 40th International Conference on Distributed Computing Systems (ICDCS), pp. 606–616, IEEE, 2020. 35
[17] T. B. Johnson and C. Guestrin, “Training deep models faster with robust, approximate importance sampling,” Advances in Neural Information Processing Systems, vol. 31, pp. 7265–7275, 2018.
[18] A. Katharopoulos and F. Fleuret, “Biased importance sampling for deep neural network training,” arXiv preprint arXiv:1706.00043, 2017. [19] Y. J. Cho, J. Wang, and G. Joshi, “Client selection in federated learning: Convergence analysis and power-of-choice selection strategies,” arXiv preprint arXiv:2010.01243, 2020.
[20] Z. Bai, Z. Zhang, Y. Zhu, and X. Jin, “Pipeswitch: Fast pipelined context switching for deep learning applications,” in 14th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 20), pp. 499–514, 2020.
[21] F. Lai, X. Zhu, H. V. Madhyastha, and M. Chowdhury, “Oort: Efficient federated learning via guided participant selection,” in 15th {USENIX} Symposium on Operating Systems Design and Implementation ({OSDI} 21), pp. 19–35, 2021.
[22] J. Koneˇcn`y, H. B. McMahan, D. Ramage, and P. Richt´arik, “Federated optimization: Distributed machine learning for on-device intelligence,” arXiv preprint arXiv:1610.02527, 2016.
[23] K. Bonawitz, H. Eichner, W. Grieskamp, D. Huba, A. Ingerman, V. Ivanov, C. Kiddon, J. Koneˇcn`y, S. Mazzocchi, H. B. McMahan, et al., “Towards federated learning at scale: System design,” arXiv preprint arXiv:1902.01046, 2019.
[24] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019. 36 [25] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al., “Advances and open problems in federated learning,” arXiv preprint arXiv:1912.04977, 2019.
[26] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020.
[27] C. Xie, S. Koyejo, and I. Gupta, “Asynchronous federated optimization,” arXiv preprint arXiv:1903.03934, 2019.
[28] F. Haddadpour and M. Mahdavi, “On the convergence of local descent methods in federated learning,” arXiv preprint arXiv:1910.14425, 2019.
[29] H. Wang, K. Sreenivasan, S. Rajput, H. Vishwakarma, S. Agarwal, J.-y. Sohn, K. Lee, and D. Papailiopoulos, “Attack of the tails: Yes, you really can backdoor federated learning,” arXiv preprint arXiv:2007.05084, 2020.
[30] L. Zhu and S. Han, “Deep leakage from gradients,” in Federated learning, pp. 17– 31, Springer, 2020.
[31] N. Rodr´ıguez-Barroso, E. Mart´ınez-C´amara, M. Luz´on, G. G. Seco, M. A. Ve- ´ ganzones, and F. Herrera, “Dynamic federated learning model for identifying adversarial clients,” arXiv preprint arXiv:2007.15030, 2020.
[32] P. Zhao and T. Zhang, “Stochastic optimization with importance sampling for regularized loss minimization,” in international conference on machine learning, pp. 1–9, PMLR, 2015.
[33] H. Ferdowsi, S. Jagannathan, and M. Zawodniok, “An online outlier identification and removal scheme for improving fault detection performance,” IEEE transactions on neural networks and learning systems, vol. 25, no. 5, pp. 908– 919, 2013. 37
[34] Z. Chai, H. Fayyaz, Z. Fayyaz, A. Anwar, Y. Zhou, N. Baracaldo, H. Ludwig, and Y. Cheng, “Towards taming the resource and data heterogeneity in federated learning,” in 2019 {USENIX} Conference on Operational Machine Learning (OpML 19), pp. 19–21, 2019.
[35] T. Huang, W. Lin, W. Wu, L. He, K. Li, and A. Y. Zomaya, “An efficiencyboosting client selection scheme for federated learning with fairness guarantee,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 7, pp. 1552– 1564, 2020.
[36] J. Xu and H. Wang, “Client selection and bandwidth allocation in wireless federated learning networks: A long-term perspective,” IEEE Transactions on Wireless Communications, vol. 20, no. 2, pp. 1188–1200, 2020.
[37] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning, pp. 5132–5143, PMLR, 2020.
[38] C. T. Dinh, N. H. Tran, M. N. Nguyen, C. S. Hong, W. Bao, A. Y. Zomaya, and V. Gramoli, “Federated learning over wireless networks: Convergence analysis and resource allocation,” IEEE/ACM Transactions on Networking, vol. 29, no. 1, pp. 398–409, 2020. 38