簡易檢索 / 詳目顯示

回結果列表
研究生： 李藝偉
Li, Yi-wei
論文名稱： 差分隱私聯邦監督式與非監督式學習
Differentially Private Federated Supervised and Unsupervised Learning
指導教授： 祁忠勇
Chi, Chong-Yung
口試委員: 簡仁宗
Chien, Jen-Tzung
鐘偉和
Chung, Wei-Ho
吳仁銘
Wu, Jen-Ming
林家祥
Lin, Chia- Hsiang
學位類別： 博士
Doctor
系所名稱： 電機資訊學院 - 通訊工程研究所
Communications Engineering
論文出版年： 2024
畢業學年度： 112
語文別： 英文
論文頁數： 113
中文關鍵詞： 聯邦學習 、差分隱私 、監督式學習 、非監督式學習 、凸優化
外文關鍵詞： Federated learning, differential privacy, federated supervised learning, federated unsupervised learnin, convex optimization
相關次數： 點閱：1  下載：0 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 聯邦學習（FL）是一種新的範式，它使許多客戶能夠在中央參數伺服器（PS）的協調下共同訓練機器學習模型，同時保持本地數據不直接暴露給任何第三方。有效的聯邦學習算法的發展面臨多個實際挑戰，包括高通信成本、數據異構性和客戶隱私保護。本論文嘗試通過在聯邦監督和非監督學習任務領域進行詳細的理論分析來應對這些挑戰。本論文的主要貢獻分為兩個部分。

    本論文的第一部分探討了一類以凸但非平滑損失函數為特徵的聯邦問題。這些函數在聯邦學習應用中普遍存在，由於其複雜的非平滑性質和通信效率與隱私保護之間相沖突的要求，我們提出了一種新穎的帶有差分隱私（DP）的聯邦隨機 原-對偶演算法，稱為 FedSPD-DP，專為解決非平滑聯的邦學習問題。我們在理論上分析了DP噪聲、本地多步隨機梯度下降（local SGD）和部分客戶參與（PCP）對模型收斂性能的影響。具體而言，我們的理論分析表明，數據採樣策略和PCP可以增強數據隱私，而本地多步SGD可能增加隱私泄漏，揭示了算法通信效率和隱私保護之間的存在折中。實驗結果展示了我們提出的演算法在聯邦分類任務上的實際性能，並證明其相對於現有最先進演算法的優越性，同時驗證了所有的分析結果。

    本論文的第二部分深入研究了聯邦聚類（FedC）問題， 其目標是在中央參數伺服器的協調下，準確地將分佈在眾多客戶端上的未標記數據樣本進行有效劃分，同時考慮數據隱私。儘管這是一個涉及表示簇中心和表示每個數據樣本的簇成員歸屬的實變量的 NP-hard 優化問題，但我們巧妙地將FedC問題重新建模為只有一個凸約束的非凸優化問題，從而得到一個擁有軟聚類解的問題。 然後，提出了一種採用DP技術的新型FedC演算法，稱為DP-FedC，該演算法考慮了部分客戶參與和本地模型多步更新策略。此外，通過對隱私保護和收斂速度的理論分析，特別是對於非獨立分佈的數據（non-i.i.d.）情況，獲得了所提出DP-FedC的各種屬性，這理論上可作為設計聯邦學習系統的指導方針。然後，我們在兩個真實數據集上展示了一些實驗結果，以驗證所提出DP-FedC演算法在聯邦聚類任務上的有效性， 以及相對於一些現有最先進的聯邦聚類算法的優越性，並與所有呈現的分析結果保持一致。

    Federated learning (FL) is a new paradigm that enables many clients to jointly train a machine learning model under the orchestration of a central parameter server (PS) while keeping the local data not directly exposed to any third party. The development of effective FL algorithms faces multiple practical challenges, including high communication costs, data heterogeneity, and clients' privacy protection. This dissertation attempts to deal with these challenges with detailed theoretical analyses in the realm of both federated supervised and unsupervised learning tasks. The main contributions of this dissertation include two parts.

    The first part of this dissertation explores a class of FL problems characterized by convex but non-smooth loss functions. These functions are prevalent in FL applications, posing a challenge due to their intricate non-smooth nature and the conflicting requirements of communication efficiency and privacy protection.
    We propose a novel federated stochastic primal-dual algorithm with differential privacy (DP), referred to as FedSPD-DP, tailored for non-smooth FL problems.
    We theoretically analyze the impact of DP noise, multiple steps of local stochastic gradient descent (local SGD) and partial client participation (PCP) on convergence performance. Specifically, our analysis reveals that the data sampling strategy and PCP can enhance data privacy, whereas a larger number of local SGD steps could increase privacy leakage, revealing a non-trivial tradeoff between algorithm communication efficiency and privacy protection. Experimental results are presented to evaluate the practical performance of the proposed algorithm on classification tasks and demonstrate its superior performance compared to state-of-the-art methods, together with the validation of all the analytical results and properties.

    The second part of this dissertation delves into the federated clustering (FedC) problem, which aims to accurately partition unlabeled data samples distributed over numerous clients into finite clusters under the orchestration of the PS, while taking data privacy into consideration.
    Though it is an NP-hard optimization problem involving real variables denoting cluster centroids and binary variables denoting the cluster membership of each data sample, we judiciously reformulate the FedC problem into a non-convex optimization problem with only one convex constraint, accordingly yielding a soft clustering solution. Then a novel FedC algorithm using DP technique, termed DP-FedC, is proposed in which PCP and local SGD are also considered. Furthermore, various attributes of the proposed DP-FedC are obtained through theoretical analyses of privacy protection and convergence rate, especially for the case of non-identically and independently distributed (non-i.i.d.) data, that ideally serve as the guidelines for the design of the proposed DP-FedC. Then some experimental results on two real datasets are provided to demonstrate the efficacy of the proposed DP-FedC on clustering tasks together with its much superior performance over some state-of-the-art FedC algorithms, and the consistency with all the presented analytical results.

    Chinese Abstract ii Abstract iv Acknowledgments vi List of Figures xi List of Tables xiv List of Notations xv 1 Introduction 1 2 The DP-FL System: A Review 6 2.1 FL System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.2 Preliminaries of DP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.3 The DP-FL System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3.1 Noise Reduction in DP-FL System . . . . . . . . . . . . . . . . . . 11 2.3.2 Noise Reduction by Privacy Amplification . . . . . . . . . . . . . . 12 2.3.3 Noise Reduction by Model Sparsification . . . . . . . . . . . . . . 14 2.3.4 Noise Reduction by Reducing Sensitivity . . . . . . . . . . . . . . 16 2.4 Related Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 3 Differentially Private Federated Supervised Learning via PDM 20 3.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3.2 Proposed FedSPD-DP Algorithm . . . . . . . . . . . . . . . . . . . . . . . 23 3.3 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.2 Privacy Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3.3 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . 29 3.4 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.4.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.4.2 Impact of DP Noise . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.3 Impact of Local SGD . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.4.4 Performance Comparison with State-of-the-Art Works . . . . . . . 35 3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4 Differentially Private Federated Unsupervised Learning 37 4.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.1.1 Centralized k-means Clustering . . . . . . . . . . . . . . . . . . . 37 4.1.2 Federated Clustering . . . . . . . . . . . . . . . . . . . . . . . . . 39 4.1.3 Federated Clustering Model . . . . . . . . . . . . . . . . . . . . . 41 4.2 Proposed DP-FedC Algorithm . . . . . . . . . . . . . . . . . . . . . . . . 42 4.2.1 Update of W and H i . . . . . . . . . . . . . . . . . . . . . . . . . 42 4.2.2 Privacy Concern . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.3 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3.1 Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.3.2 Privacy Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.3.3 Convergence Analysis . . . . . . . . . . . . . . . . . . . . . . . . 48 4.4 Experiment Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4.1 Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 4.4.2 Impact of DP Noise . . . . . . . . . . . . . . . . . . . . . . . . . . 53 4.4.3 Impact of the Number of Participated Clients (K) . . . . . . . . . . 54 4.4.4 Comparison with Existing Distributed Clustering Methods . . . . . 55 4.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 5 Conclusions and Future Works 59 A Proofs of Theorems in Chapter 2 61 A.1 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 B Proofs of Theorems and Lemmas in Chapter 3 63 B.1 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 B.2 Proof of Theorem 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 B.3 Proof of Theorem 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 B.3.1 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 B.3.2 Sketch of the Proof . . . . . . . . . . . . . . . . . . . . . . . . . . 69 B.4 Proof of Key Lemmas in Theorem 4 . . . . . . . . . . . . . . . . . . . . . 74 B.4.1 Proof of Lemma 4 . . . . . . . . . . . . . . . . . . . . . . . . . . 74 B.4.2 Proof of Lemma 5 . . . . . . . . . . . . . . . . . . . . . . . . . . 75 B.4.3 Proof of Lemma 6 . . . . . . . . . . . . . . . . . . . . . . . . . . 76 B.4.4 Proof of Lemma 7 . . . . . . . . . . . . . . . . . . . . . . . . . . 81 B.4.5 Proof of Lemma 8 . . . . . . . . . . . . . . . . . . . . . . . . . . 85 C Proofs of Theorems and Lemmas in Chapter 4 87 C.1 Proof of Lemma 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 C.2 Proof of Theorem 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 C.3 Proof of Theorem 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 C.4 Proofs of Key Lemmas for Theorem 7 . . . . . . . . . . . . . . . . . . . . 96 C.4.1 Proof of Lemma 9 . . . . . . . . . . . . . . . . . . . . . . . . . . 96 C.4.2 Proof of Lemma 10 . . . . . . . . . . . . . . . . . . . . . . . . . . 97 C.4.3 Proof of Lemma 11 . . . . . . . . . . . . . . . . . . . . . . . . . . 98 C.5 Proof of Remark 10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Bibliography 104 Publication List of The Author 112

    [1] T.-H. Chang, M. Hong, H.-T. Wai, X. Zhang, and S. Lu, "Distributed learning in the nonconvex world: From batch data to streaming and beyond," IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 26-38, 2020.
    [2] Y. Wang, Y. Xu, Q. Shi, and T.-H. Chang, "Quantized federated learning under transmission delay and outage constraints," IEEE Journal on Selected Areas in Communications, vol. 40, no. 1, pp. 323-341, 2021.
    [3] S. Wang and T.-H. Chang, "Federated matrix factorization: Algorithm design and application to data clustering," IEEE Trans. Signal Processing, vol. 70, pp. 1625-1640, 2022.
    [4] H. McMahan, E. Moore, D. Ramage, and B. A. y Arcas, "Federated learning of deep networks using model averaging," arXiv preprint arXiv 1602.05629,2016.
    [5] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, "On the convergence of FedAvg on non-IID data," in Proc. International Conference on Learning Representations (ICLR), 2020, pp. 1-26.
    [6] S. Wang, Y. Xu, Y. Yuan, and T. Q. Quek, "Towards fast personalized semi-supervised federated learning in edge networks: Algorithm design and theoretical guarantee," IEEE Trans. Wireless Communications, pp. 1-14, 2023.
    [7] M. Fredrikson, S. Jha, and T. Ristenpart, "Model inversion attacks that exploit confidence information and basic countermeasures," in Proc. ACM SIGSAC Conference on Computer and Communications Security, 2015, pp. 1322-1333.
    [8] J. Geiping, H. Bauermeister, H. Dröge, and M. Moeller, "Inverting gradients-how easy is it to break privacy in federated learning?" in Proc. Neural Information Processing Systems (NIPS), 2020, pp. 937-947.
    [9] C. Dwork, A. Roth et al., "The algorithmic foundations of differential privacy." Foundations and Trends in Theoretical Computer Science, vol. 9, pp. 211-407, 2014.
    [10] E. Bagdasaryan, A. Veit, Y. Hua, D. Estrin, and V. Shmatikov, "How to backdoor federated learning," in Proc. International Conference on Artificial Intelligence and Statistics, 2020, pp. 2938-2948.
    [11] Z. Wang, M. Song, Z. Zhang, Y. Song, Q. Wang, and H. Qi, "Beyond inferring class representatives: User-level privacy leakage from federated learning," in Proc. IEEE International Conference on Computer Communications (INFOCOM), 2019, pp. 2512-2520.
    [12] C. Ma, J. Li, M. Ding, H. H. Yang, F. Shu, T. Q. Quek, and H. V. Poor, "On safeguarding privacy and security in the framework of federated learning," IEEE Network, vol. 34, pp. 242-248, 2020.
    [13] H. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Areas, "Communication-efficient learning of deep networks from decentralized data," in Proc. International Conference on Machine Learning (ICML), 2017, pp. 1-10.
    [14] H. Yu, S. Yang, and S. Zhu, "Parallel restarted SGD with faster convergence and less communication: Demystifying why model averaging works," in Proc. AAAI Conference on Artificial Intelligence, 2019, pp. 5693-5700.
    [15] B. McMahan, E. Moore, D. Ramage, and Hampson, "Communication-efficient learning of deep networks from decentralized data," in Proc. International Conference on Artificial Intelligence and Statistics, 2017, pp. 1273-1282.
    [16] M. Hong, Z.-Q. Luo, and M. Razaviyayn, "Convergence analysis of alternating direction method of multipliers for a family of nonconvex problems," SIAM Journal on Optimization, vol. 26, no. 1, pp. 337-364,2016.
    [17] D. Hajinezhad, M. Hong, T. Zhao, and Z. Wang, ENESTT: A nonconvex primal-dual splitting method for distributed and stochastic optimization," in Proc. Advances in Neural Information Processing Systems (NIPS), 2016, pp. 3207-3215.
    [18] S. Zhou and G. Y. Li, "Federated learning via inexact ADMM," IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 45, no. 8, pp. 9699-9708, 2023.
    [19] Y. Li, C.-W. Huang, S. Wang, C.-Y. Chi, and Q. S. T. Quek, "Privacy-preserving federated primal-dual learning for non-convex problems with non-smooth regularization," in Proc. IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2023, pp. 1-6.
    [20] Y. Li, S. Wang, T.-H. Chang, and C.-Y. Chi, "Federated stochastic primal-dual learning with differential privacy," arXiv preprint arXiv:2204.12284, 2022.
    [21] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, "Scaffold: Stochastic controlled averaging for federated learning," in Proc. International Conference on Machine Learning (ICML), 2020, pp. 5132-5143.
    [22] X. Zhang, M. Hong, S. Dhople, W. Yin, and Y. Liu, "FedPD: A federated learning framework with adaptivity to non-IID data," IEEE Trans. Signal Processing, vol. 69, pp. 6055-6070, 2021.
    [23] A. K. Sahu, T. Li, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith, "On the convergence of federated optimization in heterogeneous networks," arXiv preprint arXiv:1812.06127, 2018.
    [24] W. Shi, Q. Ling, K. Yuan, G. Wu, and W. Yin, "On the linear convergence of the ADMM in decentralized consensus optimization," IEEE Trans. Signal Processing, vol. 62, pp. 1750-1761, 2014.
    [25] A. Makhdoumi and A. Ozdaglar, "Convergence rate of distributed ADMM over networks," IEEE Trans. Automatic Control, vol. 62, pp. 5082-5095, 2017.
    [26] M. Hong and T.-H. Chang, "Stochastic proximal gradient consensus over random networks," IEEE Trans. Signal Processing, vol. 65, pp. 2933-2948, 2017.
    [27] C. Dwork, K. Kenthapadi, F. McSherry, I. Mironov, and M. Naor, "Our data, ourselves: Privacy via distributed noise generation," in Proc. Annual International Conference on the Theory and Applications of Cryptographic Techniques, 2006, pp. 486 503.
    [28] P. Mohassel and Y. Zhang, "Secureml: A system for scalable privacy-preserving machine learning," in Proc. IEEE Symposium on Security and Privacy, 2017, pp. 19-38.
    [29] I. Giacomelli, S. Jha, M. Joye, C. D. Page, and K. Yoon, "Privacy-preserving ridge regression with only linearly homomorphic encryption," in Proc. International Conference on Applied Cryptography and Network Security, 2018, pp. 243-261.
    [30] K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, H. B. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth, "Practical secure aggregation for privacy-preserving machine learning," in Proc. ACM SIGSAC Conference on Computer and Communications Security, 2017, pp. 1175-1191.
    [31] P. Vepakomma, T. Swedish, R. Raskar, O. Gupta, and A. Dubey, "No peek: A survey of private distributed deep learning," arXiv preprint arXiv:1812.03288, 2018.
    [32] Y. Li, S. Wang, C.-Y. Chi, and T. Q. Quek, "Differentially private federated learning in edge networks: The perspective of noise reduction," IEEE Network, vol. 36, no. 5, pp. 167-172, 2022.
    [33] P. Kairouz, H. B. McMahan, B. Avent, Bellet et al., "Advances and open problems in federated learning," Foundations and Trends in Machine Learning, vol. 14, pp. 1-210, 2021.
    [34] A. Triastcyn and B. Faltings, "Federated learning with Bayesian differential privacy," in Proc. IEEE International Conference on Big Data, 2019, pp. 2587-2596.
    [35] S. Truex, L. Liu, K.-H. Chow, M. E. Gursoy, and W. Wei, "LDP-Fed: Federated learning with local differential privacy," in Proc. ACM International Workshop on Edge Systems, Analytics and Networking, 2020, pp. 61-66.
    [36] Ú. Erlingsson, V. Feldman, I. Mironov, A. Raghunathan, K. Talwar, and A. Thakurta, "Amplification by shuffling: From local to central differential privacy via anonymity," in Proc. ACM-SIAM Symposium on Discrete Algorithms, 2019, pp. 2468-2479.
    [37] B. Balle, G. Barthe, and M. Gaboardi, "Privacy amplification by subsampling: Tight analyses via couplings and divergences," in Proc. Neural Information Processing Systems (NIPS), 2018, pp. 6277-6287.
    [38] S. Li, S. Hou, B. Buyukates, and S. Avestimehr, "Secure federated clustering," arXiv preprint arXiv:2205.15564, 2022.
    [39] X. Shen, Y. Liu, and Z. Zhang, "Performance-enhanced federated learning with differential privacy for Internet of Things," IEEE Internet of Things Journal, vol. 9, no. 23, pp. 24079-24094,2022.
    [40] S. Wang, T.-H. Chang, Y. Cui, and J.-S. Pang, "Clustering by orthogonal NMF model and non-convex penalty optimization," IEEE Trans. Signal Processing, vol. 69, pp. 5273-5288, 2021.
    [41] J. Ma, G. Long, T. Zhou, J. Jiang, and C. Zhang, "On the convergence of clustered federated learning," arXiv preprint arXiv:2202.06187, 2022.
    [42] H. Ding, Y. Liu, L. Huang, and J. Li, "K-means clustering with distributed dimensions," in Proc. International Conference on Machine Learning (ICML), 2016, pp. 1339-1348.
    [43] A. Ghosh, J. Hong, D. Yin, and K. Ramchandran, "Robust federated learning in a heterogeneous environment," arXiv preprint arXiv: 1906.06629, 2019.
    [44] F. Sattler, K.-R. Müller, and W. Samek, "Clustered federated learning: Modelagnostic distributed multitask optimization under privacy constraints," IEEE Trans. Neural Networks and Learning Systems, vol. 32, no. 8, pp. 3710-3722, 2020.
    [45] Y. Fraboni, R. Vidal, L. Kameni, and M. Lorenzi, "Clustered sampling: Low-variance and improved representativity for clients selection in federated learning," in Proc. International Conference on Machine Learning (ICML), 2021, pp. 3407-3416.
    [46] D. K. Dennis, T. Li, and V. Smith, "Heterogeneity for the win: One-shot federated clustering," in Proc. International Conference on Machine Learning (ICML), 2021, pp. 2611-2620.
    [47] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, "Communication-efficient learning of deep networks from decentralized data," in Porc. Artificial intelligence and statistics, 2017, pp. 1273-1282.
    [48] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, "Deep learning with differential privacy," in Proc. ACM SIGSAC Conference on Computer and Communications Security, 2016, pp. 308-318.
    [49] Y. Li, T.-H. Chang, and C.-Y. Chi, "Secure federated averaging algorithm with differential privacy," in Proc. IEEE International Workshop on Machine Learning for Signal Processing (MLSP), 2020, pp. 1-6.
    [50] N. Wang, X. Xiao, Y. Yang, J. Zhao, S. C. Hui, H. Shin, J. Shin, and G. Yu, "Collecting and analyzing multidimensional data with local differential privacy," in Proc. IEEE 35th International Conference on Data Engineering (ICDE), 2019, pp. 638-649.
    [51] X. Jiang, X. Zhou, and J. Grossklags, "Signds-fl: Local differentially private federated learning with sign-based dimension selection," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 13, no. 5, pp. 1-22, 2022.
    [52] K. Wei, J. Li, M. Ding, C. Ma, H. H. Yang et al., "Performance analysis on federated learning with differential privacy," arXiv preprint arXiv:1911.00222, 2019.
    [53] T. Zhang and Q. Zhu, "Dynamic differential privacy for ADMM-based distributed classification learning," IEEE Trans. Information Forensics and Security, vol. 12, no. 1, pp. 172-187, 2016.
    [54] X. Zhang, M. M. Khalili, and M. Liu, "Improying the privacy and accuracy of ADMM-based distributed algorithms," in Proc. International Conference on Machine Learning (ICML), 2018, pp. 5796-5805.
    [55] X. Zhang, M. M. Khalili, and M. Liu, "Recyeled ADMM: Improve privacy and accuracy with less computation in distributed algerithms," in Proc. Annual Allerton Conference on Communication, Control, and Computing (Allerton), 2018, pp. 959-965.
    [56] Y. Guo and Y. Gong, "Practical collaborative learning for crowdsensing in the internet of things with differential privacy," in Proc. IEEE Conference on Communications and Network Security, 2018, pp. 1-9.
    [57] Z. Huang, R. Hu, Y. Guo, E. Chan-Tin, and Y. Gong, "DP-ADMM: ADMM-based distributed learning with differential privacy," IEEE Trans. Information Forensics and Security, vol. 15, pp. 1002-1012, 2019.
    [58] J. Ding, S. M. Errapotu, H. Zhang, Y. Gong, M. Pan, and Z. Han, "Stochastic ADMM based distributed machine learning with differential privacy," in Proc. Security and Privacy in Communication Network, 2019, pp. 257-277.
    [59] X. Wang, H. Ishii, L. Du, P. Cheng, and J. Chen, "Privacy-preserving distributed machine learning via local randomization and ADMM perturbation," IEEE Trans. Signal Processing, vol. 68, pp. 4226 -4241, 2020.
    [60] B. Bahmani, B. Moseley, A. Vattani, R. Kumar, and S. Vassilvitskii, "Scalable kmeans++," in Proc. VLDB Endowment, 2012, pp. 622-633.
    [61] T. Kucukyilmaz, "Parallel k-means algorithm for shared memory multiprocessors," Journal of Computer and Communications, vol. 2, pp. 15-23, 2014.
    [62] M. Ester, H. P. Kriegel, J. Sander, and X. Xu, "A density-based algorithm for discovering clusters in large spatial databases with noise," in Proc. Knowledge Discovery and Data Mining (KDD), 1996, pp. 226-231.
    [63] M.-F. F. Balcan, S. Ehrlich, and Y. Liang, "Distributed k-means and k-median clustering on general topologies," in Proc. Neural Information Processing Systems (NIPS), 2013, pp. 1995-2003.
    [64] H. Ding, Y. Liu, L. Huang, and J. Li, "K-means clustering with distributed dimensions," in Proc. International Conference on Machine Learning (ICML), 2016, pp. 1339-1348.
    [65] M. Stallmann and A. Wilbik, "Towards federated clustering: A federated fuzzy c means algorithm (FFCM)," arXiv preprint arXiv:2201.07316, 2022.
    [66] W. Pedrycz, "Federated FCM: Clustering under privacy requirements," IEEE Trans. Fuzzy Systems, vol. 30, no. 8, pp. 3384-3388, 2022.
    [67] E. Hernández-Pereira, O. Fontenla-Romero, B. Guijarro-Berdiñas, and B. PérezSánchez, "Federated learning approach for spectral clustering." in Proc. European Symposium on Artificial Neural Networks, 2021, pp, 423-428.
    [68] C. Li, G. Li, and P. K. Varshney, Federated learning with soft clustering," IEEE Internet of Things Journal, vol. 9, no. 10, pp. 7773-7782, 2021.
    [69] C. Xu, Y. Qu, Y. Xiang, and L. Gao, "Asynchronous federated learning on heterogeneous devices: A survey," arXivpreprint arXiv:2109.04269, 2021.
    [70] S. Wang and T.-H. Chang, "Federated matrix factorization: Algorithm design and application to data clustering,"/EEE Trans. Signal Processing, vol. 70, pp. 1625- 1640,2022.
    [71] J. Chung, K. Lee, and K. Ramchandran, "Federated unsupervised clustering with generative models," in Proc. AAAI International Workshop on Trustable, Verifiable and Auditable Federated Learning, 2022, pp. 1-9.
    [72] N. Parikh and S. Boyd, "Proximal algorithms," Foundations and Trends in Optimization, vol. 1, no. 3, pp. 127-239, 2014.
    [73] M. Hong, D. Hajinezhad, and M.-M. Zhao, "Prox-PDA: The proximal primal-dual algorithm for fast distributed nonconvex optimization and learning over networks," in Proc. ICML, 2017, pp. 1529-1538.
    [74] C.-Y. Chi, W.-C. Li, and C.-H. Lin, Convex optimization for signal processing and communications: From fundamentals to applications. CRC Press, Boca Raton, FL, Feb 2017.
    [75] H. Ouyang, N. He, L. Tran, and A. Gray, "Stochastic alternating direction method of multipliers," in Proc. International Conference on Machine Learning (ICML), 2013, pp. 80-88.
    [76] H. Yang, M. Fang, and J. Liu, "Achieving linear speedup with partial worker participation in non-IID federated learning," in Proc. International Conference on Learning Representations (ICLR), 2021, pp. 1-23.
    [77] C. L. Blake and C. J. Merz, "UCI repository of machine learning databases," 1998, Irvine. CA: University of California, Department of Information and Computer Science. [Online]. Available: http://www.ics.uci.edu/rvmlearnIMLRepository.html.
    [78] Y. LeCun, C. Cortes, and C. Burges. The MNIST database. [Online]. Available: http://yann.lecun.com/exdb/mnist.
    [79] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," in Proc. Machine Learning and Systems, 2020, pp. 429-450.
    [80] B. Yang, X. Fu, and N. D. Sidiropoulos, "Learning from hidden traits: Joint factor analysis and latent clustering," IEEE Trans. Signal Processing, vol. 65, pp. 256-269, 2016.
    [81] D. Arthur and S. Vassilvitskii, "K-means++: The advantages of careful seeding," in Proc. Symposium on Discrete Algorithms (SODA), 2007, pp. 1027-1035.
    [82] T. Li, A. K. Sahu, M. Sanjabi, M. Zaheer, A. Talwalkar, and V. Smith, "Federated optimization in heterogeneous networks," in Proc. Machine Learning and Systems, 2020, pp. 1-12.
    [83] S. Wang, T.-H. Chang, Y. Cui, and J.-S. Pang, "Clustering by orthogonal non-negative matrix factorization: A sequential non-convex penalty approach," in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2019, pp. 5576-5580.
    [84] H. Kim and H. Park, "Sparse non-negative matrix factorization via alternating nonnegative-constrained least squares for microarray data analysis," Bioinformatics, vol. 23, no. 12, pp. 1495-1502, 2007.
    [85] W. E. Zhang, M. Tan, Q. Z. Sheng, L. Yao, and Q. Shi, "Efficient orthogonal nonnegative matrix factorization over stiefel manifold," in Proc. ACM International on Conference on Information and Knowledge Management (ICKM), 2016, pp. 1743-1752.
    [86] K. Yu, S. Yu, and V. Tresp, "Soft clustering on graphs," in Proc. Neural Information Processing Systems (NIPS), 2005, pp. 1-8.
    [87] J. Kŏnecný, H. B. McMahan, and D. Ramage, "Federated optimization: Distributed optimization beyond the datacenter," in Proc. NeuIPS Optimization for Machine Learning Workshop, 2015, pp. 1-5.
    [88] J. Kǒnecný, H. B. McMahan, D. Ramage, and P. Richtarik, "Federated optimization: Distributed machine learning for on-device intelligence," arXiv preprint arXiv:1610.02527, 2016.
    [89] D. Chai, L. Wang, K. Chen, and Q. Yang, "Secure federated matrix factorization," IEEE Intelligent Systems, vol. 36, no. 5, pp. 11-20, 2020.
    [90] P. Tseng, "Convergence of a block coordinate descent method for nondifferentiable minimization," Journal of Optimization Theory and Applications, vol. 109, pp. 475 494, 2001.
    [91] R. Mclendon, A. Friedman, D. Bigner et al., "Comprehensive genomic characterization defines human glioblastoma genes and core pathways," Nature, vol. 455, pp. 1061-1068, 2008.
    [92] J. Bolte, S. Sabach, and M. Teboulle, "Proximal alternating linearized minimization for nonconvex and nonsmooth problems," Mathematical Programming, vol. 146, pp. 459-494, 2014.

    QR CODE