許多近幾年在機器學習中感興趣的問題，都可以被表示成凸最佳化問題。在大數據的時代，如何在分散式的網點中解決問題變得越來越重要。但藉由這個方法會產生隱私與安全議題。在本回顧，我們提供一個簡單但具有影響力的演算法：交替方向乘子法。交替方向乘子法可以在分散式的地點解決問題且保持隱私。 我們可以把交替方向乘子法當作是最新科技演算法的基準。

Many problem of recent interest in machine learning can be represented to convex optimization problem. In big data era, its become more and more important to solve problem in distributed site. But by this way arise the privacy and security issue. In this review, we present a simple but powerful algorithm , the Alternating Direction Method of Multipliers (ADMM) which can solve the problem in distributed site and preserve privacy. ADMM can be viewed as the benchmark to the-state-of-the-art algorithm.

[1] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al., “Distributed optimiza- tion and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine learning, vol. 3, no. 1, pp. 1–122, 2011.
[2] E.WeiandA.Ozdaglar,“Distributedalternatingdirectionmethodofmultipliers,” in Decision and Control (CDC), 2012 IEEE 51st Annual Conference on, pp. 5445– 5450, IEEE, 2012.
[3] T. Erseghe, “Distributed optimal power flow using admm,” IEEE transactions on power systems, vol. 29, no. 5, pp. 2370–2380, 2014.
[4] E. Ghadimi, A. Teixeira, I. Shames, and M. Johansson, “Optimal parameter selec- tion for the alternating direction method of multipliers (admm): quadratic prob- lems,” IEEE Transactions on Automatic Control, vol. 60, no. 3, pp. 644–658, 2015.
[5] S. Boyd and L. Vandenberghe, Convex optimization. Cambridge university press, 2004.
[6] D. P. Bertsekas, A. Nedi, A. E. Ozdaglar, et al., “Convex analysis and optimiza- tion,” 2003.
[7] R. T. Rockafellar, Convex analysis. Princeton university press, 2015.
[8] M. S. Bazaraa, H. D. Sherali, and C. M. Shetty, Nonlinear programming: theory
and algorithms. John Wiley & Sons, 2013.
[9] D. P. Bertsekas, Constrained optimization and Lagrange multiplier methods. Aca-
demic press, 2014.
[10] S. Boyd, L. Xiao, and A. Mutapcic, “Subgradient methods,” lecture notes of
EE392o, Stanford University, Autumn Quarter, vol. 2004, pp. 2004–2005, 2003.
[11] D. P. Bertsekas, Nonlinear programming. Athena scientific Belmont, 1999.
[12] J.F.Mota,J.M.Xavier,P.M.Aguiar,andM.Püschel,“Aproofofconvergencefor the alternating direction method of multipliers applied to polyhedral-constrained functions,” arXiv preprint arXiv:1112.2295, 2011.
[13] R. Nishihara, L. Lessard, B. Recht, A. Packard, and M. I. Jordan, “A general anal- ysis of the convergence of admm,” arXiv preprint arXiv:1502.02009, 2015.
[14] D. Davis and W. Yin, “Faster convergence rates of relaxed peaceman-rachford and admm under regularity assumptions,” Mathematics of Operations Research, vol. 42, no. 3, pp. 783–805, 2017.
[15] P. Giselsson and S. Boyd, “Linear convergence and metric selection for douglas- rachford splitting and admm,” IEEE Transactions on Automatic Control, vol. 62, no. 2, pp. 532–544, 2017.
[16] D. Boley, “Local linear convergence of the alternating direction method of mul- tipliers on quadratic or linear programs,” SIAM Journal on Optimization, vol. 23, no. 4, pp. 2183–2207, 2013.
[17] Z. Xu, M. A. Figueiredo, X. Yuan, C. Studer, and T. Goldstein, “Adaptive relaxed admm: Convergence theory and practical implementation,” in 2017 IEEE Confer- ence on Computer Vision and Pattern Recognition (CVPR), pp. 7234–7243, IEEE, 2017.
[18] G. Romano, “New results in subdifferential calculus with applications to convex optimization,” Applied Mathematics and Optimization, vol. 32, no. 3, pp. 213–234, 1995.
[19] S. Xie and S. Rahardja, “Alternating direction method for balanced image restora- tion.,” IEEE Trans. Image Processing, vol. 21, no. 11, pp. 4557–4567, 2012.
[20] K. Li and S. Cong, “A robust compressive quantum state tomography algorithm using admm,” arXiv preprint arXiv:1401.6533, 2014.
[21] B. Wahlberg, S. Boyd, M. Annergren, and Y. Wang, “An admm algorithm for a class of total variation regularized estimation problems,” arXiv preprint arXiv:1203.1828, 2012.
[22] M. Annergren, A. Hansson, and B. Wahlberg, “An admm algorithm for solving l 1 regularized mpc,” in Decision and Control (CDC), 2012 IEEE 51st Annual Conference on, pp. 4486–4491, IEEE, 2012.
[23] A. Mercurio, A. Di Giorgio, and F. Purificato, “Optimal fully electric vehicle load balancing with an admm algorithm in smartgrids,” in Control & Automation (MED), 2013 21st Mediterranean Conference on, pp. 119–124, IEEE, 2013.
[24] W. Pan, A. Sootla, and G.-B. Stan, “Distributed reconstruction of nonlinear net- works: An admm approach,” arXiv preprint arXiv:1403.7429, 2014.
[25] B. Wohlberg, “Admm penalty parameter selection by residual balancing,” arXiv preprint arXiv:1704.06209, 2017.
[26] G. H. Golub and C. F. Van Loan, Matrix computations, vol. 3. JHU Press, 2012.
[27] N. Parikh, S. Boyd, et al., “Proximal algorithms,” Foundations and Trends® in
Optimization, vol. 1, no. 3, pp. 127–239, 2014.
[28] W. Deng, M.-J. Lai, Z. Peng, and W. Yin, “Parallel multi-block admm with o (1/k)
convergence,” Journal of Scientific Computing, vol. 71, no. 2, pp. 712–736, 2017.
[29] R. T. Rockafellar, “Augmented lagrangians and applications of the proximal point algorithm in convex programming,” Mathematics of operations research, vol. 1, no. 2, pp. 97–116, 1976.
[30] L. Liu and Z. Han, “Multi-block admm for big data optimization in smart grid,” in Computing, Networking and Communications (ICNC), 2015 International Con- ference on, pp. 556–561, IEEE, 2015.
[31] D. P. Bertsekas and J. N. Tsitsiklis, Parallel and distributed computation: numer- ical methods, vol. 23. Prentice hall Englewood Cliffs, NJ, 1989.
[32] M. V. Afonso, J. M. Bioucas-Dias, and M. A. Figueiredo, “Fast image recovery using variable splitting and constrained optimization,” IEEE transactions on image processing, vol. 19, no. 9, pp. 2345–2356, 2010.
[33] B. He, M. Tao, and X. Yuan, “Alternating direction method with gaussian back substitution for separable convex programming,” SIAM Journal on Optimization, vol. 22, no. 2, pp. 313–340, 2012.
[34] B. He and X. Yuan, “Linearized alternating direction method with gaussian back substitution for separable convex programming,” Numerical Algebra, Control and Optimization, vol. 3, no. 2, pp. 247–260, 2013.
[35] B. He, H.-K. Xu, and X. Yuan, “On the proximal jacobian decomposition of alm for multiple-block separable convex minimization problems and its relationship to admm,” Journal of Scientific Computing, vol. 66, no. 3, pp. 1204–1217, 2016.
[36] X. Hou, Y. Xiao, J. Cai, J. Hu, and J. E. Braun, “Distributed model predictive con- trol via proximal jacobian admm for building control applications,” in American Control Conference (ACC), 2017, pp. 37–43, IEEE, 2017.
[37] M.Ma,L.Fan,andZ.Miao,“Consensusadmmandproximaladmmforeconomic dispatch and ac opf with socp relaxation,” in North American Power Symposium (NAPS), 2016, pp. 1–6, IEEE, 2016.
[38] T.-Y.Lin,S.-Q.Ma,andS.-Z.Zhang,“Onthesublinearconvergencerateofmulti- block admm,” Journal of the Operations Research Society of China, vol. 3, no. 3, pp. 251–274, 2015.
[39] T. Lin, S. Ma, and S. Zhang, “On the convergence rate of multi-block admm,” arXiv preprint arXiv:1408.4265, vol. 229, 2014.
[40] F. Wang, W. Cao, and Z. Xu, “Convergence of multi-block bregman admm for nonconvex composite problems,” arXiv preprint arXiv:1505.03063, 2015.
[41] B. He, M. Tao, and X. Yuan, “Convergence rate and iteration complexity on the alternating direction method of multipliers with a substitution procedure for sepa- rable convex programming,” Math. Oper. Res., under revision, vol. 2, 2012.
[42] M. Hong and Z.-Q. Luo, “On the linear convergence of the alternating direction method of multipliers,” Mathematical Programming, vol. 162, no. 1-2, pp. 165– 199, 2017.
[43] B. He, L. Hou, and X. Yuan, “On full jacobian decomposition of the augmented lagrangian method for separable convex programming,” SIAM Journal on Opti- mization, vol. 25, no. 4, pp. 2274–2312, 2015.
[44] I. D. Schizas, A. Ribeiro, and G. B. Giannakis, “Consensus in ad hoc wsns with noisy links􏰨part i: Distributed estimation of deterministic signals,” IEEE Trans- actions on Signal Processing, vol. 56, no. 1, pp. 350–364, 2008.
[45] H.Zhu,G.B.Giannakis,andA.Cano,“Distributedin-networkchanneldecoding,” IEEE Transactions on Signal Processing, vol. 57, no. 10, pp. 3970–3983, 2009.
[46] T.-H. Chang, M. Hong, W.-C. Liao, and X. Wang, “Asynchronous distributed admm for large-scale optimization􏰨part i: algorithm and convergence analysis,” IEEE Transactions on Signal Processing, vol. 64, no. 12, pp. 3118–3130, 2016.
[47] R. Zhang and J. Kwok, “Asynchronous distributed admm for consensus optimiza- tion,” in International Conference on Machine Learning, pp. 1701–1709, 2014.
[48] T.-H. Chang, W.-C. Liao, M. Hong, and X. Wang, “Asynchronous distributed admm for large-scale optimization􏰨part ii: Linear convergence analysis and nu- merical performance,” IEEE Transactions on Signal Processing, vol. 64, no. 12, pp. 3131–3144, 2016.
[49] E.WeiandA.Ozdaglar,“Ontheo(1=k)convergenceofasynchronousdistributed alternating direction method of multipliers,” in Global conference on signal and information processing (GlobalSIP), 2013 IEEE, pp. 551–554, IEEE, 2013.
[50] K. Schacke, “On the kronecker product,” Master’s thesis, University of Waterloo, 2004.
[51] D. Han and X. Yuan, “A note on the alternating direction method of multipliers,” Journal of Optimization Theory and Applications, vol. 155, no. 1, pp. 227–238, 2012.
[52] J.F.Mota,J.M.Xavier,P.M.Aguiar,andM.Puschel,“Distributedbasispursuit,” IEEE Transactions on Signal Processing, vol. 60, no. 4, pp. 1942–1956, 2012.
[53] J. F. Mota, J. M. Xavier, P. M. Aguiar, and M. Puschel, “D-admm: A communication-efficient distributed algorithm for separable optimization,” IEEE Transactions on Signal Processing, vol. 61, no. 10, pp. 2718–2723, 2013.
[54] F. Iutzeler, P. Bianchi, P. Ciblat, and W. Hachem, “Asynchronous distributed opti- mization using a randomized alternating direction method of multipliers,” in Deci- sion and Control (CDC), 2013 IEEE 52nd Annual Conference on, pp. 3671–3676, IEEE, 2013.
[55] D. Hallac, J. Leskovec, and S. Boyd, “Network lasso: Clustering and optimization in large graphs,” in Proceedings of the 21th ACM SIGKDD international confer- ence on knowledge discovery and data mining, pp. 387–396, ACM, 2015.
[56] Y. A. A. S. Aldeen, M. Salleh, and M. A. Razzaque, “A comprehensive review on privacy preserving data mining,” SpringerPlus, vol. 4, no. 1, p. 694, 2015.
[57] V.S.Verykios,E.Bertino,I.N.Fovino,L.P.Provenza,Y.Saygin,andY.Theodor- idis, “State-of-the-art in privacy preserving data mining,” ACM Sigmod Record, vol. 33, no. 1, pp. 50–57, 2004.
[58] L. Ying-hua, Y. Bing-ru, C. Dan-yang, and M. Nan, “State-of-the-art in dis- tributed privacy preserving data mining,” in Communication Software and Net- works (ICCSN), 2011 IEEE 3rd International Conference on, pp. 545–549, IEEE, 2011.
[59] P. C. Weeraddana, G. Athanasiou, M. Jakobsson, C. Fischione, and J. Baras, “Per- se privacy preserving distributed optimization,” arXiv preprint arXiv:1210.3283, 2012.
[60] A. D. Procaccia, S. J. Reddi, and N. Shah, “Coordination in security games via admm,” 2015.
[61] Y. Zheng, B. Wang, W. Lou, and Y. T. Hou, “Privacy-preserving link prediction in decentralized online social networks,” in European Symposium on Research in Computer Security, pp. 61–80, Springer, 2015.
[62] H.-J. Ye, D.-C. Zhan, Y. Miao, Y. Jiang, and Z.-H. Zhou, “Rank consistency based multi-view learning: a privacy-preserving approach,” in Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp. 991–1000, ACM, 2015.
[63] A. Damiani, M. Vallati, R. Gatta, N. Dinapoli, A. Jochems, T. Deist, J. Van Soest, A. Dekker, and V. Valentini, “Distributed learning to protect privacy in multi- centric clinical studies,” in Conference on artificial intelligence in medicine in europe, pp. 65–75, Springer, 2015.
[64] Y. Gong, Y. Fang, and Y. Guo, “Privacy-preserving collaborative learning for mo- bile health monitoring,” in Global Communications Conference (GLOBECOM), 2015 IEEE, pp. 1–6, IEEE, 2015.
[65] L.Liu,X.Chen,M.Bennis,G.Xue,andZ.Han,“Adistributedadmmapproachfor mobile data offloading in software defined network,” in Wireless Communications and Networking Conference (WCNC), 2015 IEEE, pp. 1748–1752, IEEE, 2015.
[66] V. Kekatos and G. B. Giannakis, “Distributed robust power system state estima- tion,” IEEE Transactions on Power Systems, vol. 28, no. 2, pp. 1617–1626, 2013.
[67] E. Dall’Anese, H. Zhu, and G. B. Giannakis, “Distributed optimal power flow for smart microgrids.,” IEEE Trans. Smart Grid, vol. 4, no. 3, pp. 1464–1475, 2013.
[68] S. Magnússon, P. C. Weeraddana, and C. Fischione, “A distributed approach for the optimal power-flow problem based on admm and sequential convex ap- proximations,” IEEE Transactions on Control of Network Systems, vol. 2, no. 3, pp. 238–253, 2015.
[69] N.Kashyap,S.Werner,Y.-F.Huang,andR.Arablouei,“Privacypreservingdecen- tralized power system state estimation with phasor measurement units,” in Sensor Array and Multichannel Signal Processing Workshop (SAM), 2016 IEEE, pp. 1–5, IEEE, 2016.
[70] C. Wang, W. Wei, J. Wang, L. Bai, and Y. Liang, “Distributed optimal gas-power flow using convex optimization and admm,” arXiv preprint arXiv:1610.04681, 2016.