自主交流微電網控制旨在於提升交流微電網內的運作效率以及提升電網的暫態表現，當今運行的微電網具備高比例的分散式發電機以及通訊基礎設備，傳統分散式控制法是中心化架構，具高成本與低彈性的特性，因此，去中心化的演算法成為當今的趨勢。無論在中心化或是去中心化的架構下，微電網的頻率與電壓暫態表現尚未有明確地討論，所以本論文在去中心化的前提，提出分散式發電機的頻率暫態控制與具暫態飽和的交流電壓自主恢復兩大主題提出解決方案。

首先，本論文考慮獨立交流微電網中每個分散式發電機的暫態，探索一種基於最佳化分佈式交替方向乘子法的控制方案，在既定的有限時間範圍內實現頻率同步。藉由引入共識變數，我們將頻率同步問題重新定義為多智能體系統框架下的線性二次跟蹤問題。每個單獨分散式發電機的目標函數定義為二次頻率誤差和二次輸入的累積。由於共識變數也是決策變數的一部分，因此在開發最佳分佈式交替方向乘子法算法時可以獲得更大的靈活度。而直接擴展交替方向乘子法用於靜態最佳化無法完全分佈，所以通過探索每個分散式發電機中的最佳分佈式控制理論，開發完全分佈式交替方向乘子法控制。為了驗證所提出方法的效能，本論文在OPAL-RT上進行即時模擬，驗證所提出的最佳分佈式交替方向乘子法控制策略在負載變化和分散式發電機的即插即用操作下的效果。

其次，本論文提出一種基於無模型數據驅動Q學習的分佈式控制方法，用於輸入信號具有飽和特性下，實現獨立交流微電網的自主電壓恢復。藉由定義局部鄰近追蹤誤差的控制目標與運用分佈式牽制共識控制問題來解決電壓恢復議題。為應對每個分散式發電機中輸入信號飽和的影響，本論文考慮了低增益回饋方法以獲得符合飽和條件的回饋增益。由於這些回饋增益矩陣是通過求解改良式代數黎卡提方程來獲得，所以需要完全了解分散式發電機的動態，因此文中提出了迭代無模型數據驅動Q學習算法。為了找到這些回饋增益矩陣，文中定義了Q學習函數和Q學習貝爾曼方程，搭配遞歸最小平方法求解技術，完成迭代Q學習演算法，實現基於低增益回饋符合飽和限制之自主電壓恢復。在驗證提出方法的性能方面，本論文對兩個獨立的交流微電網進行了模擬。模擬結果顯示，所獲得的回饋增益與改良式代數黎卡提方程的解析解高度吻合。此外，所提出的無模型分佈式Q學習方法在模型不確定性和分散式發電機的即插即用操作下仍然保持其有效性。

Autonomous control of AC microgrids aims to enhance operational efficiency and transient performance of the grid. With a high proportion of distributed generators and communication infrastructure in modern microgrids, traditional centralized control methods are costly and inflexible. Decentralized algorithms are emerging as the current trend, yet researches of transient performance remain unclear. Therefore, this dissertation proposes solutions for transient frequency control of distributed generators and autonomous restoration of AC voltage with saturation limit under the premise of decentralization.

Firstly, to restrict transient dynamics of each distributed generator (DG) in isolated AC micro-grids (MGs), an optimal distributed alternating direction method of multiplier (ADMM)-based control scheme, will be explored for achieving frequency synchronization within a given finite horizon. By introducing the consensus variable, we re-phrase the frequency synchronization problem as a linear quadratic tracking problem under the framework of multi-agent systems. The objective function of each individual DG is defined as the accumulation of quadratic frequency errors and quadratic inputs. Since the consensus variable is also part of the decision variable, more flexibility can be gained in developing the optimal distributed ADMM-based algorithm within a finite horizon. Since the direct extension of ADMM for static optimization can not be fully distributed, a fully distributed ADMM-based control is developed by exploring the optimal distributed control theory in each DG. To validate the performance of the proposed method, real-time simulations on OPAL-RT are conducted to validate the effectiveness of the proposed optimal distributed ADMM-based control strategy even under large load variations and plug-and-play operations of DGs.
ar
Secondly, a model-free data-driven Q-learning-based distributed control is proposed for achieving autonomous voltage restoration in isolated AC MGs subject to input saturation. By defining the control objective in terms of local neighborhood tracking errors, the voltage restoration problem can be solved by the distributed pinning-based consensus problem. To address the effect of input signal saturation in each DG, the low gain feedback method is considered to obtain these feedback gains. Since these feedback gain matrices are obtained by solving the modified algebraic Riccati equation (MARE) which needs the complete knowledge of DG dynamics, an iterative model-free data-driven Q-learning algorithm is presented. A Q-learning function and a Q-learning Bellman equation are defined for finding these feedback gain matrices. Based on recursive least square techniques, an iterative Q-learning algorithm is proposed for achieving autonomous voltage restoration. To validate the performance of the proposed method, simulations of two isolated AC MG are performed. Simulation results demonstrate that the acquired feedback gains closely align with these analytical solutions of the MARE. Furthermore, the proposed model-free distributed Q-learning method remains its effectiveness even under model uncertainty and plug-and-play operations of DGs.

[1] S. W. Lin and C. C. Chu, ”Distributed optimized consensus-based frequency
synchronization in isolated AC microgrids,” in Proc. 2016 IEEE Ind. Appl. Soc. Annu. Meet., Oct. 2021, pp. 1-7, Nov. 2021.
[2] N. Pogaku, M. Prodanovic and T. C. Green, ”Modeling, analysis and testing of autonomous operation of an inverter-Based microgrid,” IEEE Trans. Power Electron., vol. 22, no. 2, pp. 613-625, March 2007.
[3] C. T. Lee, C. C. Chu, and P. T. Cheng, ”A new droop control method for the autonomous operation of distributed energy resource interface converters,”IEEE Trans. Power Electron., vol. 28, no. 4, pp. 1980-1993, April 2013.
[4] L. Y. Lu and C. C. Chu, “Consensus-based droop control synthesis for multiple DICs in isolated micro-grids”, IEEE Trans. Power Syst. , vol. 30, no. 5, pp. 2243-2256, 2015.
[5] L. Y. Lu and C. C. Chu, “Consensus-based secondary frequency and voltage droop control of virtual synchronous generators for isolated AC micro-grids”, IEEE J. Emerg. Sel. Topics in Circuits and Systems, vol. 5, no. 3, pp. 443-455,2015.
[6] A. Bidram, V. Nasirian, Davoudi, and F L. Lewis, Cooperative Synchronization in Distributed Microgrid Control, Springer Int. Publishing AG, 2017.
[7] J. W. Simpson-Porco, ”On stability of distributed-averaging proportional-integral frequency control in power systems”, IEEE Control Systems Letters, vol. 5, no. 2, pp. 677-662, Apr. 2021.
[8] N. F. Avila and C. C. Chu, ”Distributed pinning droop control in isolated AC microgrids,” IEEE Tran. Ind. Appl. , vol. 53, no. 4, 3237-3249, July 2017.
[9] J. Hu, Y. Shan, K. W. Cheng and S. Islam, ”Overview of power converter control in microgrids challenges, advances, and future trends,” IEEE Trans. Power Electr., Early Access, 2022.
[10] J. Duan, C. Wang, H. Xu, W. Liu, J. C. Peng, and H. Jiang, ”Distributed control of inverter-interfaced microgrids with bounded transient line currents,”IEEE Trans. Ind. Inform., vol. 14, no. 5, pp. 2052-2061, May 2018.
[11] J. Duan, C. Wang, H. Xu, W. Liu, Y. Xue, J. C. Peng, and H. Jiang, ”Distributed control of inverter-interfaced microgrids based on consensus algorithm with improved transient performance,” IEEE Trans. Smart Grid, vol. 10, no. 2, pp. 1303-1312, March 2019.
[12] F. L. Lewis, D. Vrabie, and V. L. Syrmos, Optimal Control, John Wiley & Sons, Inc., Hoboken, New Jersey, USA, 2012.
[13] Y. Khayat, M. Naderi, Q. Shafiee, Y. Batmani, M. Fathi, J. M. Guerrero, and H. Bevrani, ”Decentralized optimal frequency control in autonomous micro-grids,” IEEE Trans. on Power Systems, vol. 34, no. 3 , pp. 2345-2353, May 2019.
[14] Y. Khayat, Q. Shafiee, R. Heydari, M. Naderi, T. Dragiˇcevi´c, J. W. Simpson-Porco, and H. Bevrani, ”On the secondary control architectures of AC microgrids: An overview,” IEEE Trans. Power Electronics, vol. 35, no. 6, pp, 6482-6500, Nov. 2019.
[15] Y. Xu, H. Sun, W. Gu, Y. Xu, and Z. Li, ”Optimal distributed control for secondary frequency and voltage regulation in an islanded microgrid,” IEEE Trans. Ind. Inform., vol. 15, no. 1, pp. 225-235, Jan. 2019.
[16] A. M. Dissanayake, and N. C. Ekneligoda, ”Transient optimization of parallel connected inverters in islanded AC microgrids,” IEEE Trans. Smart Grid, vol. 10, no. 5, pp. 4951-4961, Sep. 2019.
[17] J. Hu, Y. Shan, J. M. Guerrero, A. Ioinovici, K. W. Chan, and J. Rodriguez, ” Model predictive control of microgrids–An overview”, Renewable and Sustainable Energy Reviews, vol. 136, 110422, 2021.
[18] J. Hu, J. M. Guerrero, and S. Islam, Model Predictive Control for Microgrids: from Power Electronic Converters to Energy Management, Institution of Engineering and Technology (IET), UK, 2021.
[19] L. Busoniu, R. Babuska, and B. De Schutter, ”A comprehensive survey of multiagent reinforcement learning,” IEEE Trans. Systems, Man, and Cybernetics, Part C: Applications and Reviews, vol. 38, no. 2, pp. 156-172, 2008.
[20] M. Adibi and J. Van der Woude, ”Secondary Frequency Control of Microgrids: An Online Reinforcement Learning Approach,” IEEE Trans. Auto. Control, vol. 67, no. 9, pp. 4824-4831, 2022.
[21] S. Boyd, “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Found. Trends Mach. Learn., vol. 3, no. 1, pp. 1–122, 2010.
[22] R. Zhang and J. Kwok, ”Asynchronous distributed ADMM for consensus
optimization, ” Proc. Machine Learning Research, vol. 32, pp. 1701-1709, June, 2014.
[23] S. Abhinav, I. D. Schizas, F. Ferrese, and A. Davoudi, ” Optimization-based AC microgrid synchronization, ” IEEE Trans. Ind. Inform., vol. 13, no. 5, pp. 2339-2349, 2017.
[24] S. W. Lin and C. C. Chu, ”Distributed Q-learning secondary voltage control of isolated AC microgrids with saturated input,” in 59th Annual IEEE Industrial and Commercial Power Systems (I&CPS) Technical Conference, Las Vegas, NV, USA, May 21–25, 2023, p. 1-5.
[25] N. Pogaku, M. Prodanovic, and T. C. Green, “Modeling, analysis and testing of autonomous operation of an inverter-based microgrid,” IEEE Trans. Power Electron., vol. 22, no. 2, pp. 613–625, Mar. 2007.
[26] C.T. Lee, C.C. Chu, and P.T. Cheng, “A new droop control method for the autonomous operation of distributed energy resource interface converters,” IEEE Trans. Power Electron., vol. 28, no. 4, pp. 1980–1993, Apr. 2013.
[27] A. Maknouninejad, Z. Qu, F. L. Lewis, and A, Davoudi, ”Optimal, nonlinear, and distributed designs of droop controls for DC microgrids,” IEEE Trans. Smart Grid, Vol. 5, no. 5, pp. 2508-2516, Sept. 2014.
[28] A. Bidram, A. Davoudi, F. Lewis, and J. Guerrero, “Distributed cooperative secondary control of microgrids using feedback linearization,” IEEE Trans. Power Syst., vol. 28, no. 3, pp. 3462-–3470, Aug. 2013.
[29] D. Yue, H. Zhang, and C. Dou, Cooperative Optimal Control of Hybrid Energy Systems, Springer Verlag, Singapore, 2021.
[30] K. P. Schneider, S. Laval, J. Hansen, R. B. Melton, L. Ponder, L. Fox, J. Hart, J. Hambrick, M. Buckner, M. Baggu, K. Prabakar, M. Manjrekars, S. Essakiappan, L. M. Tolbert, M. Manjrekar, Y. Liu, J. Dong, L. Zhu, A. Smallwood,A. Jayantilal, C. Irwin, and G. Yuan, ”A distributed power system control architecture for improved distribution system resiliency,” IEEE Access, vol. 7, pp. 9957-9970, 2019.
[31] L. Lu and C. Chu, ”Consensus-based secondary frequency and voltage droop control of virtual synchronous generators for isolated AC micro-grids,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 5, no. 3, pp. 443-455, Sep. 2015.
[32] L. Lu and C. Chu, ”Consensus-based droop control synthesis for multiple DICs in isolated micro-grids,” IEEE Trans. Power Systems, vol. 30, no. 5,pp. 2243-2256, Sep. 2015.
[33] N. F. Avila and C. C. Chu, “Distributed pinning droop control in isolated AC microgrids,” IEEE Tran. Ind. Appl., vol. 53, no. 4, 3237–3249, Jul. 2017.
[34] Y. Khayat, Q. Shafiee, R. Heydari, M. Naderi, T. Dragiˇcevi´c, J. W. Simpson-Porco, and H. Bevrani, ”On the secondary control architectures of AC microgrids: an overview,” IEEE Trans. Power Electron., vol. 35, no. 6, pp. 6482-6500, Jun. 2020.
[35] J. W. Simpson-Porco, “On stability of distributed-averaging proportional-integral frequency control in power systems,” IEEE Control Syst. Lett., vol. 5, no. 2, pp. 677–662, Apr. 2021.
[36] S. W. Lin and C. C. Chu, ”Optimal distributed ADMM-based control for frequency synchronization in isolated AC microgrids,” IEEE Trans. Ind. Appl.,vol. 59, no. 2, pp. 2458 - 2472, 2023.
[37] F. L. Lewis, D. L. Vrabie, and V. L. Syrmos, Optimal Control, Third Ed.,John Wiley & Sons, Hoboken, NJ, USA, 2012.
[38] D. Vrabie, K. G. Vamvoudakis, and F. L. Lewis, Optimal Adaptive Control and Differential Games by Reinforcement Learning Principles, IET Control Engineering Series 81, Institution of Engineering and Technology, London, U.K., 2013.
[39] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, USA, 1998.
[40] V. Fran¸cois-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau, ”An Introduction to deep reinforcement learning,” Foundations and Trends in Machine Learning, vol. 11, no. 3-4, pp. 219-354, 2018.
[41] K. G. Vamvoudakis, Y. Wan, F. L. Lewis, and D. Cansever Eds., Handbook of Reinforcement Learning and Control, Springer Int. Pub., Cham, Switzerland, 2021.
[42] E. O. Arwa, and K. A. Folly, ”Reinforcement learning techniques for optimal power control in grid-connected microgrids: A comprehensive review,” IEEE Access, vol. 8, pp. 208992-209007, 2020.
[43] C. Mu, Q. Zhao, Z Gao, and C. Sun, ”Q-learning solution for optimal consensus control of discrete-time multiagent systems using reinforcement learning,” J. Franklin Institute, vol. 356, no. 13, pp. 6946-6967, 2019.
[44] F. Liu, Z. Wang, C. Zhao, and P. Yang, Merging Optimization and Control in Power Systems: Physical and Cyber Restrictions in Distributed Frequency Control and Beyond, John Wiley & Sons Inc., Hoboken, New Jersey, U.S.A, 2022.
[45] M. Long, H. Su, X. Wang, G. P. Jiang, and X. Wang, “An iterative Q-learning based global consensus of discrete-time saturated multi-agent systems,” Chaos: An Interdisciplinary J. Nonlinear Science, vol. 29, no. 10, p. 103127, 2019.
[46] Z. Lin, Low Gain Feedback, in Lecture Notes in Control and Information Sciences (LNCIS), vol, 240, Springer, London, UK, 1999.
[47] H. Su, M. Z. Chen, J. Lam, and Z. Lin. ”Semi-global leader-following consensus of linear multi-agent systems with input saturation via low gain feedback,” IEEE Trans. Circuits and Systems I: Regular Papers, vol. 60, no. 7 pp, 1881-1889, 2013.
[48] M. Z. Q. Chen, L. Zhang, H. Su, and G. Chen, “Stabilizing solution and parameter dependence of modified algebraic Riccati equation with application to discrete-time network synchronization,” IEEE Trans. Auto. Control vol. 61, no. 1, pp.228–233, Jan. 2016.
[49] S. A. A. Rizvi and Z. Lin, ”An iterative Q-learning scheme for the global stabilization of discrete-time linear systems subject to actuator saturation,” Int. J. Robust and Nonlinear Control, vol. 29, no. 9, pp. 2660-2672, 2019.
[50] F. Guo, C. Wen, J. Mao, and Y. D. Song, ”Distributed secondary voltage and frequency restoration control of droop-controlled inverter-based microgrids,” IEEE Trans. Ind. Electron., vol. 62, no. 7, pp. 4355-4364, July 2014.
[51] N. M. Dehkordi, N. Sadati, and M. Hamzeh, ”Fully distributed cooperative secondary frequency and voltage control of islanded microgrids,” IEEE Trans. Energy Conver., vol. 32, no. 2, pp. 675-685, June 2017.
[52] N. M. Dehkordi, N. Sadati, and M.Hamzeh, ”Distributed robust finite-time secondary voltage and frequency control of islanded microgrids,”IEEE Trans. on Power Syst., vol.32, no. 5, pp. 3648-3659, Sep. 2017.
[53] J. Hu and P. Bhowmick, ”A consensus-based robust secondary voltage and frequency control scheme for islanded microgrids,” Int. J. Electrical Power & Energy Systems, vol. 116, pp.105575, 2020.
[54] D. Ding, Q. L. Han, Z. Wang, and X. Ge, ”A survey on model-based distributed control and filtering for industrial cyber-physical systems,” IEEE Trans. Ind. Inform., vol. 15, no. 5, pp. 2483-2499, May 2019.
[55] E. A, Alves Coelho, D. Wu, J. M. Guerrero, J. C. Vasquez, T. Dragicevic, C Stefanovic, and P. Popovski, ” Small-signal analysis of the microgrid secondary control considering a communication time delay,” IEEE Trans. Ind. Electron., vol. 63, no. 10, pp. 6257-6269, Oct. 2016.
[56] H. Bevrani, B. Francois, and T. Ise, Microgrid Dynamics and Control, John Wiley & Sons, Inc., Hoboken, NJ 07030, USA, 2017.
[57] Q. Shafiee, V. Nasirian, J. C. Vasquez, J. M. Guerrero, and A. Davoudi, ”A multi-functional fully distributed control framework for AC microgrids, ” IEEE Trans. Smart Grid, vol. 9, no. 4, pp. 3247-3258, 2016.
[58] W. J. Ma, J. Wang, V. Gupta, and C. Chen, ”Distributed energy management for networked microgrids using online ADMM with regret,” IEEE Trans. Smart Grid, vol. 9, no. 2, pp, 847-856, 2016.
[59] G. Chen and Q. Yang, ”An ADMM-based distributed algorithm for economic dispatch in islanded microgrids, ” IEEE Trans. Ind. Inform., vol. 14, no. 9, pp. 3892-3903, 2017.
[60] Y. Zheng, Y. Song, D. J. Hill, and Y. Zhang, ”Multiagent system based microgrid energy management via asynchronous consensus ADMM,” IEEE Trans. Energy Conv., vol. 33, no. 2, pp. 886-888, 2018.
[61] D. P. Bertseka and J. N. Tsitsiklis, Parallel and Distributed Computation: Numerical Methods, New York, NY, USA: Athena Sci., 2003.
[62] Q. Wang, Z. Duan, J. Wang, and G. Chen, ”LQ synchronization of discrete time multiagent systems: a distributed optimization approach,” IEEE Trans. Auto. Control, vol. 64, no. 12, pp. 5183-5190, Dec. 2019.
[63] X. Gao and S. Z. Zhang, ”First-order algorithms for convex optimization with nonseparable objective and coupled constraints,” J. Operations Research Society of China, vol. 5, no. 2, pp. 131-159, 2017.
[64] H.-D. Chiang and R. Jean-Jumeau, ”Optimal network reconfigurations in distribution systems: II Solution algorithms and numerical results,” IEEE Trans. Power Deliv. , vol. 5, no. 3, pp. 1568-1574, July 1990.
[65] N. Mwakabuta and A. Sekar, “Comparative study of the IEEE 34 Node Test Feeder under Practical Simplifications,” 39th North American Power Symposium, pp. 484–49, 2007.
[66] J.J. E. Slotine, W. Li, and others, Applied Nonlinear Control, vol. 199. Prentice hall Englewood Cliffs, NJ, 1991.
[67] N. Shaukat, M. R. Islam, N. M. Rahman, B. Khan, B. Ullah, S. M. Ali, and A. Fekih, ” Decentralized, democratized, and decarbonized future electric power distribution grids: A survey on the paradigm shift from the conventional power system to microgrid structures,” IEEE Access, vol. 11, pp. 60957-60987, 2023.
[68] M. Long, H. Su, and Z. Zeng, ”Output-feedback global consensus of discrete-time multiagent systems subject to input saturation via Q-learning method,” IEEE Trans. Cybernetics, vol. 52, no. 3, pp. 1661-1670, 2020.
[69] Y. Jiang and Z.-P. Jiang, Robust Adaptive Dynamic Programming, IEEE Press Series on Systems Science and Engineering, John Wiley & Sons, Hoboken, NJ, USA, 2017.
[70] G. Tao,Adaptive Control Design and Analysis, Wiley Series on Adaptive and Learning Systems for Signal Processing, Communications, and Control, vol. 37. John Wiley & Sons, New York, USA, 2003.
[71] L. Y. Lu, H. J. Liu, H. Zhu, and C. C. Chu, ”Intrusion detection in distributed frequency control of isolated microgrids,” IEEE Trans. Smart Grid, vol. 10, no. 6, pp. 6502-6515, 2019.
[72] D. Shi, P. Lin, Y. Wang, C. C. Chu, Y. Xu, and P. Wang, ”Deception attack detection of isolated DC microgrids under consensus-based distributed voltage control architecture,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 11, no. 1, pp. 155-167, 2021.
[73] L. Y. Lu, J. H. Liu, S. W. Lin, and C. C. Chu, ”Concurrent cyber deception attack detection of consensus control in isolated AC microgrids,” IEEE Trans. Ind. Appl., vol. 59, no. 6, pp. 7584 - 7596, 2023.
[74] T. L. Lee and P. T. Cheng, ”Design of a new cooperative harmonic filtering strategy for distributed generation interface converters in an islanding network,” IEEE Trans. Power Electron., vol. 22, no. 5 , pp. 1919-1927, 2007.
[75] P. T. Cheng, C. A. Chen, T. L. Lee, and S. Y. Kuo, ”A cooperative imbalance compensation method for distributed-generation interface converters,” IEEE Trans. Ind. Appl. , vol. 45, no. 2, pp. 805-815, 2009.
[76] J. He, Y. W. Li, and F. Blaabjerg, ”An enhanced islanding microgrid reactive power, imbalance power, and harmonic power sharing scheme”, IEEE Trans. Power Electron., vol. 30, no. 6, pp. 3389-3401, 2014.
[77] J. Lu, M. Zhao, S. Golestan, T. Dragicevic, X. Pan, and J. M. Guerrero, ”Distributed event-triggered Control for reactive, unbalanced and harmonic power sharing in islanded AC microgrids,” IEEE Trans. Ind. Electron. , vol. 69, no. 2, pp. 1548-1560, 2022.
[78] https://www.keysight.com/us/en/assets/3122-1402/flyers/EXata-Network-Modeling-Multi-Domain-Networks.pdf