本文研究可重構智慧表面 (reconfigurable intelligent surface, RIS) 輔助的多用戶多輸入單輸出 (MU-MISO) 毫米波 (mmWave) 無線通訊系統，考慮有限精度 (limited resolution)、低傳輸功率、硬體損耗 (hardware impairments, HWI) 及非完美通道資訊 (channel state information, CSI) 的影響下，透過控制多個 RIS 元件去做波束成形 (beamforming) 設計，以最小化各用戶間最差通訊狀況時接收訊號的均方誤差 (mean squared error, MSE)。
在硬體損耗、通道不確定性及離散性的限制下，使得此設計問題變為一個非凸 (non-convex) 的最佳化問題，為此本文結合傳統通訊演算法及機器學習 (machine learning, ML) 技術，設計一個基於深度強化學習 (deep reinforcement learning, DRL) 的框架，我們採用近端策略優化 (proximal policy optimization, PPO) 演算法來設計 RIS 的離散相位配置，並利用最大比率傳輸 (max ratio transmission, MRT) 技術來獲得相應的預編碼 (precoding) 設計。
模擬結果表明，我們所提出的方法在不同設定中最壞情況的均方誤差表現上都優於傳統最優波束成形方法。傳統基準方法假定已知完美分離的通道資訊並使用連續的相位偏移設計，但即使擁有這些優勢，本文的強化學習方法仍能達到更優異的均方誤差表現。我們的程式碼已在 GitHub 上開源，為進一步的研究和應用提供一些參考。這項研究不僅促進對硬體損耗和非完美通道資訊等挑戰的理解，還提供簡單的解決方案和實現方法，以加速無線通訊領域快速發展的進程，為未來技術的進步貢獻一小份心力。

In this study, we introduce a novel deep reinforcement learning (DRL)-based optimization framework aimed at mitigating various hardware impairments (HWI) and channel state information (CSI) imperfections in reconfigurable intelligent surface (RIS)-assisted millimeter-wave (mmWave) multi-input-single-output (MU-MISO) systems. By integrating with traditional algorithms, our approach jointly designs active and passive beamforming, considering phase-dependent amplitude response, phase error and CSI error, aspects often neglected in prior research. Employing proximal policy optimization (PPO), our method discretely addresses HWI and CSI challenges without continuous relaxation.
The simulation results demonstrate the superiority of our approach over the traditional optimal beamforming baseline in minimizing the mean squared error (MSE) of the signal received by users in a worst-case scenario. Additionally, we develop a custom RL environment optimized for basic GPU acceleration, facilitating ease of use and high extensibility. The code has been made open-source on GitHub, providing a valuable reference for further research and application in RIS-assisted communication systems.
This work not only advances the understanding of HWI and imperfect CSI challenges but also offers simple solutions and implementations to expedite progress in the fast-evolving field of wireless communication, paving the way for future advancements in the technologies.

[1] S. Abeywickrama, R. Zhang, Q. Wu, and C. Yuen, “Intelligent reflecting surface: Practical phase shift model and beamforming optimization,” IEEE Trans. Commun., vol. 68, no. 9, pp. 5849–5863, Jun. 2020.
[2] M.-A. Badiu and J. P. Coon, “Communication through a large reflecting surface with phase errors,” IEEE Wireless Commun. Lett., vol. 9, no. 2, pp. 184–188, Feb. 2020.
[3] T. Wang, M.-A. Badiu, G. Chen, and J. P. Coon, “Outage probability analysis of RIS- assisted wireless networks with von mises phase errors,” IEEE Wireless Commun. Lett., vol. 10, no. 12, pp. 2737–2741, Dec. 2021.
[4] Z. Peng, W. Xu, L.-C. Wang, and C. Zhao, “Achievable rate analysis and feedback design for multiuser MIMO relay with imperfect CSI,” IEEE Trans. Wireless Commun., vol. 13, no. 2, pp. 780–793, Feb. 2014.
[5] W.-Y. Chen, C.-Y. Wang, R.-H. Hwang, W.-T. Chen, and S.-Y. Huang, “Impact of hard- ware impairment on the joint reconfigurable intelligent surface and robust transceiver de- sign in MU-MIMO system,” IEEE Trans. Mobile Comput., pp. 1–16, Jun. 2023.
[6] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy opti- mization algorithms,” arXiv preprint arXiv:1707.06347, Jul. 2017.
[7] N. K. Kundu and M. R. McKay, “RIS-assisted MISO communication: Optimal beamform- ers and performance analysis,” in IEEE Global Commun. Conf. Workshops (GC Wkshps), Dec. 2020, pp. 1–6.
[8] V. Petrov, T. Kurner, and I. Hosako, “IEEE 802.15.3d: First standardization efforts for sub-terahertz band communications toward 6G,” IEEE Commun. Mag., vol. 58, no. 11, pp. 28–33, Nov. 2020.
[9] S. He, Y. Zhang, J. Wang, J. Zhang, J. Ren, Y. Zhang, W. Zhuang, and X. Shen, “A survey of millimeter-wave communication: Physical-layer technology specifications and enabling transmission technologies,” Proc. IEEE, vol. 109, no. 10, pp. 1666–1705, Oct. 2021.
[10] A. Mchangama, J. Ayadi, V. P. G. Jiménez, and A. Consoli, “Mmwave massive MIMO small cells for 5G and beyond mobile networks: An overview,” in Int. Symp. on Commun. Syst., Networks and Digit. Signal Process. (CSNDSP), Nov. 2020, pp. 1–6.
[11] B. K. J. Al-Shammari, I. Hburi, H. R. Idan, and H. F. Khazaal, “An overview of mmwave communications for 5G,” in Int. Conf. on Commun. & Inf. Technol. (ICICT), Jun. 2021, pp. 133–139.
[12] L. Yang, J. Yang, W. Xie, M. O. Hasna, T. Tsiftsis, and M. D. Renzo, “Secrecy perfor- mance analysis of RIS-aided wireless communication systems,” IEEE Trans. Veh. Tech- nol., vol. 69, no. 10, pp. 12 296–12 300, Oct. 2020.
[13] A. C. Pogaku, D.-T. Do, B. M. Lee, and N. D. Nguyen, “UAV-assisted RIS for future wireless communications: A survey on optimization and performance analysis,” IEEE Access, vol. 10, pp. 16 320–16 336, Feb. 2022.
[14] I. Z. Ahmed, H. R. Sadjadpour, and S. Yousefi, “An information-theoretic branch-and- prune algorithm for discrete phase optimization of RIS in massive MIMO,” IEEE Trans. Veh. Technol., vol. 72, no. 6, pp. 7395–7410, Jan. 2023.
[15] H. Zhou, M. Erol-Kantarci, Y. Liu, and H. V. Poor, “A survey on model-based, heuristic, and machine learning optimization approaches in RIS-aided wireless networks,” IEEE Commun. Surv. & Tut., pp. 1–1, Dec. 2023.
[16] M. A. ElMossallamy, H. Zhang, L. Song, K. G. Seddik, Z. Han, and G. Y. Li, “Recon- figurable intelligent surfaces for wireless communications: Principles, challenges, and opportunities,” IEEE Trans. on Cogn. Commun. Netw., vol. 6, no. 3, pp. 990–1002, May 2020.
[17] M. Di Renzo, A. Zappone, M. Debbah, M.-S. Alouini, C. Yuen, J. de Rosny, and S. Tretyakov, “Smart radio environments empowered by reconfigurable intelligent sur- faces: How it works, state of research, and the road ahead,” IEEE J. Sel. Areas Commun., vol. 38, no. 11, pp. 2450–2525, Nov. 2020.
[18] F. Diaby, A. Clemente, R. Sauleau, K. T. Pham, and L. Dussopt, “2 bit reconfigurable unit-cell and electronically steerable transmitarray at ka -band,” IEEE Trans. Antennas Propag., vol. 68, no. 6, pp. 5003–5008, Jun. 2020.
[19] J. Wang, W. Tang, J. C. Liang, L. Zhang, J. Y. Dai, X. Li, S. Jin, Q. Cheng, and T. J. Cui, “Reconfigurable intelligent surface: Power consumption modeling and practical measure- ment validation,” IEEE Trans. Commun., pp. 1–1, Mar. 2024.
[20] K. M. Kossifos, J. Georgiou, and M. A. Antoniades, “ASIC-enabled programmable meta- surfaces—part 1: Design and characterization,” IEEE Trans. Antennas Propag., vol. 72, no. 3, pp. 2790–2799, Mar. 2024.
[21] C. Pan, H. Ren, K. Wang, J. F. Kolb, M. Elkashlan, M. Chen, M. Di Renzo, Y. Hao, J. Wang, A. L. Swindlehurst, X. You, and L. Hanzo, “Reconfigurable intelligent surfaces for 6G systems: Principles, applications, and research directions,” IEEE Commun. Mag., vol. 59, no. 6, pp. 14–20, Jun. 2021.
[22] S. Basharat, S. A. Hassan, A. Mahmood, Z. Ding, and M. Gidlund, “Reconfigurable in- telligent surface-assisted backscatter communication: A new frontier for enabling 6G IoT networks,” IEEE Trans. Wireless Commun., vol. 29, no. 6, pp. 96–103, Dec. 2022.
[23] S. Jeong, A. Farhang, N. S. Perović, and M. F. Flanagan, “Low-complexity joint CFO and channel estimation for RIS-aided OFDM systems,” IEEE Wireless Commun. Lett., vol. 11, no. 1, pp. 203–207, Oct. 2022.
[24] R. Schroeder, J. He, G. Brante, and M. Juntti, “Two-stage channel estimation for hybrid RIS assisted MIMO systems,” IEEE Trans. Commun., vol. 70, no. 7, pp. 4793–4806, May 2022.
[25] N. Agrawal, A. Bansal, K. Singh, and C.-P. Li, “Performance evaluation of RIS-assisted UAV-enabled vehicular communication system with multiple non-identical interferers,” IEEE Trans. Intell. Transp. Syst., vol. 23, no. 7, pp. 9883–9894, Nov. 2022.
[26] Y. Cheng, W. Peng, C. Huang, G. C. Alexandropoulos, C. Yuen, and M. Debbah, “RIS- aided wireless communications: Extra degrees of freedom via rotation and location opti- mization,” IEEE Trans. Wireless Commun., vol. 21, no. 8, pp. 6656–6671, Feb. 2022.
[27] K. Ying, Z. Gao, S. Lyu, Y. Wu, H. Wang, and M.-S. Alouini, “GMD-based hybrid beamforming for large reconfigurable intelligent surface assisted millimeter-wave mas- sive MIMO,” IEEE Access, vol. 8, pp. 19 530–19 539, Jan. 2020.
[28] Z. Peng, Z. Chen, C. Pan, G. Zhou, and H. Ren, “Robust transmission design for RIS- aided communications with both transceiver hardware impairments and imperfect CSI,” IEEE Wireless Commun. Lett., vol. 11, no. 3, pp. 528–532, Mar. 2022.
[29] J. Gao, C. Zhong, X. Chen, H. Lin, and Z. Zhang, “Unsupervised learning for passive beamforming,” IEEE Commun. Lett., vol. 24, no. 5, pp. 1052–1056, Jan. 2020.
[30] H. Song, M. Zhang, J. Gao, and C. Zhong, “Unsupervised learning-based joint active and passive beamforming design for reconfigurable intelligent surfaces aided wireless net- works,” IEEE Commun. Lett., vol. 25, no. 3, pp. 892–896, Dec. 2021.
[31] B. Sheen, J. Yang, X. Feng, and M. M. U. Chowdhury, “A deep learning based model- ing of reconfigurable intelligent surface assisted wireless communications for phase shift configuration,” IEEE Open J. Commun. Soc., vol. 2, pp. 262–272, Jan. 2021.
[32] Z. Peng, Z. Zhang, L. Kong, C. Pan, L. Li, and J. Wang, “Deep reinforcement learning for RIS-aided multiuser full-duplex secure communications with hardware impairments,” IEEE Internet Things J., vol. 9, no. 21, pp. 21 121–21 135, Nov. 2022.
[33] P. Chen, X. Li, M. Matthaiou, and S. Jin, “DRL-based RIS phase shift design for OFDM communication systems,” IEEE Wireless Commun. Lett., vol. 12, no. 4, pp. 733–737, Apr. 2023.
[34] B. Saglam, D. Gurgunoglu, and S. S. Kozat, “Deep reinforcement learning based joint downlink beamforming and RIS configuration in RIS-aided MU-MISO systems under hardware impairments and imperfect CSI,” in IEEE Int. Conf. on Commun. Workshops (ICC Workshops), May 2023, pp. 66–72.
[35] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” arXiv preprint arXiv:1606.01540, Jun. 2016.
[36] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable- baselines3: Reliable reinforcement learning implementations,” J. Mach. Learn. Res. (JMLR), vol. 22, no. 1, pp. 12 348–12 355, Jan. 2021.
[37] X. Wei, D. Shen, and L. Dai, “Channel estimation for RIS assisted wireless communica- tions—part I: Fundamentals, solutions, and future opportunities,” IEEE Commun. Lett., vol. 25, no. 5, pp. 1398–1402, May 2021.
[38] X. Gan, C. Zhong, C. Huang, and Z. Zhang, “RIS-assisted multi-user MISO communica- tions exploiting statistical CSI,” IEEE Trans. Commun., vol. 69, no. 10, pp. 6781–6792, Oct. 2021.
[39] L. Wei, C. Huang, G. C. Alexandropoulos, C. Yuen, Z. Zhang, and M. Debbah, “Channel estimation for RIS-empowered multi-user MISO wireless communications,” IEEE Trans. Commun., vol. 69, no. 6, pp. 4144–4157, Jun. 2021.
[40] J. Zhu, K. Liu, Z. Wan, L. Dai, T. J. Cui, and H. V. Poor, “Sensing RISs: Enabling dimension-independent CSI acquisition for beamforming,” IEEE Trans. Inf. Theory, vol. 69, no. 6, pp. 3795–3813, Jun. 2023.
[41] G. C. Alexandropoulos and E. Vlachos, “A hardware architecture for reconfigurable in- telligent surfaces with minimal active elements for explicit channel estimation,” in IEEE Int. Conf. on Acoust., Speech and Signal Process. (ICASSP), May 2020, pp. 9175–9179.
[42] E. Vlachos, G. C. Alexandropoulos, and J. Thompson, “Wideband MIMO channel esti- mation for hybrid beamforming millimeter wave systems via random spatial sampling,” IEEE J. Sel. Topics Signal Process., vol. 13, no. 5, pp. 1136–1150, Sep. 2019.
[43] N. T. Nguyen, Q.-D. Vu, K. Lee, and M. Juntti, “Hybrid relay-reflecting intelligent surface-assisted wireless communications,” IEEE Trans. Veh. Technol., vol. 71, no. 6, pp. 6228–6244, Jun. 2022.
[44] C. Hu, L. Dai, S. Han, and X. Wang, “Two-timescale channel estimation for reconfig- urable intelligent surface aided wireless communications,” IEEE Trans. Commun., vol. 69, no. 11, pp. 7736–7747, Nov. 2021.
[45] P. Wang, J. Fang, H. Duan, and H. Li, “Compressed channel estimation for intelligent reflecting surface-assisted millimeter wave systems,” IEEE Signal Process. Lett., vol. 27, pp. 905–909, May 2020.
[46] Z. Wang, L. Liu, and S. Cui, “Channel estimation for intelligent reflecting surface assisted multiuser communications: Framework, algorithms, and analysis,” IEEE Trans. Wireless Commun., vol. 19, no. 10, pp. 6607–6620, Oct. 2020.
[47] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in IEEE Conf. on Comput. Vision and Pattern Recognit. (CVPR), Jun. 2009, pp. 248–255.
[48] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, Dec. 2014.
[49] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Ried- miller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602, Dec. 2013.
[50] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, Feb. 2015.
[51] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist rein- forcement learning,” Mach. Learn., vol. 8, pp. 229–256, May 1992.
[52] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy opti- mization,” in Int. Conf. on Mach. Learn. (ICML). PMLR, Jul. 2015, pp. 1889–1897.
[53] S. Kakade and J. Langford, “Approximately optimal approximate reinforcement learning,” in Int. Conf. on Mach. Learn. (ICML). PMLR, Jul. 2002, pp. 267–274.
[54] P. Wang, J. Fang, L. Dai, and H. Li, “Joint transceiver and large intelligent surface design for massive MIMO mmwave systems,” IEEE Trans. Wireless Commun., vol. 20, no. 2, pp. 1052–1064, Oct. 2021.
[55] T.-Y. Kan, R. Y. Chang, F.-T. Chien, B.-J. Chen, and H. V. Poor, “Hybrid relay and recon- figurable intelligent surface assisted multiuser MISO systems,” IEEE Trans. Veh. Technol., vol. 72, no. 6, pp. 7653–7668, Feb. 2023.
[56] H. Guo, Y.-C. Liang, J. Chen, and E. G. Larsson, “Weighted sum-rate maximization for in- telligent reflecting surface enhanced wireless networks,” in IEEE Global Commun. Conf. (GLOBECOM), Dec. 2019, pp. 1–6.
[57] S.-Y. Huang, J.-Y. Lin, C.-Y. Wang, and R.-H. Hwang, “MSE minimization for RIS- assisted wireless networks with phase error and phase-dependent amplitude response,” in IEEE Veh. Technol. Conf. (VTC), Jun. 2024, pp. 1–6.
[58] D. Tse and P. Viswanath, Fundamentals of wireless communication. Cambridge Univ. Press, Feb. 2005.
[59] A. Kanervisto, C. Scheller, and V. Hautamäki, “Action space shaping in deep reinforce- ment learning,” in IEEE Conf. on Games (CoG), Aug. 2020, pp. 479–486.
[60] S. Ragi, E. K. P. Chong, and H. D. Mittelmann, “Polynomial-time methods to solve uni- modular quadratic programs with performance guarantees,” IEEE Trans. Aerosp. Electron. Syst., vol. 55, no. 5, pp. 2118–2127, Nov. 2019.