為了解決在某些區域可能收不到定位訊號的問題，本論文開發一種結合卡爾曼濾波器及深度強化學習的自適應卡爾曼濾波器，降低慣性導航系統的位置預測誤差。在卡爾曼濾波器的狀態更新過程中，過程雜訊協方差矩陣，簡稱Q矩陣，扮演重要角色，此矩陣描述了系統狀態的雜訊特徵，但目前尚不存在可精確描述Q矩陣的方法，故我們使用深度決定性策略梯度網路，簡稱DDPG，來更新Q矩陣，此模型可處理連續動作空間及較高維度的問題，符合估計Q矩陣的需求。
我們在預測位置誤差超過臨界點時進行Q矩陣之更新，將狀態的過程雜訊向量放入DDPG中，產生更新之Q矩陣並傳回卡爾曼濾波器使用，在實驗中我們調整了此模型的各種參數、試著縮短其執行時間、並實際模擬定位系統訊號斷訊之情形，最後，我們也將我們的模型與各種非深度學習更新Q矩陣方法做了比較，評估了其中之優劣。

To address the issue of positioning signal loss in certain areas, this thesis develops an adaptive Kalman filter that combines the Kalman filter and deep reinforcement learning for autonomous positioning systems. The aim is to reduce the position errors predicted by the Inertial Navigation System (INS). This is achieved by updating the covariance matrix of the process noise, referred to as Q matrix, which captures the noise characteristics of the system state and significantly affects the prediction accuracy of the Kalman filter. Since it is difficult to characterize the Q matrix in closed form, a deep learning based approach is used to update the Q matrix by defining the amount of update as the action space. To allow an infinitely small update step size, the deep deterministic policy gradient network (DDPG) with continuous action space is employed in this work.
We update the Q matrix when the predicted position error exceeds a critical threshold. The state's process noise vector is feed into DDPG to generate the updated Q matrix, which is then passed back to the Kalman filter for utilization. Extensive simulations are performed to study the impact of various parameters. In addition, we compare the proposed model with various non-deep learning based methods for updating the Q matrix and evaluate their strengths and weaknesses.

[1] X. Gao, H. Luo, B. Ning, F. Zhao, L. Bao, Y. Gong, Y. Xiao, and J. Jiang, “RL-AkF: An adaptive kalman filter navigation algorithm based on reinforcement learning for ground vehicles,” Remote Sensing, vol. 12, no. 11, pp. 1704–1728, 2020.
[2] C. G. Boncelet and B. W. Dickinson, “An approach to robust kalman filtering,” in Proc. IEEE Conference on Decision and Control, 1983, pp. 304–305.
[3] P. W. McBurney, “A robust approach to reliable real-time kalman filtering,” in Proc. IEEE Symposium on Position Location and Navigation. A Decade of Excellence in the Navigation Sciences, 1990, pp. 549–556.
[4] J. Liu, B.-g. Cai, and J. Wang, “Cooperative localization of connected vehicles: Integrating gnss with dsrc using a robust cubature kalman filter,” IEEE Transactions on Intelligent Transportation Systems, vol. 18, no. 8, pp. 2111–2125, 2016.
[5] S. S. Saab, “Discrete-time kalman filter under incorrect noise covariances,” in Proc. American Control Conference-ACC’95, vol. 2, 1995, pp. 1152–1156.
[6] J. Duník, O. Kost, O. Straka, and E. Blasch, “State and measurement noise in positioning and tracking: Covariance matrices estimation and gaussianity assessment,” in Proc. IEEE/ION Position, Location and Navigation Symposium (PLANS), 2018, pp. 1326–1335.
[7] J. J. Wang, W. Ding, and J. Wang, “Improving adaptive kalman filter in gps/sdins integration with neural network,” in Proc. 20th International Technical Meeting of the Satellite Division of The Institute of Navigation (ION GNSS), 2007, pp. 571–578.
[8] T. T. Ngoc, A. Khenchaf, and F. Comblet, “Evaluating process and measurement noise in extended kalman filter for gnss position accuracy,” in Proc. 13th European Conference on Antennas and Propagation (EuCAP), 2019, pp. 1–5.
[9] W. Ding, J. Wang, and C. Rizos, “Improving covariance based adaptive estimation for gps/ins integration,” in Proc. Korean Institute of Navigation and Port Research Conference, vol. 1, 2006, pp. 259–264.
[10] L. Zhang, D. Sidoti, A. Bienkowski, K. R. Pattipati, Y. Bar-Shalom, and D. L. Kleinman, “On the identification of noise covariances and adaptive kalman filtering: A new look at a 50 year-old problem,” IEEE Access, vol. 8, pp. 59 362–59 388, 2020.
[11] P. Abbeel, A. Coates, M. Montemerlo, A. Y. Ng, and S. Thrun, “Discriminative training of kalman filters,” in Proc. Robotics: Science and systems, vol. 2, 2005, pp. 1–8.
[12] A. Almagbile, J. Wang, and W. Ding, “Evaluating the performances of adaptive kalman filter methods in gps/ins integration,” Journal of Global Positioning Systems, vol. 9, no. 1, pp. 33–40, 2010.
[13] S. Akhlaghi, N. Zhou, and Z. Huang, “Adaptive adjustment of noise covariance in kalman filter for dynamic state estimation,” in Proc. IEEE Power & Energy Society General Meeting, 2017, pp. 1–5.
[14] Z. Zou, T. Huang, L. Ye, and K. Song, “Cnn based adaptive kalman filter in high-dynamic condition for low-cost navigation system on highspeed uav,” in Proc. 5th Asia-Pacific Conference on Intelligent Robot Systems (ACIRS), 2020, pp. 103–108.
[15] D. Kim, K. Min, H. Kim, and K. Huh, “Vehicle sideslip angle estimation using deep ensemble-based adaptive kalman filter,” Mechanical Systems and Signal Processing, vol. 144, pp. 106 862–106 878, 2020.
[16] R. Wang, M.-S. Liu, Y. Zhou, Y.-Q. Xun, and W.-B. Zhang, “A deep belief networks adaptive kalman filtering algorithm,” in Proc. 7th IEEE International Conference on Software Engineering and Service Science (ICSESS), 2016, pp. 178–181.
[17] K. Xiong, C. Wei, and H. Zhang, “Q-learning for noise covariance adaptation in extended kalman filter,” Asian Journal of Control, vol. 23, no. 4, pp. 1803–1816, 2021.
[18] J.-N. Chi, C. Qian, P. Zhang, W. Xiao, and L. Xie, “A novel elm based adaptive kalman filter tracking algorithm,” Neurocomputing, vol. 128, pp. 42–49, 2014.
[19] H. Luo, Y. Luo, B. Han, and M. Zeng, “A learning-based noise tracking method of adaptive kalman filter for uav positioning,” in Proc. IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), 2022, pp. 440–445.
[20] B. Or, “Tuning Q matrix for CV and CA models in kalman filter,” Jan 26, 2021. [Online]. Available: https://towardsdatascience.com/tuning-q-matrix-for-cv-and-ca-models-in-kalman-filter-67084185d08c
[21] Aceinna, “gnss-ins-sim,” 2019. [Online]. Available: https://github.com/Aceinna/gnss-ins-sim
[22] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.
[23] E.-H. Shin, “Estimation techniques for low-cost inertial navigation,” Ph.D. dissertation, 2005.
[24] I. Skog and P. Händel, “A low-cost gps aided inertial navigation system for vehicle applications,” in Proc. 13th European Signal Processing Conference, 2005, pp. 1–4.
[25] P. Malekzadeh, M. Salimibeni, M. Hou, A. Mohammadi, and K. N. Plataniotis, “Akf-sr: Adaptive kalman filtering-based successor representation,” Neurocomputing, vol. 467, pp. 476–490, 2022.
[26] A. Assa and K. N. Plataniotis, “Adaptive kalman filtering by covariance sampling,” IEEE Signal Processing Letters, vol. 24, no. 9, pp. 1288–1292, 2017.
[27] L. Zhang, W. Shaoping, M. S. Selezneva, and K. A. Neusypin, “A new adaptive kalman filter for navigation systems of carrier-based aircraft,” Chinese Journal of Aeronautics, vol. 35, no. 1, pp. 416–425, 2022.