|Table of Contents|

Citation:
　Chuanfa Chen,Xiang Gao,Yueming Li,et al.Deep Reinforcement Learning Approach for X-rudder AUVs Fault Diagnosis Based on Deep Q-network[J].Journal of Marine Science and Application,2025,(6):1239-1251.[doi:10.1007/s11804-025-00652-1]
Click and Copy

Deep Reinforcement Learning Approach for X-rudder AUVs Fault Diagnosis Based on Deep Q-network

Info

Title:
Deep Reinforcement Learning Approach for X-rudder AUVs Fault Diagnosis Based on Deep Q-network
Author(s):
Chuanfa Chen1;  Xiang Gao2;  Yueming Li1;  Xuezhi Chen1;  Jian Cao1;  Yinghao Zhang3
Affilations:
Author(s):
Chuanfa Chen1;  Xiang Gao2;  Yueming Li1;  Xuezhi Chen1;  Jian Cao1;  Yinghao Zhang3
1. National Key Laboratory of Autonomous Marine Vehicle Technology, Harbin Engineering University, Harbin, 150001, China;
2. Jiangsu Automation Research Institute, Lianyungang, 222061, China;
3. Wuhan Second Ship Design and Research institute, Wuhan, 430000, China
Keywords:
Autonomous underwater cehicles; X-rudder; Fault diagnosis; Deep Q network; Dropout technique
分类号:
-
DOI:
10.1007/s11804-025-00652-1
Abstract:
The rudder mechanism of the X-rudder autonomous underwater cehicle (AUV) is relatively complex, and fault diagnosis capability is an important guarantee for its task execution in complex underwater environments. However, traditional fault diagnosis methods currently rely on prior knowledge and expert experience, and lack accuracy. In order to improve the autonomy and accuracy of fault diagnosis methods, and overcome the shortcomings of traditional algorithms, this paper proposes an X-steering AUV fault diagnosis model based on the deep reinforcement learning deep Q network (DQN) algorithm, which can learn the relationship between state data and fault types, map raw residual data to corresponding fault patterns, and achieve end-to-end mapping. In addition, to solve the problem of few X-steering fault sample data, Dropout technology is introduced during the model training phase to improve the performance of the DQN algorithm. Experimental results show that the proposed model has improved the convergence speed and comprehensive performance indicators compared to the unimproved DQN algorithm, with precision, recall, F1-score, and accuracy reaching up to 100%, 98.07%, 99.02%, and 98.50% respectively, and the model’s accuracy is higher than other machine learning algorithms like back propagation, support vector machine.

References:

[1] Alessandri A (2003) Fault diagnosis for nonlinear systems using a bank of neural estimators. Computers in Industry 52(3): 271-289. https://doi.org/10.1016/S0166-3615(03)00131-3
[2] Alex Gong CS, Simon Su CH, Tseng KH (2020) Implementation of Machine Learning for Fault Classification on Vehicle Power Transmission System. IEEE Sensors Journal 20(24): 15163-15176. https://doi.org/10.1109/JSEN.2020.3010291
[3] Antonelli G, Caccavale F, Sansone C, Villani L (2004) Fault diagnosis for AUVs using support vector machines. IEEE International Conference on Robotics and Automation 4486-4491. DOI: 10.1109/ROBOT.2004.1302424
[4] Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) Deep Reinforcement Learning: A Brief Survey. IEEE Signal Processing Magazine 34(6): 26-38. https://doi.org/10.1109/MSP.2017.2743240
[5] Chang ZH, Jia KW, Han T, Wei YM (2024) Towards more reliable photovoltaic energy conversion systems: A weakly-supervised learning perspective on anomaly detection. Energy Conversion and Management 316: 118845. https://www.sciencedirect.com/science/article/pii/S0196890424007866
[6] Chen Y, Mabu S, Hirasawa K, & Hu J (2007) Enhancement of trading rules on stock markets using genetic network programming with Sarsa learning. SICE Annual Conference 2007: 2700-2707. https://doi.org/10.1109/SICE.2007.4421448
[7] Ding Y, Ma L, Ma J, Suo M, Tao L, Cheng Y, Lu C (2019) Intelligent fault diagnosis for rotating machinery using deep Q-network based health state classification: A deep reinforcement learning approach. Advanced Engineering Informatics 42: 100977. https://doi.org/10.1016/j.aei.2019.100977
[8] Fang M, Li H, Zhang X (2012) A Heuristic Reinforcement Learning Based on State Backtracking Method. 2012 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology, 673-678. https://doi.org/10.1109/WI-IAT.2012.187
[9] Ferguson J, Pope A (2000) Explorer-a modular AUV for commercial site survey. Proceedings of the 2000 International Symposium on Underwater Technology (Cat. No. 00EX418). IEEE, 129-132
[10] Frank PM (1990) Fault Diagnosis in Dynamic Systems Using Analytical and Knowledge-based Redundancy A Survey and Some New Results. Automatica 26(3): 459-474 https://doi.org/10.1016/0005-1098(90)90018-D
[11] Geng H, Liu H, Wang B, Sun F (2018) Reinforcement Extreme Learning Machine for Mobile Robot Navigation. In J. Cao, E. Cambria, A. Lendasse, Y. Miche, & C. M. Vong (Eds.), Proceedings of ELM-2016, Vol. 9, pp. 61-73. Springer International Publishing, https://doi.org/10.1007/978-3-319-57421-9_6
[12] Hinton GE, Srivastava N, Krizhevsky A, Sutskever I, Salakhutdinov RR (2012) Improving neural networks by preventing co-adaptation of feature detectors (arXiv: 1207.0580) arXiv. http://arxiv.org/abs/1207.0580
[13] Huang Y, Li Y, Yu JC, Li S, Feng XS (2020) State-of-the-Art and Development Trends of AUV Intelligence. Robot 42(02): 215-231. (in Chinese) https://doi.org/10.13973/j.cnki.robot.190392
[14] Ji D, Yao X, Li S, Tang Y, Tian Y (2021) Model-free fault diagnosis for autonomous underwater vehicles using sequence Convolutional Neural Network. Ocean Engineering 232: 108874. https://doi.org/10.1016/j.oceaneng.2021.108874
[15] Jiang Y, Yin S (2019) Recent Advances in Key-Performance-Indicator Oriented Prognosis and Diagnosis With a MATLAB Toolbox: DB-KIT. IEEE Transactions on Industrial Informatics 15(5): 2849-2858. https://doi.org/10.1109/TII.2018.2875067
[16] Jiang Y, Yin S, Kaynak O (2018) Data-driven monitoring and Safety Control of Industrial Cyber-Physical Systems: Basics and Beyond. IEEE Access 6: 47374-47384. https://doi.org/10.1109/ACCESS.2018.2866403
[17] Liu F, Xu D (2016) Fault Localization and Fault-Tolerant Control for rudders of AUVs. 2016 35th Chinese Control Conference (CCC), 6537-6541
[18] Miskovic N, Barisic M (2005) Fault Detection and Localization on Underwater Vehicle Propulsion Systems Using Principal Component Analysis. Proceedings of the IEEE International Symposium on Industrial Electronics, 1721-1728. https://doi.org/10.1109/ISIE.2005.1529192.
[19] Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M, Rusu AA, Veness J, Bellemare MG, Krizhevsky A, Hinton G, Hassabis D (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602
[20] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518(7540): 529-533. https://doi.org/10.1038/nature14236
[21] Pan Y, Zheng Z, Fu D (2021) Bayesian-based water leakage detection with a novel multisensor fusion method in a deep manned submersible. Applied Ocean Research 106: 102459. https://doi.org/10.1016/j.apor.2020.102459
[22] Prestero T (2001) Verification of a six-degree of freedom simulation model for the REMUS autonomous underwater vehicle Massachusetts Institute of Technology and Woods Hole Oceanographic Institution. https://doi.org/10.1575/1912/3040
[23] Shao Z, Hua YX, Shi JY (2020) Research on Fault Diagnosis Method for Rudder Surface Based on Multiple Models. Flight Dynamics 38(3): 24-27. (in Chinese) https://doi.org/10.13645/j.cnki.f.d20200312.010
[24] Sun Y, Ran X, Li Y, Zhang G, Zhang Y (2016) Thruster fault diagnosis method based on Gaussian particle filter for autonomous underwater vehicles. International Journal of Naval Architecture and Ocean Engineering 8(3): 243-251. https://doi.org/10.1016/j.ijnaoe.2016.03.003
[25] Sun Y, Wang Z, Zhang G (2019) Fault Diagnosis Method of Autonomous Underwater Vehicle Based on Deep Learning. IOP Conference Series: Materials Science and Engineering 470: 012035
[26] Tun W, Wong JK W, Ling SH (2021) Hybrid Random Forest and Support Vector Machine Modeling for HVAC Fault Detection and Diagnosis. Sensors 21(24): 8163. https://doi.org/10.3390/s21248163
[27] Wang F, Wan L, Su Y, Xu Y (2010) AUV modeling and motion control strategy design. Journal of Marine Science and Application 9(4): 379-385. https://doi.org/10.1007/s11804-010-1023-4
[28] Wang LR, Gan Y, Xu YR, Wan L (2005) Sliding-mode observers used in thruster fault diagnosis of an autonomous underwater vehicle. Journal of Harbin Engineering University 26(4): 425-429. https://doi.org/10.3969/j.issn.1006-7043.2005.04.002
[29] Wang X, Fang X (2023) A multi-agent reinforcement learning algorithm with the action preference selection strategy for massive target cooperative search mission planning. Expert Systems with Applications 231: 120643. https://doi.org/10.1016/j.eswa.2023.120643
[30] Wang YJ, Zhang MJ, Che ZZ (2011) Qualitative diagnostic model for underwater robot propeller fault. Proceedings of the Seventh National Conference on Fault Diagnosis and Safety of Technical Processes
[31] Wang YJ, Zhang MJ, Wu J (2007) Research of the fault diagnosis method for the thruster of AUV based on information fusion. Third International Conference on Agent Computing, ICIC
[32] Watkins, CJ, Dayan, P (1992) Q-learning. Machine learning, 8, 279-292
[33] Wen L, Wang Y, Li X (2022) A new automatic convolutional neural network based on deep reinforcement learning for fault diagnosis. Frontiers of Mechanical Engineering 17(2): 17. https://doi.org/10.1007/s11465-022-0673-7
[34] Wu X, Xiong W (2021) kNN Fault Detection Based on Multi-block Information Extraction and Mahalanobis Distance. Information and Control 50(3): 287-296. https://doi.org/10.13976/j.cnki.xk.2021.0279
[35] Xing B, Wang X, Liu Z (2024) The Wide-Area Coverage Path Planning Strategy for Deep-Sea Mining Vehicle Cluster Based on Deep Reinforcement Learning. Journal of Marine Science and Engineering 12(2): 316. https://doi.org/10.3390/jmse12020316
[36] Xu L, Teoh SS, Ibrahim H (2024). A deep learning approach for electric motor fault diagnosis based on modified InceptionV3. Scientific Reports 14(1): 12344. https://doi.org/10.1038/s41598-024-63086-9
[37] Yeo R (2007) Surveying the underside of an Artic ice ridge using a man-portable GAVIA AUV deployed through the ice. OCEANS, IEEE, 2007: 1-8
[38] Yoshida H, Hyakudome T, Ishibashi S (2013) Yumeiruka-The AUV Equipped with an X-type Canard Rudder. The Twenty-third International Offshore and Polar Engineering Conference 397-401. https://api.semanticscholar.org/CorpusID:115030016
[39] Yuan C, Shuai C, Ma J, Fang Y (2023) An efficient control allocation algorithm for over-actuated AUVs trajectory tracking with fault-tolerant control. Ocean Engineering 273: 113976. https://doi.org/10.1016/j.oceaneng.2023.113976
[40] ?arkovi? M, Stojkovi? Z (2017) Analysis of artificial intelligence expert systems for power transformer condition monitoring and diagnostics. Electric Power Systems Research 149: 125-136. https://doi.org/10.1016/j.epsr.2017.04.025
[41] Zhai JQ, Yang X, Cheng YQ, Li L (2021) A review of the application of machine learning in the field of fault detection and diagnosis. Computer Measurement and Control (3): 1-9. DOI: 10.16526/j.cnki.11-4762/tp.2021.03.001
[42] Zhang MJ, Yin BJ, Liu WX, Wang YJ (2015) Feature extraction and fusion for thruster faults of AUV with random disturbance. Journal of Huazhong University of Science and Technology (Nature Science Edition) 43(6): 22-26, 54. (in Chinese) https://doi.org/10.13245/j.hust.150605
[43] Zhao H, Gao Y, Deng W (2024) Defect Detection Using Shuffle Net-CA-SSD Lightweight Network for Turbine Blades in IoT. IEEE Internet of Things Journal 11(20): 32804-32812. https://doi.org/10.1109/JIOT.2024.3409823

Memo

Memo:
Received date:2024-7-12;Accepted date:2024-10-11。<br>Foundation item:Supported by the National Natural Science Foundation of China under Grant Nos. 52071099, 52071104; National Key Project of Research and Development Program under Grant No. 2021YFC2801300; Research Fund from National Key Laboratory of Autonomous Marine Vehicle Technology under Grant No. 2023-SXJQR-SYSJJ01.<br>Corresponding author:Yueming Li,E-mail:liyueming@hrbeu.edu.cn
Last Update: 2025-12-26