|Table of Contents|

Citation:
　Zinuo Tian,Sai Deng,Zhengxing Wu,et al.Reinforcement Learning Applications in Unmanned Underwater Vehicles: A Review[J].Journal of Marine Science and Application,2026,(4):1078-1095.[doi:10.1007/s11804-025-00723-3]
Click and Copy

Reinforcement Learning Applications in Unmanned Underwater Vehicles: A Review

Info

Title:
Reinforcement Learning Applications in Unmanned Underwater Vehicles: A Review
Author(s):
Zinuo Tian1; 2;  Sai Deng1; 2;  Zhengxing Wu1; 2;  Chao Zhou1; 2
Affilations:
Author(s):
Zinuo Tian1; 2;  Sai Deng1; 2;  Zhengxing Wu1; 2;  Chao Zhou1; 2
1. The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China;
2. The School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
Keywords:
Unmanned underwater vehicles; Reinforcement learning; Motion planning; Navigation control; Multiagent collaboration
分类号:
-
DOI:
10.1007/s11804-025-00723-3
Abstract:
The ocean, which is a key component of Earth’s ecosystem, requires advanced technologies for deep and highly comprehensive exploration. Unmanned underwater vehicles (UUVs) play an important role in this task, but their development encounters great challenges due to the complex and dynamic underwater environment. Reinforcement learning (RL) has recently emerged as a promising method to improve the capabilities of UUVs. This study comprehensively reviews the implementations of RL in UUVs, with a focus on key tasks such as motion planning, navigation and control, and multiagent coordination. We investigate current difficulties and emerging trends, as illustrated by a case study. This review aims to provide a foundation for RL-based control and decision-making in UUVs and offer actionable insights for advancing studies in this rapidly evolving domain.

References:

[1] Baharin UNA, Kamarudin KR (2025) Using remote-controlled vehicles (ROV) as tools for sea cucumber conservation: A review. Maritime Technology and Research 7: 274346-274346. https://doi.org/10.33175/mtr.2025.274346
[2] Bao H, Zhang Y, Song M, Kong Q, Hu X, An X (2023) A review of underwater vehicle motion stability. Ocean Engineering 287: 115735. https://doi.org/10.1016/j.oceaneng.2023.115735
[3] Bellman R (1966) Dynamic programming. Science 153: 34-37. https://doi.org/10.1126/science.153.3731.34
[4] Bertsekas DP (1996) Neuro-dynamic programming. Athena Scientific
[5] Borase RP, Maghade DK, Sondkar SY, Pawar SN (2021) A review of PID control, tuning methods and applications. Int. J. Dynam. Control 9: 818-827. https://doi.org/10.1007/s40435-020-00665-4
[6] Chen T, Zhang Z, Fang Z, Jiang D, Li G (2024) Imitation learning from imperfect demonstrations for AUV path tracking and obstacle avoidance. Ocean Engineering 298: 117287. https://doi.org/10.1016/j.oceaneng.2024.117287
[7] Christensen L, De Gea Fernández J, Hildebrandt M, Koch CES, Wehbe B (2022) Recent advances in AI for navigation and control of underwater robots. Curr Robot Rep 3: 165-175. https://doi.org/10.1007/s43154-022-00088-3
[8] Coates RFW (1990) Underwater acoustic systems. Macmillan Education UK, London. https://doi.org/10.1007/978-1-349-20508-0
[9] Cohen JE, Small C, Mellinger A, Gallup J, Sachs J (1997) Estimates of coastal populations. Science 278: 1209-1213. https://doi.org/10.1126/science.278.5341.1209c
[10] Cong Y, Gu C, Zhang T, Gao Y (2021) Underwater robot sensing technology: A survey. Fundamental Research 1: 337-345. https://doi.org/10.1016/j.fmre.2021.03.002
[11] Dunbabin M, Marques L (2012) Robots for environmental monitoring: significant advancements and applications. IEEE Robotics & Automation Magazine 19: 24-39. https://doi.org/10.1109/MRA. 2011.2181683
[12] El-Fakdi A, Carreras M (2013) Two-step gradient-based reinforcement learning for underwater robotics behavior learning. Robotics and Autonomous Systems 61: 271-282. https://doi.org/10.1016/j.robot. 2012.11.009
[13] Feinberg V, Wan A, Stoica I, Jordan MI, Gonzalez JE, Levine S (2018) Model-based value estimation for efficient model-free reinforcement Learning. https://doi.org/10.48550/arXiv.1803.00101
[14] Fernandez DC, Hollinger, GA (2017) Model predictive control for underwater robots in ocean waves. IEEE Robot. Autom. Lett. 2: 88-95. https://doi.org/10.1109/LRA.2016.2531792
[15] Fossen TI (2011) Handbook of marine craft hydrodynamics and motion control. John Willy & Sons Ltd
[16] Gaskett C, Wettergreen D, Zelinsky A (1999) Reinforcement learning applied to the control of an autonomous underwater vehicle.Proceedings of the AustraLian Conference on Robotics and Automation (AuCRA)
[17] Griffiths G (2002) Technology and applications of autonomous underwater vehicles. CRC Press
[18] Hadi B, Khosravi A, Sarhadi P (2024) Adaptive formation motion planning and control of autonomous underwater vehicles using deep reinforcement learning. IEEE J. Oceanic Eng. 49: 311-328. https://doi.org/10.1109/JOE.2023.3278290
[19] Han Z, Wang Y, Sun Q (2023) Straight-path following and formation control of USVs using distributed deep reinforcementlearning and adaptive neural network. IEEE/CAA J. Autom. Sinica 10: 572-574. https://doi.org/10.1109/JAS.2023.123255
[20] Hou X, Wang J, Bai T, Deng Y, Ren Y, Hanzo L (2023) Environment-aware AUV trajectory design and resource management for multi-tier underwater computing. IEEE J. Select. Areas Commun. 41: 474-490. https://doi.org/10.1109/JSAC.2022.3227103
[21] Howard RA (1960) Dynamic programming and markov processes. MIT Press 2: 39-47
[22] Huang H, Tang Q, Li J, Zhang W, Bao X, Zhu H, Wang G (2020) A review on underwater autonomous environmental perception and target grasp, the challenge of robotic organism capture. Ocean Engineering 195: 106644. https://doi.org/10.1016/j.oceaneng.2019. 106644
[23] Huy DQ, Sadjoli N, Azam AB, Elhadidi B, Cai Y, Seet G (2023) Object perception in underwater environments: a survey on sensors and sensing methodologies. Ocean Engineering 267: 113202
[24] Kelasidi E, Pettersen KY, Gravdahl JT (2015) Energy efficiency of underwater robots. 10th IFAC Conference on Manoeuvring and Control of Marine Craft MCMC 48: 152-159. https://doi.org/10.1016/j.ifacol.2015.10.273
[25] Konda V, Tsitsiklis J (1999) Actor-critic algorithms. Advances in neural information processing systems 12: 1008-1014
[26] Lan W, Jin X, Chang X, Wang T, Zhou H, Tian W, Zhou L (2022) Path planning for underwater gliders in time-varying ocean current using deep reinforcement learning. Ocean Engineering 262: 112226. https://doi.org/10.1016/j.oceaneng.2022.112226
[27] LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521: 436-444
[28] Li D, Du L (2021) AUV trajectory tracking models and control strategies: a review. Journal of Marine Science and Engineering 9: 1020. https://doi.org/10.3390/jmse9091020
[29] Lillicrap TP (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509.02971
[30] Luo Y, Xu H, Li Y, Tian Y, Darrell T, Ma T (2021) Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. https://doi.org/10.48550/arXiv.1807.03858
[31] Lyu X, Sun Y, Wang L, Tan J, Zhang L (2023) End-to-end AUV local motion planning method based on deep reinforcement learning. Journal of Marine Science and Engineering 11: 1796. https://doi.org/10.3390/jmse11091796
[32] Ma D, Chen X, Ma W, Zheng H, Qu F (2024) Neural network model-based reinforcement learning control for AUV 3-D path following. IEEE Trans. Intell. Veh. 9: 893-904. https://doi.org/10.1109/TIV. 2023.3282681
[33] Macreadie PI, McLean DL, Thomson PG, Partridge JC, Jones DOB, Gates AR, Benfield MC, Collin SP, Booth DJ, Smith LL, Techera E, Skropeta D, Horton T, Pattiaratchi C, Bond T, Fowler AM (2018) Eyes in the sea: unlocking the mysteries of the ocean using industrial, remotely operated vehicles (ROVs). Science of The Total Environment 634: 1077-1091. https://doi.org/10.1016/j.scitotenv.2018.04.049
[34] Mai C, Pedersen S, Hansen L, Jepsen KL, Yang Z (2016) Subsea infrastructure inspection: A review study. IEEE International Conference on Underwater System Technology: Theory and Applications (USYS), Penang, Malaysia, 71-76. https://doi.org/10.1109/USYS.2016.7893928
[35] Mnih V (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602
[36] Morozs N, Gorma W, Henson BT, Shen L, Mitchell PD, Zakharov YV (2020) Channel modeling for underwater acoustic network simulation. IEEE Access 8: 136151-136175. https://doi.org/10.1109/ACCESS.2020.3011620
[37] Paull L, Saeedi S, Seto M, Li H (2014) AUV navigation and localization: a review. IEEE Journal of Oceanic Engineering 39: 131-149. https://doi.org/10.1109/JOE.2013.2278891
[38] Pereira MI, Pinto AM (2024) Reinforcement learning based robot navigation using illegal actions for autonomous docking of surface vehicles in unknown environments. Engineering Applications of Artificial Intelligence 133: 108506. https://doi.org/10.1016/j.engappai. 2024.108506
[39] Politi E, Stefanidou A, Chronis C, Dimitrakopoulos G, Iraklis Varlamis (2024) Adaptive deep reinforcement learning for efficient 3D navigation of autonomous underwater vehicles. IEEE Access 12: 178209-178221. https://doi.org/10.1109/ACCESS.2024.3508031
[40] Rummery GA, Niranjan M (1994) On-line Q-learning using connectionist systems. University of Cambridge, Department of Engineering Cambridge, Cambridge, UK
[41] Schettini R, Corchs S (2010) Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP J. Adv. Signal Process. 746052. https://doi.org/10.1155/2010/746052
[42] Schulman J (2015) Trust region policy optimization. arXiv preprint arXiv:1502.05477
[43] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. https://doi.org/10.48550/arXiv.1707.06347
[44] Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M (2016) Mastering the game of Go with deep neural networks and tree search. Nature 529: 484-489
[45] Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2018) A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362: 1140-1144. https://doi.org/10.1126/science.aar6404
[46] Singh B, Kumar R, Singh VP (2022) Reinforcement learning in robotic applications: a comprehensive survey. Artif Intell Rev 55: 945-990. https://doi.org/10.1007/s10462-021-09997-9
[47] Su Z, Zhou M, Han F, Zhu Y, Song D, Guo T (2019) Attitude control of underwater glider combined reinforcement learning with active disturbance rejection control. J Mar Sci Technol 24: 686-704. https://doi.org/10.1007/s00773-018-0582-y
[48] Sun Y, Luo X, Ran X, Zhang G (2021) A 2D optimal path planning algorithm for autonomous underwater vehicle driving in unknown underwater canyons. JMSE 9: 252. https://doi.org/10.3390/jmse9030252
[49] Sun Y, Ran X, Zhang G, Xu H, Wang X (2020) AUV 3D path planning based on the improved hierarchical deep Q network. Journal of Marine Science and Engineering 8: 145. https://doi.org/10.3390/jmse8020145
[50] Sun Y, Yan C, Xiang X, Zhou H, Tang D, Zhu Y (2023) Towards end-to-end formation control for robotic fish via deep reinforcement learning with non-expert imitation. Ocean Engineering 271: 113811. https://doi.org/10.1016/j.oceaneng.2023.113811
[51] Sutton RS (1990) Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Machine Learning Proceedings, Elsevier. 216-224
[52] Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3: 9-44. https://doi.org/10.1007/BF00115009
[53] Sutton RS, Barto AG (2018) Reinforcement learning. Second edition: an introduction. MIT Press
[54] Terracciano DS, Bazzarello L, Caiti A, Costanzi R, Manzari V (2020) Marine robots for underwater surveillance. Curr Robot Rep 1: 159-167. https://doi.org/10.1007/s43154-020-00028-z
[55] Thorpe SA (2007) An introduction to ocean turbulence. Cambridge University Press, Cambridge
[56] Tong R, Feng Y, Wang J, Wu Z, Tan M, Yu J (2023) A survey on reinforcement learning methods in bionic underwater robots. Biomimetics 8: 168. https://doi.org/10.3390/biomimetics8020168
[57] Van Hasselt H, Guez A, Silver D (2016) Deep reinforcement learning with double q-learning. Proceedings of the AAAI Conference on Artificial Intelligence, Phoenix, AZ, 2094-2100
[58] Vu QT, Pham MH, Nguyen VD, Duong VT, Nguyen HH, Nguyen TT (2023) Optimization of central pattern generator-based locomotion controller for fish robot using deep deterministic policy gradient. In: Nguyen DC, Vu NP, Long BT, Puta H, Sattler KU (Eds.), Advances in engineering research and application, lecture notes in networks and systems. Springer International Publishing, Cham, 764-770. https://doi.org/10.1007/978-3-031-22200-9_80
[59] Wang Y, Cao J, Sun J, Zou X, Sun C (2024a) Path following control for unmanned surface vehicles: a reinforcement learning-based method with experimental validation. IEEE Trans. Neural Netw. Learning Syst. 35: 18237-18250. https://doi.org/10.1109/TNNLS. 2023.3313312
[60] Wang Y, Lu C, Wu P, Zhang X (2024b) Path planning for unmanned surface vehicle based on improved Q-Learning algorithm. Ocean Engineering 292: 116510. https://doi.org/10.1016/j.oceaneng.2023. 116510
[61] Wang Y, Xi M, Weng Y (2025) Intelligent path planning algorithm of autonomous underwater vehicle based on vision under ocean current. Expert Systems 42: e13399. https://doi.org/10.1111/exsy. 13399
[62] Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N (2016) Dueling network architectures for deep reinforcement learning. International Conference on Machine Learning. PMLR, 1995-2003
[63] Wang Z, Wen Z, Xia Q, Cai W (2022) Deep reinforcement learning based multi-UUV cooperative control for target capturing. IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech), Falerna, Italy, 1-6. https://doi.org/10.1109/DASC/PiCom/CBDCom/Cy55231.2022.9927810
[64] Wang Z, Xiang X, Duan Y, Yang S (2024c) Adversarial deep reinforcement learning based robust depth tracking control for underactuated autonomous underwater vehicle. Engineering Applications of Artificial Intelligence 130: 107728. https://doi.org/10.1016/j.engappai.2023.107728
[65] Wang Z, Zhang S, Feng X, Sui Y (2021) Autonomous underwater vehicle path planning based on actor-multi-critic reinforcement learning. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering 235: 1787-1796. https://doi.org/10.1177/0959651820937085
[66] Watkins CJCH (1989) Learning from delayed rewards
[67] Wei W, Wang J, Du J, Fang Z, Ren Y, Chen CLP (2023a) Differential game-based deep reinforcement learning in underwater target hunting task. IEEE Transactions on Neural Networks and Learning Systems 36(1):462-474. https://doi.org/10.1109/TNNLS.2023. 3325580
[68] Wei X, Wang H, Tang Y (2023b) Deep hierarchical reinforcement learning based formation planning for multiple unmanned surface vehicles with experimental results. Ocean Engineering 286: 115577. https://doi.org/10.1016/j.oceaneng.2023.115577
[69] Wen J, Wang A, Zhu J, Xia F, Peng Z, Zhang W (2024) Adaptive energy-efficient reinforcement learning for AUV 3D motion planning in complex underwater environments. Ocean Engineering 312: 119111. https://doi.org/10.1016/j.oceaneng.2024.119111
[70] Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8: 229-256. https://doi.org/10.1007/BF00992696
[71] Wong C, Yang E, Yan XT, Gu D (2018) Autonomous robots for harsh environments: a holistic overview of current solutions and ongoing challenges. Systems Science & Control Engineering 6: 213-219. https://doi.org/10.1080/21642583.2018.1477634
[72] Wynn RB, Huvenne VAI, Le Bas TP, Murton BJ, Connelly DP, Bett BJ, Ruhl HA, Morris KJ, Peakall J, Parsons DR, Sumner EJ, Darby SE, Dorrell RM, Hunt JE (2014) Autonomous underwater vehicles (AUVs): their past, present and future contributions to the advancement of marine geoscience. Marine Geology 352: 451-468. https://doi.org/10.1016/j.margeo.2014.03.012
[73] Xia J, Luo Y, Liu Z, Zhang Y, Shi H, Liu Z (2023) Cooperative multi-target hunting by unmanned surface vehicles based on multi-agent reinforcement learning. Defence Technology 29: 80-94. https://doi.org/10.1016/j.dt.2022.09.014
[74] Xu J, Huang F, Wu D, Cui Y, Yan Z, Zhang K (2021) Deep reinforcement learning based multi-AUVs cooperative decision-making for attack–defense confrontation missions. Ocean Engineering 239: 109794. https://doi.org/10.1016/j.oceaneng.2021. 109794
[75] Xu J, Zhang Z, Wang J, Han Z, Ren Y (2024) Multi-AUV pursuit-evasion game in the internet of underwater things: an efficient training framework via offline reinforcement learning. IEEE Internet of Things Journal 11: 31273-31286. https://doi.org/10.1109/JIOT.2024.3416616
[76] Yan Z, Yan J, Wu Y, Cai S, Wang H (2023) A novel reinforcement learning based tuna swarm optimization algorithm for autonomous underwater vehicle path planning. Mathematics and Computers in Simulation 209: 55-86. https://doi.org/10.1016/j.matcom.2023. 02.003
[77] Yang X, Wu Z, Yu J (2016) Design and implementation of a robotic shark with a novel embedded vision system. IEEE International Conference on Robotics and Biomimetics (ROBIO), Qingdao, China, 841-846. https://doi.org/10.1109/ROBIO.2016.7866428
[78] Yao X, Wang F, Yuan C, Wang J, Wang X (2021) Path planning for autonomous underwater vehicles based on interval optimization in uncertain flow fields. Ocean Engineering 234: 108675. https://doi.org/10.1016/j.oceaneng.2021.108675
[79] Yoerger DR, Jakuba M, Bradley AM, Bingham B (2007) Techniques for deep sea near bottom survey using an autonomous underwater vehicle. In: Thrun S, Brooks R, Durrant-Whyte H (Eds.), Robotics research, springer tracts in advanced robotics. Springer Berlin Heidelberg, Berlin, Heidelberg, 416-429. https://doi.org/10.1007/978-3-540-48113-3_36
[80] Yoo B, Kim J (2016) Path optimization for marine vehicles in ocean currents using reinforcement learning. J Mar Sci Technol 21: 334-343. https://doi.org/10.1007/s00773-015-0355-9
[81] You K, Wu H, Hsu Y, Song S (2020) A selected review of reinforcement learning-based control for autonomous underwater vehicles. Sci. Sin. -Inf. 50: 1798. https://doi.org/10.1360/SSI-2020-0065
[82] Yu H, Zhang S (2024) Collaborative task decision-making of multi-UUV in dynamic environments based on deep reinforcement learning. Ships and Offshore Structures: 1-11. https://doi.org/10.1080/17445302.2024.2391788
[83] Yu J, Wu Z, Yang X, Yang Y, Zhang P (2021) Underwater target tracking control of an untethered robotic fish with a camera stabilizer. IEEE Transactions on Systems, Man, and Cybernetics: Systems 51: 6523-6534. https://doi.org/10.1109/TSMC.2019. 2963246
[84] Yu R, Shi Z, Huang C, Li T, Ma Q (2017) Deep reinforcement learning based optimal trajectory tracking control of autonomous underwater vehicle. 36th Chinese Control Conference (CCC), IEEE, Dalian, China, 4958-4965. https://doi.org/10.23919/ChiCC. 2017.8028138
[85] Yuh J (2000) Design and control of autonomous underwater robots: a survey. Autonomous Robots 8: 7-24. https://doi.org/10.1023/A:1008984701078
[86] Zhang D, Pan G, Cao Y, Huang Q, Cao Y (2023) Depth control of a biomimetic manta robot via reinforcement learning. In: Sun F, Cangelosi A, Zhang J, Yu Y, Liu H, Fang B (Eds.), Cognitive Systems and Information Processing, Communications in Computer and Information Science. Springer Nature Singapore, Singapore, 59-69. https://doi.org/10.1007/978-981-99-0617-8_5
[87] Zhang J, Ren J, Cui Y, Fu D, Cong J (2024) Multi-USV task planning method based on improved deep reinforcement learning. IEEE Internet of Things Journal 11: 18549-18567. https://doi.org/10.1109/JIOT.2024.3363044
[88] Zhang T, Tian R, Yang H, Wang C, Sun J, Zhang S, Xie G (2022) From simulation to reality: a learning framework for fish-like robots to perform control tasks. IEEE Trans. Robot. 38: 3861-3878. https://doi.org/10.1109/TRO.2022.3181014
[89] Zhu S, Han G, Lin C, Tao Q (2024) Underwater target tracking based on hierarchical software-defined multi-AUV reinforcement learning: a multi-AUV advantage-attention actor-critic approach. IEEE Trans. on Mobile Comput. 23: 13639-13653. https://doi.org/10.1109/TMC.2024.3437376
[90] Zhu Y, Dai C, Xia J, Wu H, An L (2019) Adaptive Q-learning algorithm for AUV route planning. IOP Conf. Ser.: Mater. Sci. Eng. 569: 052029. https://doi.org/10.1088/1757-899X/569/5/052029

Memo

Memo:
Received date:2025-2-7;Accepted date:2025-5-8。<br>Foundation item:Funding Supported by the Youth Innovation Promotion Association of the Chinese Academy of Sciences (Grant No. 2023142) and in part by the National Natural Science Foundation of China (Grant No. 62033013).<br>Corresponding author:Sai Deng,E-mail:sai.deng@ia.ac.cn
Last Update: 2026-10-16