|Table of Contents|

Citation:
　Aniket Malviya,Suresh Rajendran,Xueqian Zhou.Transfer Learning for Deep Reinforcement Learning-Based Path Following of Autonomous Surface Vessels[J].Journal of Marine Science and Application,2026,(3):728-744.[doi:10.1007/s11804-026-00820-x]
Click and Copy

Transfer Learning for Deep Reinforcement Learning-Based Path Following of Autonomous Surface Vessels

Info

Title:
Transfer Learning for Deep Reinforcement Learning-Based Path Following of Autonomous Surface Vessels
Author(s):
Aniket Malviya1;  Suresh Rajendran1;  Xueqian Zhou2
Affilations:
Author(s):
Aniket Malviya1;  Suresh Rajendran1;  Xueqian Zhou2
1 Ocean Engineering, Indian Institute of Technology Madras, Chennai, 600036, India;
2 Harbin Engineering University, Shipbuilding Engineering University, Harbin, 150001, China
Keywords:
Deep reinforcement learning (DRL); Autonomous surface vessels (ASVs); Deep deterministic policy gradient (DDPG); Transfer learning; Proportional-integral-derivative (PID) controller; Line of sight (LOS) guidance algorithm
分类号:
-
DOI:
10.1007/s11804-026-00820-x
Abstract:
Deep Reinforcement Learning (DRL) offers a powerful, model-free, and data-driven approach for the navigation and control of Autonomous Surface Vessels (ASVs). The primary challenge, however, lies in the extensive training required for an agent to converge to an effective policy within a complex simulation, leading to significant computational overhead. This paper presents a multi-stage training framework that uses Transfer Learning to pass knowledge between different simulation models, resulting in a highly robust DRL controller for ASVs. The proposed framework utilizes the Deep Deterministic Policy Gradient (DDPG) algorithm to develop the data-driven controller. First, a foundational policy is efficiently learned using a simplified first-order Nomoto dynamics and second-order Nomoto dynamics, which captures the fundamental vessel dynamics. This pre-trained policy is then transferred to a complex, nonlinear Manoeuvring Modelling Group (MMG) model, significantly accelerating training convergence. Subsequently, the agent is fine-tuned within the MMG simulation with environmental disturbances. The models are evaluated on various trajectories during testing to ensure robust performance. The accuracy of the DRL controller is assessed by measuring heading error (eψ) and cross-track error (ye). A traditional Proportional-Integral-Derivative (PID) controller is implemented and compared to benchmark the DRL controller’s effectiveness, to highlight the relative advantages and limitations of each approach.

References:

[1] Busoniu L, De Bruin T, Toli? D, Kober J, Palunko I (2018) Reinforcement learning for control: Performance, stability, and deep approximators. Annual Reviews in Control 46: 8–28. https://doi.org/10.1016/j.arcontrol.2018.09.005
[2] Katebi MR, Moradi MH (2001) Predictive PID controllers. IEE Proceedings-Control Theory and Applications, 148(6): 478–487. https://doi.org/10.1049/ip-cta:20010786
[3] Kiumarsi B, Vamvoudakis KG, Modares H, Lewis FL (2017) Optimal and autonomous control using reinforcement learning: A survey. IEEE transactions on neural networks and learning systems 29(6): 2042–2062. https://doi.org/10.1109/TNNLS.2017.2773458
[4] Lekkas AM, Fossen TI (2013) Line-of-sight guidance for path following of marine vehicles. Advanced in marine robotics 5: 63–92
[5] Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv: 1509.02971. https://doi.org/10.48550/arXiv.1509.02971
[6] Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M (2013) Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.5602. https://doi.org/10.48550/arXiv.1312.5602
[7] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Hassabis D (2015) Human-level control through deep reinforcement learning. Nature 518(7540): 529–533. https://doi.org/10.1038/nature14236
[8] Pan SJ, Yang Q (2009) A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22(10): 1345–1359. https://doi.org/10.1109/TKDE.2009.191
[9] Paramesh S, Rajendran S (2021) A unified seakeeping and manoeuvring model with a PID controller for path following of a KVLCC2 tanker in regular waves. Applied Ocean Research 116: 102860. https://doi.org/10.1016/j.apor.2021.102860
[10] Perera LP, Ferrari V, Santos FP, Hinostroza MA, Soares CG (2014) Experimental evaluations on ship autonomous navigation and collision avoidance by intelligent guidance. IEEE Journal of Oceanic Engineering 40(2): 374–387. https://doi.org/10.1109/JOE.2014.2304793
[11] Puterman ML (2014) Markov decision processes: discrete stochastic dynamic programming. John Wiley and Sons
[12] Salvesen N (1974) Second-Order Steady-State Forces and Moments on Surface Ships in Oblique Regular Waves. DTNSRDC Report 3170. David Taylor Naval Ship Research and Development Center, Bethesda, Maryland
[13] Sandeepkumar R, Rajendran Suresh, Mohan Ranjith, Pascoal, Antonio (2021) A unified ship manoeuvring model with a nonlinear model predictive controller for path following in regular waves. Ocean Engineering. 243. https://doi.org/10.1016/j.oceaneng.2021.110165
[14] Silver D, Lever G, Heess N, Degris T, Wierstra D, Riedmiller M (2014) Deterministic policy gradient algorithms. In International conference on machine learning (pp. 387–395). Pmlr https://proceedings.mlr.press/v32/silver14.html
[15] Sivaraj S, Dubey A, Rajendran S (2023) On the performance of different deep reinforcement learning based controllers for the path-following of a ship. Ocean Engineering 286: 115607. https://doi.org/10.1016/j.oceaneng.2023.115607
[16] Sutton RS, Barto AG (2018) Reinforcement learning: An introduction second edition. Adaptive computation and machine learning: The MIT Press Cambridge MA and London
[17] Taylor ME, Stone P (2009) Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research: 10(7). https://doi.org/10.48550/arXiv.2009.07888
[18] Tomera M (2017) Fuzzy self-tuning PID controller for a ship autopilot. In Marine Navigation (pp. 93–103). CRC Press. https://doi.org/10.1201/9781315099132
[19] Yasukawa H, Yoshimura Y (2015) Introduction of MMG standard method for ship maneuvering predictions. Journal of marine science and technology 20(1): 37–52. https://doi.org/10.1007/s00773-014-0293-y

Memo

Memo:
Received date:2025-5-24。<br>Corresponding author:Suresh Rajendran,E-mail:sureshr@iitm.ac.in
Last Update: 2026-06-25