|Table of Contents|

Citation:
　Oleksiy Bondarenko,Yasushi Kitagawa,Ryohei Sawada,et al.Enhancing Transient Performance of Propulsion Engines in Actual Sea Using Reinforcement Learning-Based Proportional-Integral-Derivative Gain Tuning[J].Journal of Marine Science and Application,2026,(4):1062-1077.[doi:10.1007/s11804-025-00620-9]
Click and Copy

Enhancing Transient Performance of Propulsion Engines in Actual Sea Using Reinforcement Learning-Based Proportional-Integral-Derivative Gain Tuning

Info

Title:
Enhancing Transient Performance of Propulsion Engines in Actual Sea Using Reinforcement Learning-Based Proportional-Integral-Derivative Gain Tuning
Author(s):
Oleksiy Bondarenko1;  Yasushi Kitagawa2;  Ryohei Sawada3;  Tetsugo Fukuda1
Affilations:
Author(s):
Oleksiy Bondarenko1;  Yasushi Kitagawa2;  Ryohei Sawada3;  Tetsugo Fukuda1
1. Power and Energy System, National Maritime Research Institute, Tokyo 1810004, Japan;
2. Ship Hydrodynamics Performance Evaluation, National Maritime Research Institute, Tokyo 1810004, Japan;
3. Knowledge & Data System, National Maritime Research Institute, Tokyo 1810004, Japan
Keywords:
Propulsion engine; Proportional-integral-derivative speed control; Adaptive tuning; Reinforcement learning
分类号:
-
DOI:
10.1007/s11804-025-00620-9
Abstract:
The engine speed control system is critical to marine propulsion, particularly for ships navigating in wave conditions. Fluctuations in propeller torque, caused by wave-induced variations in the ship’s speed and direction, can lead to undesirable effects such as increased fuel consumption and greenhouse gas emissions, increased engine wear and tear, and degraded performance. In the field of marine diesel engines, a proportional-integral-derivative (PID) control algorithm is the most widely used method for speed regulation. Aiming to maintain a constant engine speed, the PID governor continuously adjusts the fuel injection into the engine cylinders in response to propeller torque fluctuations. The quality of control, and thus engine performance, depends on a set of PID control gains. However, these optimal PID gains are typically determined during engine testbed commissioning and remain fixed, which leads to suboptimal performance under varying operational conditions. While model-based tuning techniques can achieve satisfactory results in relatively simple control tasks, particularly when the number of tunable parameters is small, adaptive tuning of controller parameters, while meeting multiple, often conflicting objectives, presents a considerable challenge. This paper explores the application of a reinforcement learning (RL) algorithm to adaptively tune PID gains, aiming to optimize engine performance under varying sea conditions and operational demands. A specific reward function is designed to effectively balance variations in engine shaft rotational speed and fuel pump index, ensuring optimal engine efficiency. The performance of the trust-region policy optimization and proximal policy optimization algorithms is evaluated for their suitability in continuous-action space control, demonstrating stable convergence and robustness. Simulation results indicate that the proposed RL algorithm outperforms the reference PID controller, offering promising results for adaptive gain scheduling in propulsion system control.

References:

[1] Alejandro Sanjines U, Maisincho Jivaja A, Asanza V, Lorente Leyva LL, Peluffo Ordóñez DH (2023) Adaptive PI controller based on a reinforcement learning algorithm for speed control of a DC motor. Biomimetics 8(5): 434. https://doi.org/10.3390/biomimetics 8050434
[2] Bertsekas DP (2012) Dynamic Programming and Optimal Control.Vol. II, 4th Edition: Approximate Dynamic Programming 2012. Athena Scientic, 4th ed
[3] Blake XJ, Aphiratsakun N (2024) Reinforcement learning for PID gain tuning in ball-and-beam systems: A comparative study. 2024 1st International Conference on Robotics, Engineering, Science, and Technology (RESTCON) 187-190. https://doi.org/10.1109/RESTCON60981.2024.10463594
[4] Bondarenko O, Kashiwagi M (2012) Ensuring safe operation of ship propulsion plant in extreme sea condition. 11th International Marine Design Conference (IMDC) 3: 83-96. https://doi.org/10.13140/RG. 2.2.30079.73124
[5] Bondarenko O, Fukuda T (2018) Intelligent air management for ultimate flexibility of ship propulsion plant operation-1st Report: Experiment and Simulation. The Japan Institute of Marine Engineers 53(3): 348-354. https://doi.org/10.5988/jime.53.348
[6] Bondarenko O, Fukuda T (2020) Development of a diesel engine’s digital twin for predicting propulsion system dynamics. Energy 196: 117126. https://doi.org/10.1016/j.energy.2020.117126
[7] Bondarenko O, Kitagawa Y (2021) Application of the Kalman filtering technique for nonlinear state estimation in propulsion system. Journal of Marine Science and Technology 26: 618–631.https://doi.org/10.1007/s00773-020-00763-0
[8] Bondarenko O, Kitagawa Y (2024) A comprehensive propulsion system analysis: an integrated approach utilizing engine simulation and free-running model experiments. J. of Mar. Eng. & Techn., 1-12. https://doi.org/10.1080/20464177.2024.2412442
[9] Calvo Rolle JL; García RF, López García H, Machón González IJ (2009) Developed an expert system of an empirical method to choose correct expressions for PID controllers tuning in open loop. Proceedings of the 35th Annual Conference of IEEE Industrial Electronics. Piscataway: IEEE 1-6: 2044-2049. http://dx.doi.org/10.1109/IECON.2009.5415064
[10] Dogru O, Velswamy K, Ibrahim F, Wu Y, Sundaramoorthy AS, Huang B, Xu S, Nixon M, Bell N (2022) Reinforcement learning approach to autonomous PID tuning. Computers & Chemical Engineering 161: 107760. https://doi.org/10.1016/j.compchemeng. 2022.107760
[11] Hendrics E (1989) Mean value modelling of large turbocharged two-stroke diesel engines. Journal of Engines 98: 986-998. https://doi.org/10.4271/890564
[12] IMO (2020) Fourth Greenhouse Gas Study. [accessed 2023 Nov 1]. https://www.imo.org/en/ourwork/Environment/Pages/Fourth-IMO- Greenhouse-Gas-Study-2020.aspx
[13] Iwano J, Hirooka H, Kondo K, Shigaki T (2009) Onboard testing results for new fuel efficient main engine governor system–Development of NABTESCO 70X. Journal of Marine Engineering 44(5): 105-110. https://doi.org/10.5988/jime.44.787. (in Japanese)
[14] Karray F, Gueaieb W, Al-Sharhan S (2002) The hierarchical expert tuning of PID controllers using tools of soft computing. IEEE Transactions on Systems Man and Cybernetics, Part B-Cybernetics 32(1): 77-90. https://doi.org/10.1109/3477.979962
[15] Kitagawa Y, Bondarenko O, Tsukada Y (2019) An experimental method to identify a component of wave orbital motion in propeller effective inflow velocity and its effects on load fluctuations of a ship main engines in waves. Applied Ocean Research 92: 101922. https://doi.org/10.1016/j.apor.2019.101922
[16] Kitagawa Y, Kashiwagi M (2019) A strip method including nx-related terms and its effects on propeller inflow velocity in waves. Journal of the Japan Society of Naval Architects and Ocean Engineers 29: 39-50. https://doi.org/10.2534/jjasnaoe.29.39
[17] Lloyd ARJM (1998) Seakeeping, Ship Behaviour in Rough Weather. Hoboken: John Willey & Sons
[18] Ray A (1980) Dynamic modelling of power plant turbines for controller design. Applied Mathematical Modelling 4(2): 109-112. https://doi.org/10.1016/0307-904X(80)90114-6
[19] Schulman J, Levine S, Moritz P, Jordan MI, Abbeel P (2015) Trust region policy optimisation. Proceedings of the 32nd International Conference on Machine Learning 37: 1889-1897
[20] Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimisation algorithms arXiv: abs/1707.06347. https://doi.org/10.48550/arXiv.1707.06347
[21] Syromyatnikov VF (1983) Fundamentals of automation and complex automation of ship’s power plants. (in Rus.). Transport, Moscow
[22] Theotokatos G (2010) On the cycle mean value modelling of a large two-stroke marine diesel engine. Proceedings of the Institution of Mechanical Engineers, Part M: Journal of Engineering for the Maritime Environment 224(3): 193-205. https://doi.org/10.1243/14750902JEME188
[23] Tsujimoto M, Sogihara N, Kuroda M, Sakurada A (2015) Development of a ship performance simulator in actual seas. ASME 2015 34th Interna tional Conference on Ocean, Offshore and Arctic Engineering. Newfoundland: ASME OMAE2015-41708, Vo11T12A047. https://doi.org/10.1115/ omae2015-41708
[24] van Veldhuizen V (2020) Autotuning PID control using Actor-Critic Deep Reinforcement Learning. Amsterdam: University of Amsterdam. https://doi.org/10.48550/arXiv.2212.00013
[25] Wang H, Ricardez-Sandoval LA (2024) A deep reinforcement learning-based PID tuning strategy for nonlinear MIMO systems with time-varying uncertainty. IFAC-PapersOnline 58(14): 887-892. https://doi.org/10.1016/j.ifacol.2024.08.449
[26] Xu XG, Chen XH, Zheng SG (2007) Research on fuzzy PID control to electronic speed regulator. Proceedings of the SPIE 6794: article id. 679407
[27] Xiros N (2002) Robust control of diesel ship propulsion. Berlin: Springer
[28] Xiros N (2004) PID marine engine speed regulation under full load conditions for sensitivity H∞-norm specifications against propeller disturbance. Journal of Marine Engineering & Technology 3(2): 3-11. https://doi.org/10.1080/20464177.2004.11020179
[29] Zhou Y, Zhang QJ, Zhou QX, Jiang HK (2011) Research on nonlinear PID control of digital governor for diesel engine. In: Zhang W (eds) Advanced Technology in Teaching. Advances in Intelligent and Soft Computing. Berlin: Springer 163: 295-303. https://doi.org/10.1007/978-3-642-29458-7_45

Memo

Memo:
Received date:2025-2-7;Accepted date:2025-5-8。<br>Corresponding author:Oleksiy Bondarenko，E-mail:bond@m.mpat.go.jp
Last Update: 2026-10-16