Optimizing Lunar Descent Efficiency Using Reinforcement earning: A Computational Physics Investigation of Fuel-Constrained Landing Dynamics

Planetary landings can only be autonomous by having control systems that are able to balance accuracy and efficiency under nonlinear conditions.

This research project examines the use of reinforcement learning (RL) which in this case is Proximal Policy Optimization (PPO) algorithm to simulate and optimize the dynamics of lunar descent in a one dimensional environment with Newtonian physics.

The agent was trained to successfully complete soft landings at different fuel and thrust-cost settings, where a reward function punished the use of fuel and high terminal velocity.

On 40 000 training episodes, the agent always shared similar stable policies that reduced fuel usage and yet reached safe touchdown speeds (less than 2 m/s-1).

Quantitative analysis showed the existence of a strong negative correlation between available fuel and final velocity and unique dual-phase kinetics of fuel-use patterns were found which were similar to real-world powered-descent sequences.

These findings confirm that reinforcement learning is a physically consistent, adaptive control technique to optimize descent, and has the potential to be useful in autonomous guidance systems in future trips to the moon and other planets.

Robotics and AI / Aerospace AI Systems

1. Blackmore, L., Fathpour, N., & Sutter, B. (2010). Autonomous precision landing of space rockets. AIAA Guidance, Navigation, and Control Conference. [Google Scholar] [Crossref]

2. Bryson, A. E. (1975). Applied optimal control: Optimization, estimation, and control. Taylor & Francis. [Google Scholar] [Crossref]

3. Chobotov, V. (2001). Orbital mechanics (3rd ed.). AIAA. [Google Scholar] [Crossref]

4. Farama Foundation. (2023). Gymnasium documentation. https://gymnasium.farama.org/ [Google Scholar] [Crossref]

5. Fujimoto, S., van Hoof, H., & Meger, D. (2018). Addressing function approximation error in actor-critic methods. Proceedings of the 35th International Conference on Machine Learning. [Google Scholar] [Crossref]

6. Gupta, M., & Kochenderfer, M. (2019). Online planning for autonomous planetary landing. Journal of Guidance, Control, and Dynamics, 42(6), 1256–1267. [Google Scholar] [Crossref]

7. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic algorithms and applications. arXiv:1812.05905. [Google Scholar] [Crossref]

8. Harris, C., & D’Souza, C. (2011). Powered descent guidance and control for Mars landing. NASA Technical Reports. [Google Scholar] [Crossref]

9. Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep reinforcement learning that matters. Proceedings of the AAAI Conference on Artificial Intelligence. [Google Scholar] [Crossref]

10. Humphries, S. (2020). Propulsion efficiency modelling for low-fuel space landing operations. Aerospace Propulsion Journal, 9(4), 200–214. [Google Scholar] [Crossref]

11. Hunter, J. D. (2007). Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3), 90– 95. [Google Scholar] [Crossref]

12. ISRO. (2023). Chandrayaan-3 mission report. Indian Space Research Organisation. [Google Scholar] [Crossref]

13. Kakade, S. (2002). A natural policy gradient. MIT Press. [Google Scholar] [Crossref]

14. Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., & Wierstra, D. (2016). Continuous control with deep reinforcement learning. arXiv:1509.02971. [Google Scholar] [Crossref]

15. Mattingly, J. (2017). Elements of propulsion: Gas turbines and rockets (2nd ed.). AIAA Education Series. [Google Scholar] [Crossref]

16. Mihail, J. C. (2022). Modelling lunar descent dynamics using computational physics. Acta Astronautica, 175, 58–69. [Google Scholar] [Crossref]

17. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. [Google Scholar] [Crossref]

18. NASA. (1969). Apollo 11 mission report. NASA Headquarters. [Google Scholar] [Crossref]

19. NASA. (2019). Lunar landing and descent trajectory analysis. NASA Technical Publications. [Google Scholar] [Crossref]

20. NASA. (2020). Artemis program: Lunar surface mission planning. NASA Exploration Systems Directorate. [Google Scholar] [Crossref]

21. Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., Dormann, N., & Plappert, M. (2021). Stablebaselines3: Reliable reinforcement learning framework for Python. Journal of Machine Learning Tools. [Google Scholar] [Crossref]

22. Raman, V., & Patel, S. (2023). Autonomous lunar descent optimization using machine learning models. Journal of Aerospace Systems Engineering, 14(2), 44–60. [Google Scholar] [Crossref]

23. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). Proximal policy optimization algorithms. arXiv:1707.06347. [Google Scholar] [Crossref]

24. Serway, R. A., & Jewett, J. W. (2014). Physics for scientists and engineers (9th ed.). Brooks Cole. [Google Scholar] [Crossref]

25. Silver, D., Hubert, T., Schrittwieser, J., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and Go. Science, 362(6419), 1140–1144. [Google Scholar] [Crossref]

26. Sutton, G. P., & Biblarz, O. (2017). Rocket propulsion elements (9th ed.). Wiley. [Google Scholar] [Crossref]

27. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press. [Google Scholar] [Crossref]

28. Tsiolkovsky, K. E. (1903). Exploration of cosmic space by means of reaction devices. Russian Academy of Sciences. [Google Scholar] [Crossref]

29. Vallado, D. A. (2013). Fundamentals of astrodynamics and applications (4th ed.). Microcosm Press. [Google Scholar] [Crossref]

• "Towards GDP to GEP-Centric Model: A Proposed GEP Index Framework and Its Application in Haryana"
• Physicochemical Analysis of Petroleum Products in Selected Depots in Calabar Metropolis Cross River State and The Effects on Motor Engine
• Impact of Employee Welfare Measures on Workforce Well-Being in SBI: Insights from the Public Banking Sector
• Problem of Small-Scale Farmers in Agricutlure Sector in Tirunelveli Taluk
• Globalization, Economic Development, and Ecological Footprint in Tunisia: A QARDL Approach