Automated Assessment of Maritime Simulator Training: A Data-Driven Approach Using Large Language Models

This paper presents the technical development and implementation of an automated assessment system within the i-Master EU project for maritime simulator training. Automating performance evaluation in maritime training presents challenges due to the diverse nature of training tasks and the need for flexible assessment methods. To address this, the paper outlines architecture designed to support a wide range of basic tasks by enabling flexible task definition and automatic assignment of relevant metrics and evaluation criteria. This adaptability ensures that assessments remain consistent while allowing customization to fit different training scenarios, instructor styles and student needs. The system performs evaluation by collecting structured, self-describing metrics that represent individual skills during an exercise. These metrics are aggregated at the end of a training session and assessed against predefined evaluation criteria before being processed by a Large Language Model (LLM). The LLM generates tailored, natural language feedback that mirrors instructor-style debriefings, providing personalized insights into student performance. All recorded metrics, assessment criteria, and generated evaluations are stored in a database with version control, ensuring retrospective re-evaluation as assessment models improve. By providing the history of assessments the system allows for iterative refinement of both evaluation parameters and LLM prompting strategies, ensuring adaptability to evolving maritime training requirements. The technical implementation provides a foundation for future validation studies with students and instructors at maritime universities. This work presents a novel technical approach to automated maritime assessment, demonstrating the feasibility of LLM integration in maritime training systems.