Krauss, ChristopherChristopherKraussBalitzki, EmilEmilBalitzkiBassbouss, LouayLouayBassboussUpravitelev, MaxMaxUpravitelevAn, Truong-SinhTruong-SinhAnEl Tamimi, TarekTarekEl TamimiAltun, DanielaDanielaAltunReray, LisaLisaRerayKaragülle, MehmetMehmetKaragülle2024-07-302024-07-302024-06-01https://publica.fraunhofer.de/handle/publica/47227910.1007/978-3-031-60609-0_16The paper explores the opportunities and challenges for metaverse learning environments with AI-Assistants based on Large Language Models. A proof of concept based on popular but proprietary technologies is presented that enables a natural language exchange between the user and an AI-based medical expert in a highly immersive environment based on the Unreal Engine. The answers generated by ChatGPT are not only played back lip-synchronously, but also visualized in the VR environment using a 3D model of a skeleton. Usability and user experience play a particularly important role in the development of the highly immersive AI-Assistant. The proof of concept serves to illustrate the opportunities and challenges that lie in the merging of large language models, metaverse applications and educational ecosystems, which are self-contained research areas. Development strategies, tools and interoperability standards will be presented to facilitate future developments in this triangle of tension.enAI-AssistantsEducationInteroperabilityLarge Language ModelsLearning TechnologiesLLMconference paper