A Modular Pipeline for Context-Aware AI Assistance in XR Environments

Integrating large language models into virtual environments opens the door for countless enhancement opportunities deemed computationally infeasible before the commercialization of AI. While this is the case, consumer hardware is still not equipped to deal with the processing requirements of local realtime 3D rendering and LLM inference. This paper presents a modular approach to building a computation pipeline designed to enable context-aware AI assistance in 3D environments, making use of remote rendering technology to offload computing to remote servers. The pipeline consists of a speech-to-text module, a text-to-speech module, and an LLM-based action system controlling a virtual agent. The architecture enables a composable approach to deploying immersive LLM assisted 3D environments that can be used in gaming and educational systems alike.