COMPASS: A Process Mining-based Methodology for Prompt Optimization of Large Language Model Agents

Exploring and optimizing the behavior of LLM agents in complex environments is challenging. We present COMPASS, a methodology that systematically applies Process Mining to discover, analyze, and refine agent behavior. COMPASS consists of five phases: planning the project, extracting data from the LLM agent, transforming data into event logs, exploring and analyzing agent behavior, and providing feedback through conformance checking or prompt design guidelines. We evaluate COMPASS in a case study with an LLM agent that uses three tools to generate SPARQL queries while exploring a complex knowledge graph. Through COMPASS, we identified ineffective behavior patterns and optimized prompts to improve performance. Our methodology supports data-driven prompt optimization with interpretable behavioral models, enhancing explainability and reliability in complex environments. This ongoing research aims to systematically improve agent performance through exploratory behavioral analysis that increases transparency in agents’ decision-making processes.