Towards engaging MDPs

One of the challenges that a computer game developer meets when creating a new game is setting the difficulty ``right''. Not only is it a complicated task to balance all the game parameters, but the developer also needs to cater for players with very different skill levels. Providing a game with an ability to automatically scale the difficulty depending on the current player would make the games more engaging over longer time. While a few commercial games boast about having such a system, to the best of our knowledge it was not researched as a learning problem. In this paper we first give a problem definition of the automatic difficulty scaling problem we call \emph{Engagement Problem}. Then, we also outline a framework based on nested Markov Decision Processes, called \emph{Engaging Markov Decision Process} for solving it. Preliminary experiments in a small grid world show the effectiveness of our approach.