Evaluation of action spaces for reinforcement learning in optical design

Nowadays, sophisticated ray tracing software packages are used for the design of optical systems, including local and global optimization algorithms. Nevertheless, the design process is still time-consuming with many manual steps, taking days or even weeks until an optical design is finished. To address this shortcoming, with reinforcement learning, an agent can be trained to use ray tracing and optimization software designing an optical system. In this setting, the agent can modify the current state of the system with a predefined set of actions. One of the primary challenges is the selection of an appropriate action space. Different types of discrete and continuous action spaces are compared and their advantages and disadvantages in terms of the cumulated reward, convergence rate and resulting optical design are examined.