Uncertainties complicate the task of designing optimal controllers for complex systems. This work introduces FMUGym, a novel open source interface that connects reinforcement learning libraries following the Gymnasium standard with co-simulation Functional Mock-up Units. As the latter encapsulate the model of the control plant, FMUGym can transform them into an environment of a reinforcement learning setup. FMUGym allows to inject uncertainties into system dynamics, inputs and outputs during training. This fosters robust control policies that adapt to possible variations and aims to bridge the simulation-to-reality gap. We demonstrate FMUGym's effectiveness by training an agent to control a nonlinear system with and without uncertainties, highlighting the benefit of noise injection. A second example showcases applicability in heating, ventilation and air conditioning systems. The source code and additional resources for this project are available on GitHub (https://github.com/Fraunhofer-IIS/fmugym), with further development planned based on community feedback.