Towards Continuous Audit-based Certification for MLOps

DevOps did increase the frequency of the cycle in which software can be developed and deployed while maintaining quality. The MLOps paradigm is about to do the same for machine learning. Models get deployed and quality checked in production until their obsolescence triggers a new cycle. Traditional point-in-time certification cannot keep up with this pace which requires a more frequent highly automated type of certification and compliance checks. To achieve this high grade of automation our approach utilizes the artifacts produced at each run of the MLOps cycle for quality measurements based on standards like ISO 25012. In this paper, we will introduce three main contributions: a continuous audit methodology, a quality attributes catalog, and a trustworthy infrastructure for continuous auditing. For the methodology, we describe the roles and procedures that are necessary for enabling and executing continuous audit-based certification. We have compiled an initial catalog of quality attributes for data and model quality as well as for the quality of the MLOps implementation. To put continuous audit-based certification into production we are introducing a four layered architecture separating the responsibilities of the certification process. Additionally, we are laying out a concept of implementing a tampered proof automated audit concept that collects reliable evidence.