Addressing a new Paradigm Shift: An Empirical Study on Novel Project Characteristics for Foundation Model Projects

In recent years, data science and machine learning (ML) has become common across sectors and industries. Project methodologies are aimed at supporting projects and try catching up with ML trends and paradigm shifts. However, they are hardly successful, since still 80% of data science projects never reach deployment. The latest paradigm shift in the area of ML - the trend of generative AI and foundation models - changes the nature of data science projects and is not yet addressed by existing project methodologies. In this work, we present novel requirements that arise from real-world projects incorporating foundation models based on 29 case studies from the NLU domain. Furthermore, we assess existing data science methodologies and identify their shortcomings. Finally, we provide guidance on adapting projects to address the new challenges in the development and operation of foundation model based solutions.