Five Measures to Counter Obscure Hazardous Failure Conditions in Foundation Model Systems: A Practical Guide for Safety Engineers

Machine learning models for natural language processing have achieved remarkable performance across many domains, yet their adoption in safety engineering remains limited. A key barrier are hazardous failure conditions - such as hallucinations - which arise from an insufficient functionality of a foundation model system and must be controlled in safety-critical contexts. This article examines such behavior through a safety lens and traces root causes for obscure
hazardous failure conditions in generic foundation model setups. From this analysis, we distill five broadly applicable concepts that provide a structured way to counter these challenges (narrow domain, setup diligence, uncertainty estimation and interpretability, iterative approaches, hybrid approaches). Together, they offer a principled starting point for deriving concrete design and assessment practices for safety engineers working with language-model-based systems. We
illustrate our approach through three representative use cases that demonstrate how these five countermeasures can be applied in real-world safety engineering scenarios.