MLLM Interpreter: A Model-Agnostic Method to Explain Multimodal Large Language Models

We present MLLM Interpreter, a model-agnostic framework conditioned on a selected text-span, cross-modal attribution in multimodal large language models (MLLMs). Unlike methods that explain only text, only vision, or consider explanations only at the level of the entire response—our approach quantifies how visual segments and verbal tokens jointly support a user-selected answer text-span. The pipeline is purely black-box: it proposes image regions via open-vocabulary detection with mask refinement, forms interpretable text units, perturbs features through targeted masking, scores the effect using a semantic value function, and estimates contributions by fitting a linear surrogate model in the Shapley framework over sampled coalitions. It requires neither parameters, gradients, nor access to attention weights, enabling uniform application to closed- and open-source MLLMs. For evaluation, we introduce modality-specific metrics: verbal and visual scores reward relevant items placed above a neutral threshold. Ablations across value functions, zero-shot detectors, Shapley estimators, and sampling budgets show consistently strong, robust scores. The method is time efficient in practice, supports manual feature selection when needed, and provides actionable evidence for auditing, diagnosing hallucinations, and debugging. MLLM Interpreter advances transparent, target-specific, and multimodal interpretability for modern language models.