Analysis and Evaluation of VLMs in multimodal scene understanding

The development of Autonomous Driving (AD) has transitioned from modular designs to End-to-End (E2E) architectures using Vision-Language Models (VLMs). VLMs generate intermediate reasoning traces which are critical for safety verification. However, existing evaluation metrics focus primarily on trajectory error or Question Answering (QA) accuracy leaving a gap in explicit reasoning assessment. This thesis addresses this gap by establishing a privileged judge framework that assesses the quality of VLMs reasoning traces. The evaluator uses a Large Language Model (LLM) with access to scenario ground truth metadata, which are unavailable to the driving model during inference. Three experiments evaluate the open weights Qwen model family using this framework. The first experiment investigates the influence of prompt structure and scoring rubrics on evaluator calibration. The second experiment tests the sensitivity of the evaluator by increasingly removing metadata context. The third experiment compares different formatting modalities of the scenario metadata regarding scoring quality and token consumption. The results demonstrate that evaluating internal reasoning processes reveals safety critical errors in grounding, coverage, kinematics, and reasoning validity that traditional metrics overlook. The findings confirm that structured grading rubrics successfully anchor evaluator scores and reduce variance. Additionally, advanced models exhibit a decoupling between choice selection and reasoning quality, highlighting the insufficiency of accuracy checks. Finally formatting comparisons establish trade offs between dense semantic representations, which optimize latency, and raw coordinate structures, which are required to verify fine grained kinematics.