CC BY 4.0Sinha, Saptarshi NeilSaptarshi NeilSinhaGraf, HolgerHolgerGrafWeinmann, MichaelMichaelWeinmann2025-07-172025-07-172025https://publica.fraunhofer.de/handle/publica/489671https://doi.org/10.24406/publica-490010.1016/j.isprsjprs.2025.06.00810.24406/publica-4900We propose a novel cross-spectral rendering framework based on 3D Gaussian Splatting (3DGS) that generates realistic and semantically meaningful splats from registered multi-view spectrum and segmentation maps. This extension enhances the representation of scenes with multiple spectra, providing insights into the underlying materials and segmentation. We introduce an improved physically-based rendering approach for Gaussian splats, estimating reflectance and lights per spectra, thereby enhancing accuracy and realism. In a comprehensive quantitative and qualitative evaluation, we demonstrate the superior performance of our approach with respect to other recent learning-based spectral scene representation approaches (i.e., XNeRF and SpectralNeRF) as well as other non-spectral state-of-the-art learning-based approaches. Our work also demonstrates the potential of spectral scene understanding for precise scene editing techniques like style transfer, inpainting, and removal. Thereby, our contributions address challenges in multi-spectral scene representation, rendering, and editing, offering new possibilities for diverse applications.enBranche: HealthcareBranche: BioeconomyBranche: Cultural and Creative EconomyResearch Line: Computer graphics (CG)Research Line: Computer vision (CV)Research Line: Machine learning (ML)LTA: Generation, capture, processing, and output of images and 3D modelsComputer graphicsDeep learningSpectral imaging3D reconstructionAppearance modelingScene understandingNovel view synthesis3D Gaussian splattingjournal article