SegDecon bridges histology and transcriptomics through AI-based nuclei segmentation and image-informed spatial deconvolution

Precise spatial mapping of cellular composition is a central goal in spatial transcriptomics (ST), yet current methods often assume uniform or manually estimated cell counts across spatial spots, potentially distorting biological interpretation. Here, we present SegDecon, a computational framework that integrates image-derived cell count estimation into Bayesian deconvolution. SegDecon enhances nuclei segmentation using Hue-Saturation-Value (HSV) color space transformation, morphological filtering, and deep learning-based instance segmentation. It quantifies nuclei per spatial spot and refines cell-type deconvolution through tailored Gamma priors in a modified cell2location model. Evaluated on high-resolution mouse brain ST data, SegDecon demonstrates improved correlation with ground truth, particularly in resolving low-abundance and spatially restricted cell types. This approach provides a reproducible and accessible solution to bridge histology with transcriptomic deconvolution, improving both resolution and biological fidelity. Source code is available at: https://github.com/CiiM-Bioinformatics-group/SegDecon