Austenite grain growth during welding is a critical factor for controlling weld microstructure in addition to nominal composition and thermal cycles. Recently, experimental data suggesting a decrease in martensite start temperature with a decrease in prior austenite grain size has been published. However, the actual sensitivity of this phenomenon on residual stresses evolution in the heat-affected zone has not been investigated, yet. Therefore, a numerical model was modified to consider this phenomenon. Numerical simulations were performed for welding of a low-alloy structural steel with minimum yield strength of 355 MPa (S355J2+N) and a heat-resistant steel P91 or 9Cr-1Mo, respectively. The results clarify the influence of prior austenite grain size on the residual stress development and show the importance martensite transformation temperatures and final martensite fraction. Consequently, the residual stress evolution of P91, which completely transforms to martensite while cooling, based on the enhanced model leads to maximum stress differences of 200 MPa in the heat-affected zone.