Coded video sequence generating apparatus has intermediate layer predictor to calculate enhancement prediction residual error pictures by using enhancement sequence and base sequence of residual error pictures

A base picture coder calculates coded base scaling layer from base sequence of residual error pictures by using base motion data. A compensator determines enhancement motion data. A predictor calculates enhancement sequence of residual error pictures. An intermediate layer predictor calculates enhancement prediction residual error pictures by using enhancement sequence and base sequence of residual error pictures. An enhancement picture coder codes enhancement prediction residual error pictures to obtain coded enhancement scaling layer.