Modifications to Neural Radiance Fields for Object Pose Estimation on unseen Object Appearances

Due to the rise of differential and neural rendering, solving the long-standing computer vision task of pose estimation via analysis-by-synthesis has been explored more in recent years. The core idea is to reconstruct the scene first in order to understand it. The leading approach for finding a high quality neural scene representation from 2DRGB images only are Neural Radiance Fields (NeRF)[25]. They enable generating images from novel view points with great fidelity. Its application in the field of pose estimation has been proposed in iNeRF[47] which inverts a trained NeRF, that already represents a scene, to retrieve the pose from an unseen view. However, the performance decreases when the new view displays lighting variation or occlusion as NeRF is only able to represent static scenes in a controlled setting. Thus, in order to mitigate the need of retraining a new model for a slightly changed scene, this work proposes to replace the underlying vanilla NeRF with NeRF in the Wild (NeRF-W)[23] in iNeRF, denoted as iNeRF-W. With its extension of two latent codes, NeRF-W is able to represent scene variation and its authors focused on uncontrolled, “in the wild” training images. With more introduced adaptations, such as a joint optimization scheme, weareable to prove its superiority over vanilla iNeRF and are able to outperform iNeRF-VAE[16] that also focused on the generalization of iNeRF.