Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis

Neural rendering techniques allow for accurate reconstruc-tion of the geometry and color appearance of 3D scenes. Some methodshave extended their use to additional imaging modalities, such as multi-spectral, infrared, or polarimetric data. However, all of these approachesrequire expensive sensors and calibrated setups to capture new multi-modal frames for each new scene. We propose Spectral and PolarimetricImplicit Learned Representation (SPoILeR), a novel method to obtainmulti-view consistent renderings of unconventional modalities for sceneswhere either only RGB frames or very few of the additional modali-ties are available. Thanks to a multimodal pre-training phase, the modellearns the mutual correlation between different modalities. This step al-lows predicting accurate renderings of unconventional modalities duringa fine-tuning phase supervised only by RGB images. Experimental resultsshow that the approach can accurately render infrared, polarimetric, andmultispectral frames for scenes where no input sample captured by thesetypes of sensors is provided.