Under One Sun: Multi-Object Generative Perception of Materials and Illumination

We introduce Multi-Object Generative Perception (MultiGP),a generative inverse rendering method for stochastic sampling of all radio-metric constituents—reflectance, texture, and illumination—underlyingobject appearance from a single image. Our key idea to solve this in-herently ambiguous radiometric disentanglement is to leverage the factthat while their texture and reflectance may differ, objects in the samescene are all lit by the same illumination. MultiGP exploits this consen-sus to produce samples of reflectance, texture, and illumination from asingle image of known shapes based on four key technical contributions: acascaded end-to-end architecture that combines image-space and angular-space disentanglement; Coordinated Scheduling for diffusion convergenceto a single consistent illumination estimate; Axial Attention applied tofacilitate “cross-talk” between objects of different reflectance; and a Tex-ture Extraction ControlNet to preserve high-frequency texture detailswhile ensuring decoupling from estimated lighting. Experimental resultsdemonstrate that MultiGP effectively leverages the complementary spatialand frequency characteristics of multiple object appearances to recoverindividual texture and reflectance as well as the common illumination.