Do Flat Minima Improve Sparse Novel View Synthesis?

Despite the success of recent novel view synthesis methods,they tend to struggle in sparse-view settings. This poor generalizationto unseen viewpoints is an inherent challenge when training with lim-ited data. To address this, we investigate the relationship between losssharpness and generalization in novel view synthesis—an underexploreddirection. Interestingly, while pursuing flatter minima is widely knownto improve generalization in deep learning, reducing loss sharpness is notalways beneficial in novel view synthesis. We demonstrate that this dif-ference arises because high-detail regions inherently require a sharp losslandscape for accurate reconstruction, whereas low-detail regions benefitfrom a flat loss landscape for improving generalization. Based on this in-sight, we introduce structure-aware sharpness, defined within structure-adaptive neighborhoods, and propose to adaptively adjust the sharpnessregularization weight according to the local image structure. This strat-egy encourages flatter minima for generalization while preserving the losssharpness necessary to reconstruct fine details. Across various datasetsand configurations, our strategy consistently improves a wide range ofbaselines. Code is available at https://bbangsik13.github.io/FASR.