Keep Your Friends Close, and the Right Neighbours Closer: Disaster-Conditioned Kernel-Regularized Graph Attention for Building Damage Classification

Disaster damage is spatial: buildings rarely fail in isolation.Yet using spatial context for damage classification remains surprisinglyunderexplored, and many pipelines still rely primarily on per-buildingappearance cues even when the dominant uncertainty is spatially struc-tured. Complicating matters, the right neighbourhood is not the sameacross events. Floods, hurricanes, and wildfires can exhibit very differ-ent clustering behaviour, making spatial reasoning valuable but easy tomisuse—naive context aggregation can improve visual coherence whileoversmoothing boundaries or propagating structured errors. We studythis tension on xBD (the dataset used in the xView2 challenge) in a con-trolled post-localization, classification-only setup: each building is repre-sented by a pre/post combined (PPC) patch cropped from the providedpolygons, and spatial context is modelled with GPS-derived buildinggraphs. Our approach keeps local evidence “close” by preserving strongspatial relationships in disaster damage patterns, while bringing onlythe right neighbours “closer” through a disaster-type-conditioned graphmodel that injects a learnable multi-scale spatial kernel prior into atten-tion, allowing the effective neighbourhood scale to adapt across disastertypes rather than being learned as a single global smoothing rule. Todiscourage coherence-by-smoothing, we add a residual de-correlation lossthat penalizes positive Moran’s I in prediction residuals. We evaluate themethod with (i) an xView2 holdout external-reference comparison un-der fixed building instances and (ii) zero-shot transfer evaluations thatstress-test generalization under event and dataset shift: leave-one-event-out (LOEO) on xBD and cross-dataset transfer from xBD to Ida-BD.The model improves macro-F1 and substantially reduces residual spa-tial autocorrelation under zero-shot event shift, indicating better use ofspatial context rather than naive smoothing and enabling more reliabletransfer to unseen events within known disaster types.