EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removal

Object removal must prevent the masked target from reap-pearing and reconstruct the occluded background with structural andcontextual fidelity, rather than merely filling a hole plausibly. Recentdataset-free approaches manipulate the diffusion model’s internal self-attention to prevent it from referencing the masked region, yet they failin two critical ways: (i) they treat the masked region as the sole fore-ground, misinterpreting non-target objects as background and regener-ating them, and (ii) they apply uniform attention constraints withoutdistinguishing diverse background subtypes, leading to textural blurringand structural misalignment. Both failures stem from the absence of ex-plicit background-aware reasoning. We propose EraseLoRA, a dataset-free framework that replaces attention surgery with background-awarereasoning and test-time adaptation. The first stage, Background-awareForeground Exclusion (BFE), leverages a multimodal large-language modelto separate target foreground, non-target foregrounds, and clean back-ground from a single image-mask pair. The second stage, Background-aware Reconstruction with Subtype Aggregation (BRSA), performs test-time optimization that treats inferred background subtypes as comple-mentary pieces, enforcing their consistent integration through recon-struction and alignment objectives without explicit attention interven-tion. As a model-agnostic plug-in applicable to diverse diffusion back-bones, EraseLoRA reconstructs backgrounds at least 23% more faithfulto the original scene than previous dataset-free methods while nearlyhalving unwanted foreground re-generation, and surpasses all dataset-driven approaches in both aspects despite requiring no training data.Code is available at https://shjo-april.github.io/EraseLoRA.