Self-Evolving Just-In-Time Memory for Proactive Embodied Safety
Abstract
While Vision-Language Models (VLMs) have empowered em-bodied agents to execute complex household tasks, they struggle toproactively handle dynamically emerging hazards during closed-loop in-teractions. Existing safety approaches often rely on runtime guardrailsto block unsafe actions or induce excessive caution, which severely stallstask progress instead of actively resolving the underlying risks. To breakthis safety–progress trade-off, we introduce the Self-Evolving Just-In-Time Memory framework, which reframes embodied safety from progress-stalling guardrails to proactive hazard mitigation. The framework con-sists of a Risk-Sufficient Topological Belief Graph (RSG) for persistentsafety-relevant state tracking under partial observability, an Agency-Grounded Factual Memory for precise hazard anticipation, and an Expe-rience Memory that injects procedural Meta-Skills to guide executable,progress-preserving mitigation. Furthermore, we propose an automatedTest-Verify-Write loop, allowing agents to continually refine their mit-igation Meta-Skills from execution traces at test time. Experiments onIS-Bench demonstrate that our framework substantially boosts the Safe-Success rate across multiple VLM backbones (e.g., +30.3% on Qwen3-VL-8B), enabling agents to proactively mitigate hazards without stallingtask progress. Code is available at https://github.com/DyMessi/JIT-Memory.