SPEAR: A Simulator for Photorealistic Embodied AI Research

Interactive simulators have become powerful tools for train-ing embodied agents and generating synthetic visual data, but existingphotorealistic simulators suffer from limited generality, programmabil-ity, and rendering speed. We address these limitations by introducingSPEAR: A Simulator for Photorealistic Embodied AI Research. At itscore, SPEAR is a Python library that can connect to, and program-matically control, any Unreal Engine (UE) application via a modularplugin architecture. SPEAR exposes over 14K unique UE functions toPython, representing an order-of-magnitude increase in programmablefunctionality over existing UE-based simulators. Additionally, a singleSPEAR instance can render 1920×1080 photorealistic beauty images di-rectly into a user’s NumPy array at 73 frames per second – an order ofmagnitude faster than existing UE plugins – while also providing groundtruth image modalities that are not available in any existing UE-basedsimulator (e.g., a non-diffuse intrinsic image decomposition, material IDs,and physically based shading parameters). Finally, SPEAR introduces anexpressive high-level programming model that enables users to specifycomplex graphs of UE work with arbitrary data dependencies amongwork items, and to execute these graphs deterministically within a sin-gle UE frame. We demonstrate the utility of SPEAR through a diversecollection of example applications: controlling multiple embodied agentswith distinct action spaces (e.g., humans, cars, and robots) across sev-eral in-the-wild UE projects; rendering photorealistic city-scale environ-ments; manipulating UE’s procedural content generation systems; ren-dering synchronized multi-view images of detailed human faces; coordi-nating an interactive co-simulation with the MuJoCo physics simulator;and editing scenes with natural language via an AI coding assistant.