mmIR: Frequency-Space Inverse Rendering for 3D Millimeter-Wave Radar ADC Synthesis

High-resolution 3D radar data is scarce. Commodity mmWavesensors use small antenna arrays that limit angular resolution to several de-grees, and existing datasets provide only 2D range–azimuth maps or sparsepoint clouds rather than raw analog-to-digital converter (ADC) signals.Hardware scaling is expensive, synthetic-aperture scanning is impracticalat fleet scale, and learned synthesis methods are bottlenecked by the verydata shortage they aim to address. We present mmIR, an open-sourcedifferentiable frequency-modulated continuous-wave (FMCW) radar in-verse renderer that fits a physics-based forward model to real capturesand re-renders from dense virtual apertures to synthesize high-resolution3D radar data. Because radar resolution is too coarse to recover geom-etry directly, mmIR performs LiDAR-assisted inverse rendering: usingLiDAR-derived meshes as a geometric scaffold, mmIR optimizes per-vertex International Telecommunication Union (ITU) physics materials,vertex normals, and antenna beam patterns through end-to-end auto-matic differentiation of a phase-coherent multiple-input multiple-output(MIMO) forward model with multi-bounce propagation, polarization,and free-space diffraction. On seven outdoor and six indoor ColoRadarscenes, mmIR achieves 0.914 mean Pearson correlation on range–azimuthmaps versus 0.307 for Sionna-RT. Scenes trained on a cascaded imag-ing radar transfer to a co-located single-chip radar without re-training(0.554 correlation), and dense virtual arrays (100×100 elements) pro-duce single-frame 3D occupancy validated against LiDAR. Project page:https://mmwave-inverse-rendering.github.io/.