NoPA: Non-Parametric Online 3D Scene Graph Generation

Classic 3D scene graph generation approaches fail to workin real-time due to the heavy computational cost of environment map-ping and the need to generate intermediate point-cloud representations.To alleviate this issue, a recent work eschews point clouds in favor ofa lightweight Gaussian distribution for each object. This approximationdrastically speeds up inference and enables real-time 3D scene graph gen-eration. However, the representation has two key weaknesses. 1) Each ob-ject is approximated by a single 3D Gaussian, which causes a severe lossof 3D geometric detail. 2) The discrepancy between this approximationand the true object geometry exacerbates the inaccurate merging of ob-ject candidates during online inference. To address these issues, we pro-pose NoPA, which represents each object as a separate non-parametricdistribution. This formulation retains 3D geometric information whilepreserving real-time inference of the parametric Gaussian formulation. Tobuild upon our novel object representation, we propose a tailored merg-ing strategy to recover coherent object instances. Specifically, we leveragemaximum mean discrepancy on kernel density estimates to enable robustmerging of object candidates during online exploration while minimizingadded computational complexity. The key is to maintain a fixed particleset per object. Furthermore, to rectify the relation loss caused by misclas-sified objects, NoPA propagates relationships between objects with highaffinity. Experiments show that NoPA substantially outperforms currentmethods without sacrificing real-time inference speed.