MSVS-VAE: Multi-Scale Anchored VecSet for High-Fidelity 3D Reconstruction

High-fidelity 3D generative modeling increasingly relies onthe latent diffusion paradigm, where the reconstruction quality of theunderlying 3D VAE becomes a primary bottleneck. Existing approacheslargely follow two paradigms: sparse voxel-based representations achievestrong reconstruction quality but incur significant memory and compu-tational overhead, while set-based representations are compact and con-tinuous yet typically lag in fidelity due to latent sparsity and excessiveglobal smoothness. We propose MSVS-VAE, a hierarchical set-based VAEthat closes this fidelity gap without sacrificing compactness. Our keyidea is to progressively densify anchored VecSet latents via hierarchicalpoint-shuffle upsampling, increasing spatial capacity for fine-grained ge-ometry modeling. To efficiently decode from the densified hierarchy, wereplace global cross-attention with AVS-Conv, a geometry-aware localaggregation operator operating within local neighborhoods rather thanthe exhaustive latent set. We further introduce multi-scale query decod-ing to fuse coarse-to-fine latent features, where coarse scales providestable global context, and fine scales refine localized geometry, reducingartifacts from overly local receptive fields. Extensive experiments on Ob-javerse, ABO, and in-the-wild benchmarks demonstrate that MSVS-VAEconsistently outperforms prior set-based and voxel-based VAEs, deliver-ing ∼10× faster decoding than prior set-based methods and ∼10× highercompactness than voxel-based baselines.