RayRoPE: Projective Ray Positional Encoding for Multi-view Attention

We study positional encodings for multi-view transformersthat process tokens from a set of posed input images, and seek a mech-anism that encodes patches uniquely, allows SE(3)-invariant attentionwith multi-frequency similarity, and can adapt to the geometry of theunderlying 3D scene. We find that prior (absolute or relative) encod-ing schemes for multi-view attention do not meet these desiderata, andpresent RayRoPE to address this gap. RayRoPE represents patch po-sitions based on associated rays and computes query-frame projectivecoordinates to ensure SE(3) invariance. To adapt to scene geometry,RayRoPE predicts (without direct supervision) a per-token depth to ob-tain its position along the corresponding ray, while also modeling uncer-tainty and analytically computing the expected positional encoding. Wevalidate our method on the tasks of novel-view synthesis, stereo depthestimation, and feed-forward 3DGS reconstruction. While remaining ef-ficient, RayRoPE consistently improves over alternate position encodingschemes (e.g. 24% relative improvement on LPIPS in RE10K).