Molmo-Point: Better Pointing for VLMs with Grounding Tokens

Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coor-dinates as part of their text output, which requires learning a complicatedcoordinate system and results in a high token count. Instead, we pro-pose a more intuitive pointing mechanism that directly selects the visualtokens that contain the target concept. Our model generates a specialpointing token that cross-attends to the input image or video tokens andselects the appropriate one. To make this model more fine-grained, we fol-low these pointing tokens with an additional special token that selects afine-grained subpatch within the initially selected region, and then a thirdtoken that specifies a location within that subpatch. We further showthat performance improves by generating points sequentially in a consis-tent order, encoding the relative position of the previously selected point,and including a special no-more-points class when selecting visual to-kens. Using this method, we set a new state-of-the-art on image pointing(70.7% on PointBench), set a new state-of-the-art for fully open modelson GUI pointing (61.1% on ScreenSpotPro), substantially improve VLMvideo tracking (62.5 on J &F vs 56.7 for Molmo2 on Molmo2Track), andimprove video pointing (59.1% human preference win rate vs. Molmo2).We additionally show that our method significantly improves learningefficiency and discuss the qualitative differences that emerge with thisdesign change. Our model weights, new datasets, and source code areavailable at https://allenai.org/blog/molmopoint.