Seeing as Humans Do: Learning from Motion to Segment Anything Without Supervision

The Segment Anything Model (SAM) relies heavily on mas-sive manual annotations, creating a fundamental bottleneck for modelscaling. While unsupervised methods attempt to learn object conceptsfrom motion, they typically overfit to moving entities, lacking both multi-granularity understanding and the ability to generalize to static objects.To overcome this, we introduce Motion-Grounded Segment Anything(MoSA), a highly scalable unsupervised framework that learns a trans-ferable objectness prior from unlabeled videos. MoSA operates in threeprogressive stages: (1) automatically generating multi-granularity mo-tion pseudo-labels from large-scale video data; (2) training a PerceptualGrouping Model (PGM) via contrastive learning to internalize a gen-eralized, appearance-driven concept of objects; and (3) transferring thislearned prior into a prompt-guided architecture for segment-anything-style inference on images. Extensive zero-shot evaluations across sevenchallenging benchmarks (e.g., COCO and ADE20K) demonstrate thatMoSA significantly outperforms existing unsupervised methods. Notably,despite using zero manual annotations, MoSA achieves segmentation per-formance comparable to the fully supervised SAM. Our findings revealthat harnessing large-scale unlabeled motion is a feasible and highly scal-able alternative to annotation-driven segment-anything pipelines.