Structural Assessment for Understanding and Guiding Dataset Distillation in Discrete Token Space

Dataset distillation (DD) has proven to reduce training costwhile preserving accuracy. While promising, the factors that make onedistilled dataset more effective than another remain poorly understood.In this work, we investigate this question through the lens of discretevisual tokenizers. Whereas many prior DD efforts emphasize matchingglobal data distributions, we suggest that the effectiveness depends onwhich semantic concepts are captured and how they are composed. Dis-crete visual tokenizers provide a finite vocabulary that enables directstatistical analysis of such compositional structure. Through quantita-tive analysis of token-level statistics, we introduce the structural scoreto measure the adequacy of token compositions. We observe that dis-tilled datasets with balanced token composition yield higher validationperformance. On the other hand, divergence from the original data doesnot necessarily harm performance. We further show that samples withhigh structural scores in the discrete token space can effectively guidediffusion-based DD. Our findings highlight the importance of token com-position in dataset effectiveness, offering a principled complement to dis-tributional similarity considerations in DD.