Hi-DiT: Hybrid Latent-Pixel Diffusion Transformer for Image Generation

Current diffusion models face a fundamental tension betweenthe computational efficiency of latent-space generation and the fine-detailfidelity of pixel-space generation. Latent Diffusion Models (LDMs) relyon VAE compression, which can discard high-frequency visual cues andlimit pixel-level quality, while pixel-space diffusion often suffers from diffi-cult optimization when global structure and high-frequency details mustbe learned jointly. In this paper, we propose the Hybrid Latent-PixelDiffusion Transformer (Hi-DiT), a unified Diffusion Transformer thatbridges these two regimes within a single architecture. Motivated by thetemporal heterogeneity of denoising—early timesteps primarily establishcoarse global structure, whereas late timesteps increasingly emphasizehigh-frequency detail—we introduce a dual-stream design with temporalspecialization. Hi-DiT performs semantic planning in a compact latentpathway at early stages, and activates a pixel pathway at later stagesto synthesize high-frequency details using time-consistent noisy pixelembeddings. A Time-Gated Injection mechanism schedules the partic-ipation of pixel tokens only in the low-noise regime, and a lightweightsub-pixel prediction head enables efficient dense detail generation. Ex-tensive experiments demonstrate that Hi-DiT achieves state-of-the-artperformance, obtaining an FID of 1.06 on ImageNet 256 × 256, 1.26 onImageNet 512 × 512, and 5.15 on MS-COCO text-to-image generation.Code is available at: https://github.com/HiDream-ai/Hi-DiT.