DiffPro: Joint Timestep and Layer-Wise Precision Optimization for Efficient Diffusion Inference

Diffusion models produce high-quality images, but their de-ployment remains expensive because generation requires many denoisingsteps through large transformer or U-Net backbones. Existing methodsimprove efficiency through post-training quantization or timestep reduc-tion, but they optimize these two axes separately. Quantization methodsusually assume a fixed precision, timestep reduction methods assume afixed schedule, and both often rely on proxy costs such as BitOps insteadof real hardware measurements. We introduce DiffPro, a post-trainingframework that jointly optimizes per-layer quantization precision and thedenoising schedule under a unified budget defined by measured memoryand latency on real hardware. DiffPro combines three key ideas: (i) acomposite sensitivity metric that combines PCA dimensionality withdiagonal Hessian curvature for bit allocation, (ii) Dynamic ActivationQuantization (DAQ), which adapts activation scales across timesteps tohandle temporal activation drift, and (iii) a drift-guided timestep selectorthat removes redundant steps while preserving the late refinement phase.On DiT-XL/2 (ImageNet 256 × 256), DiffPro achieves 50% step reduction,7.9× compression, and 2.7× speedup at FID 5.89 (∆FID 3.62), improvingover uniform thinning by 2.5 FID at matched precision. On SDXL U-Net(COCO 2014 1024 × 1024), DiffPro achieves 5.6× compression with FID26.50 (∆FID 2.0) and 45% energy reduction, all without retraining.