OrthoTailor: Geometric Orthogonalization for Conflict-Free Unified Fashion Generation

Unified fashion generation integrates tasks like virtual try-onand garment reconstruction into a single model to reduce task-specificadaptation costs. However, naive parameter sharing across semanticallydistinct tasks induces negative transfer through severe inter-task gradientconflict. We propose OrthoTryOn, a unified framework mitigating thisinterference within a shared Low-Rank Adaptation (LoRA) module. ItsOrthogonal Subspace Projection (OSP) applies task-specific orthogonalrotations to bottleneck features, mapping them into decorrelated coordi-nate frames. To address residual semantic coupling at inference time, wefurther propose Fisher-guided Negative Guidance (FNG), a parameter-free strategy that utilizes diagonal Fisher information to quantify inter-task sensitivity overlap and explicitly repels generation trajectories fromthe most confusable task via Classifier-Free Guidance. Extensive exper-iments demonstrate that OrthoTryOn avoids the severe performancedegradation typical of naive unified training and even surpasses inde-pendently trained task-specific models, achieving state-of-the-art resultsacross multiple benchmarks while generalizing robustly across diversediffusion backbones. Code is available at https://github.com/NJU-PCALab/OrthoTryOn.