[modular] Add Ultimate SD Upscale pipeline for SDXL#2
Draft
akshan-main wants to merge 20 commits into
Draft
Conversation
Add a Modular Diffusers pipeline for tiled img2img upscaling using Stable Diffusion XL, targeting feature request huggingface#9740 and the Modular Diffusers contribution call huggingface#13295.
|
HI @akshan-main, not sure if it will help you but just so you don't waste time, we won't be adding this to the core of diffusers. This is ideal to have as as custom blocks you can just publish in the hub without any reviews, the main reason for this feature in diffusers is for something like what you're doing here. |
Owner
Author
|
Thanks @asomoza that makes sense. I'll publish this as custom Hub blocks instead. Appreciate the guidance. |
Features: - Chess (checkerboard) tile traversal: white squares first, then black - Seam-fix band re-denoise with linearly-feathered mask blending - Gradient overlap blending as alternative to hard core paste - Configurable seam_fix_width, seam_fix_padding, seam_fix_mask_blur, seam_fix_strength, blend_mode, gradient_blend_overlap
The prepare sub-block reads timesteps and latent_timestep from block_state but never declared them as inputs, so get_block_state never pulled them from the PipelineState.
Plain tiled img2img produces per-tile drift because each tile's UNet pass generates features independently. ControlNet Tile constrains each tile to stay faithful to the input structure, which is the key ingredient for quality tiled upscaling.
This reverts commit 06780ea.
Tiled UNet processing produces visible seam artifacts regardless of blending, padding, or ControlNet conditioning. The fix is to process the full image in a single UNet pass and use VAE tiling for memory. Changes: - Enable VAE tiling automatically in the tile loop - Change default tile_size from 512 to 2048 so single-pass is default for typical upscale sizes (512→1024, 256→1024) - Tiled mode still available by setting tile_size=512 for very large images where single UNet pass would OOM
…tadata, scheduler selection
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a Modular Diffusers pipeline for Ultimate SD Upscale-style tiled img2img upscaling using Stable Diffusion XL, as a contribution to the Modular Diffusers contribution call (huggingface#13295).
Built from the standard SDXL img2img modular workflow, with per-tile latent preparation / conditioning / denoising moved into an outer tile loop (
LoopSequentialPipelineBlocks).Pass 1 uses padded crop context for denoising and non-overlapping core paste for reconstruction; seam-fixing and alternative traversal/blending strategies are planned as follow-up work.
This first PR focuses on a minimal mergeable implementation: SDXL only, Lanczos upscale, linear traversal, strict tile validation, tiled micro-conditioning, and tests/examples. Chess traversal and seam-fixing modes are intentionally left for follow-up PRs.
Architecture
Reused SDXL blocks (8 total, via public
__call__)TextEncoderStep,InputStep,Img2ImgSetTimestepsStep,VaeEncoderStep,Img2ImgPrepareLatentsStep,Img2ImgPrepareAdditionalConditioningStep,DenoiseStep,DecodeStepBefore submitting
Who can review?
@sayakpaul @yiyixuxu @asomoza