Hi! @yiyixuxu @asomoza @sayakpaul
Would you review this PR?

Could you also accompany the PR with a short description and perhaps some results?

@sayakpaul

GLIGEN uses a fuser called gated-self-attention between unet's self-attention and cross-attention to reflect extra conditions other than text prompts to the generated image. And the network that obtains the embedding that becomes the input of the fuser is different for each type of extra condition given.

InstanceDiffusion belongs to the layout2image model and uses object-specific positions and phrases as extra conditions. However, unlike GLIGEN, it proposed a network called Unifusion that can encompass all types of object-specific position information, no matter they are box, point, scribble, or mask.

In addition, it proposed a ScaleU block that scales the embedding in the up-block of unet based on Fourier transform so better handle small objects.

Furthermore, to prevent overlapping objects from being blurred during inference, it proposed multi-instance sampling, whereby denoising is performed independently for each object at the beginning, aggregation is performed at a certain point, and the remaining denoising steps are performed afterwards.

I validated the implementation following the config and example prompt from the official repository. The example usage with diffusers and the result can be checked in huggingface model card.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

any updates on this?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.