Thanks to visit codestin.com
Credit goes to github.com

Skip to content

Support custom_pipeline argument for Diffuser T2V Community Pipelines #11183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ParagEkbote opened this issue Apr 1, 2025 · 2 comments
Open
Labels
stale Issues that haven't received updates

Comments

@ParagEkbote
Copy link
Contributor

What API design would you like to have changed or added to the library? Why?

In the diffusers community pipelines, the T2V pipelines such as Spatiotemporal Skip Guidance (STG) or CogVideoX DDIM Inversion Pipeline do not yet support the custom_pipeline argument since it is only supported for T2I pipelines like:

pipeline = DiffusionPipeline.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    custom_pipeline="pipeline_flux_with_cfg"
)

Due to this, I cannot contribute colab notebooks for the pipelines. This is further referenced here.

What use case would this enable or better enable? Can you give us a code example?

Ideally, I would like to load the CogVideoX DDIM Inversion Pipeline like:

import torch

pipeline = CogVideoXPipelineForDDIMInversion.from_pretrained(
    "THUDM/CogVideoX1.5-5B",
    torch_dtype=torch.bfloat16,
    custom_pipeline="cogvideox_ddim_inversion"
).to("cuda")

output = pipeline_for_inversion(
    prompt="prompt that describes the edited video",
    video_path="path/to/input.mp4",
    guidance_scale=6.0,
    num_inference_steps=50,
    skip_frames_start=0,
    skip_frames_end=0,
    frame_sample_step=None,
    max_num_frames=81,
    width=720,
    height=480,
    seed=42,
)
pipeline.export_latents_to_video(output.inverse_latents[-1], "path/to/inverse_video.mp4", fps=8)
pipeline.export_latents_to_video(output.recon_latents[-1], "path/to/recon_video.mp4", fps=8)

Could you please let me know if this feasible or not?

@ParagEkbote ParagEkbote changed the title Support custom_pipeline argument for Diffuser T2V Pipelines Support custom_pipeline argument for Diffuser T2V Community Pipelines Apr 1, 2025
@tolgacangoz
Copy link
Contributor

Hi @ParagEkbote! Could you try DiffusionPipeline.from_pretrained(..., custom_pipeline="cogvideox_ddim_inversion") with diffusers installed from the source?

Copy link
Contributor

github-actions bot commented May 1, 2025

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions bot added the stale Issues that haven't received updates label May 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale Issues that haven't received updates
Projects
None yet
Development

No branches or pull requests

2 participants