LLM/diffusers
1
0
Fork 0
mirror of https://github.com/huggingface/diffusers.git synced 2026-02-14 23:05:20 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,528 Commits 743 Branches 94 Tags
9f011c088bc9443ebdcc048b384bc1a58bc8dffa
Commit Graph

848 Commits

Author SHA1 Message Date
Jinzhe Pan
9f011c088b [docs] Add xDiT in section optimization (#9365) 
* docs: add xDiT to optimization methods

* fix: picture layout problem

* docs: add more introduction about xdit & apply suggestions

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:16 +05:30
Igor Filippov
e8d15b22d1 [Pipeline] animatediff + vid2vid + controlnet (#9337) 
* add animatediff + vid2vide + controlnet

* post tests fixes

* PR discussion fixes

* update docs

* change input video to links on HF + update an example

* make quality fix

* fix ip adapter test

* fix ip adapter test input

* update ip adapter test
 2024-12-23 13:02:16 +05:30
YiYi Xu
04809b0ccb refactor get_timesteps for SDXL img2img + add set_begin_index (#9375) 
* refator + add begin_index

* add kolors img2img to doc
 2024-12-23 13:02:16 +05:30
Vishnu V Jaddipal
2a72888dc0 Add Flux inpainting and Flux Img2Img (#9135) 
---------

Co-authored-by: yiyixuxu <yixu310@gmail.com>
 2024-12-23 13:02:16 +05:30
Aryan
91168b5436 [core] Support VideoToVideo with CogVideoX (#9333) 
* add vid2vid pipeline for cogvideox

* make fix-copies

* update docs

* fake context parallel cache, vae encode tiling

* add test for cog vid2vid

* use video link from HF docs repo

* add copied from comments; correctly rename test class
 2024-12-23 13:02:16 +05:30
Aryan
ae69462e0e [docs] Add a note on torchao/quanto benchmarks for CogVideoX and memory-efficient inference (#9296) 
* add a note on torchao/quanto benchmarks and memory-efficient inference

* apply suggestions from review

* update

* Update docs/source/en/api/pipelines/cogvideox.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/api/pipelines/cogvideox.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* add note on enable sequential cpu offload

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:15 +05:30
Steven Liu
4fc5e29ab0 [docs] Add pipelines to table (#9282) 
update pipelines
 2024-12-23 13:02:15 +05:30
王奇勋
cfd12e2ece [Flux] Support Union ControlNet (#9175) 
* refactor
---------

Co-authored-by: haofanwang <haofanwang.ai@gmail.com>
 2024-12-23 13:02:15 +05:30
zR
977afbe83b Cogvideox-5B Model adapter change (#9203) 
* draft of embedding

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
Elias Rad
de303b9a15 Docs fix spelling issues (#9219) 
* fix PHILOSOPHY.md

* fix CONTRIBUTING.md

* fix tutorial_overview.md

* fix stable_diffusion.md

* Update tutorial_overview.md
 2024-12-23 13:02:15 +05:30
satani99
8a17331c29 Add StableDiffusionXLControlNetPAGImg2ImgPipeline (#8990) 
* Added pad controlnet sdxl img2img pipeline

---------

Co-authored-by: YiYi Xu <yixu310@gmail.com>
 2024-12-23 13:02:15 +05:30
Jiwook Han
523fda4722 Reflect few contributions on contribution.md that were not reflected on #8294 (#8938) 
* incorrect_number_fix

* add_TOC

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* fix: manual edits

* fix: manual edtis

* fix: manual edits

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* Update docs/source/ko/conceptual/contribution.md

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>

* fix: manual edits

---------

Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com>
 2024-12-23 13:02:15 +05:30
Tolga Cangöz
259b12a077 [Docs] Fix CPU offloading usage (#9207) 
* chore: Fix cpu offloading usage

* Trim trailing white space

* docs: update Kolors model link in kolors.md
 2024-12-23 13:02:15 +05:30
Aryan
0f74c69416 [refactor] CogVideoX followups + tiled decoding support (#9150) 
* refactor context parallel cache; update torch compile time benchmark

* add tiling support

* make style

* remove num_frames % 8 == 0 requirement

* update default num_frames to original value

* add explanations + refactor

* update torch compile example

* update docs

* update

* clean up if-statements

* address review comments

* add test for vae tiling

* update docs

* update docs

* update docstrings

* add modeling test for cogvideox transformer

* make style
 2024-12-23 13:02:15 +05:30
林金鹏
095393a5b8 Support SD3 controlnet inpainting (#9099) 
* add controlnet inpainting pipeline

* [SD3] add controlnet inpaint example

* update example and fix code style

* fix code style with ruff

* Update controlnet_sd3.md : add control inpaint pipeline

* Update docs/source/en/api/pipelines/controlnet_sd3.md

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update docs/source/en/api/pipelines/controlnet_sd3.md

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update docs/source/en/api/pipelines/controlnet_sd3.md

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet_inpainting.py

Co-authored-by: Aryan <contact.aryanvs@gmail.com>

* Update __init__.py : add sd3 control pipelines

* Update pipeline : add new param doc & check input reference.

* fix typo

* make style & make quality

* add unittest for sd3 controlnet inpaint

---------

Co-authored-by: 鹏徙 <linjinpeng.ljp@alibaba-inc.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul
0c78d2af0b Update distributed_inference.md to include a fuller example on distributed inference (#9152) 
* Update distributed_inference.md

* Update docs/source/en/training/distributed_inference.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:15 +05:30
Steven Liu
052edeba21 [docs] Resolve internal links to PEFT (#9144) 
* resolve peft links

* fuse_lora
 2024-12-23 13:02:15 +05:30
David Steinberg
d8d8e86924 Fix a dead link (#9116) 
Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
Steven Liu
c690fc2635 [docs] Organize model toctree (#9118) 
* toctree

* fix
 2024-12-23 13:02:15 +05:30
zR
dbf5d348e6 Add CogVideoX text-to-video generation model (#9082) 
* add CogVideoX

---------

Co-authored-by: Aryan <aryan@huggingface.co>
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:15 +05:30
latentCall145
f771be1d7b Flux fp16 inference fix (#9097) 
* clipping for fp16

* fix typo

* added fp16 inference to docs

* fix docs typo

* include link for fp16 investigation

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Álvaro Somoza
3510d0ef5e [Kolors] Add PAG (#8934) 
* txt2img pag added

* autopipe added, fixed case

* style

* apply suggestions

* added fast tests, added todo tests

* revert dummy objects for kolors

* fix pag dummies

* fix test imports

* update pag tests

* add kolor pag to docs

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Dhruv Nair
47874e837d [Single File] Add single file support for Flux Transformer (#9083) 
* update

* update

* update

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Ahn Donghoon (안동훈 / suno)
f25823781d add PAG support for Stable Diffusion 3 (#8861) 
add pag sd3


---------

Co-authored-by: HyoungwonCho <jhw9811@korea.ac.kr>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: crepejung00 <jaewoojung00@naver.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Aryan <contact.aryanvs@gmail.com>
Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
Dhruv Nair
4a91ee80c2 [Docs] Add community projects section to docs (#9013) 
* update

* update

* update
 2024-12-23 13:02:15 +05:30
Dhruv Nair
faa0826328 update 2024-12-23 13:02:15 +05:30
Tolga Cangöz
51f45da25f Update CLIPFeatureExtractor to CLIPImageProcessor and DPTFeatureExtractor to DPTImageProcessor (#9002) 
* fix: update `CLIPFeatureExtractor` to `CLIPImageProcessor` in codebase

* `make style && make quality`

* Update `DPTFeatureExtractor` to `DPTImageProcessor` in codebase

* `make style`

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:15 +05:30
omahs
49544cc1fb Fix typos (#9077) 
* fix typo
 2024-12-23 13:02:15 +05:30
Sayak Paul
8881fc9872 [Docs] add stable cascade unet doc. (#9066) 
* add stable cascade unet doc.

* fix path
 2024-12-23 13:02:15 +05:30
Aryan
9dbffc8c60 PAG variant for HunyuanDiT, PAG refactor (#8936) 
* copy hunyuandit pipeline

* pag variant of hunyuan dit

* add tests

* update docs

* make style

* make fix-copies

* Update src/diffusers/pipelines/pag/pag_utils.py

* remove incorrect copied from

* remove pag hunyuan attn procs to resolve conflicts

* add pag attn procs again

* new implementation for pag_utils

* revert pag changes

* add pag refactor back; update pixart sigma

* update pixart pag tests

* apply suggestions from review

Co-Authored-By: yixu310@gmail.com

* make style

* update docs, fix tests

* fix tests

* fix test_components_function since list not accepted as valid __init__ param

* apply patch to fix broken tests

Co-Authored-By: Sayak Paul <spsayakpaul@gmail.com>

* make style

* fix hunyuan tests

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:15 +05:30
Sayak Paul
0db81141b9 [Flux] minor documentation fixes for flux. (#9048) 
* minor documentation fixes for flux.

* clipskip

* add gist
 2024-12-23 13:02:15 +05:30
Philip Rideout
9c02c40a13 Fix grammar mistake. (#9072) 2024-12-23 13:02:15 +05:30
Tolga Cangöz
c6ac793955 Errata: Fix typos & \s+$ (#9008) 
* Fix typos

* chore: Fix typos

* chore: Update README.md for promptdiffusion example

* Trim trailing white spaces

* Fix a typo

* update number

* chore: update number

* Trim trailing white space

* Update README.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update README.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:14 +05:30
Sayak Paul
c8a236ba5c [Core] Add PAG support for PixArtSigma (#8921) 
* feat: add pixart sigma pag.

* inits.

* fixes

* fix

* remove print.

* copy paste methods to the pixart pag mixin

* fix-copies

* add documentation.

* add tests.

* remove correction file.

* remove pag_applied_layers

* empty
 2024-12-23 13:02:14 +05:30
Sayak Paul
7739beb740 Flux pipeline (#9043) 
add flux!

Signed-off-by: Adrien <adrien@huggingface.co>
Co-authored-by: Adrien <adrien.69740@gmail.com>
Co-authored-by: Anatoly Belikov <abelikov@singularitynet.io>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
 2024-12-23 13:02:14 +05:30
Aryan
e28e5373f9 PAG variant for AnimateDiff (#8789) 
* add animatediff pag pipeline

* remove unnecessary print

* make fix-copies

* fix ip-adapter bug

* update docs

* add fast tests and fix bugs

* update

* update

* address review comments

* update ip adapter single test expected slice

* implement test_from_pipe_consistent_config; fix expected slice values

* LoraLoaderMixin->StableDiffusionLoraLoaderMixin; add latest freeinit test
 2024-12-23 13:02:14 +05:30
Aryan
cf513e4205 [core] Move community AnimateDiff ControlNet to core (#8972) 
* add animatediff controlnet to core

* make style; remove unused method

* fix copied from comment

* add tests

* changes to make tests work

* add utility function to load videos

* update docs

* update pipeline example

* make style

* update docs with example

* address review comments

* add latest freeinit test from #8969

* LoraLoaderMixin -> StableDiffusionLoraLoaderMixin

* fix docs

* Update src/diffusers/utils/loading_utils.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* fix: variable out of scope

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:14 +05:30
Yoach Lacombe
030a134311 Stable Audio integration (#8716) 
* WIP modeling code and pipeline

* add custom attention processor + custom activation + add to init

* correct ProjectionModel forward

* add stable audio to __initèè

* add autoencoder and update pipeline and modeling code

* add half Rope

* add partial rotary v2

* add temporary modfis to scheduler

* add EDM DPM Solver

* remove TODOs

* clean GLU

* remove att.group_norm to attn processor

* revert back src/diffusers/schedulers/scheduling_dpmsolver_multistep.py

* refactor GLU -> SwiGLU

* remove redundant args

* add channel multiples in autoencoder docstrings

* changes in docsrtings and copyright headers

* clean pipeline

* further cleaning

* remove peft and lora and fromoriginalmodel

* Delete src/diffusers/pipelines/stable_audio/diffusers.code-workspace

* make style

* dummy models

* fix copied from

* add fast oobleck tests

* add brownian tree

* oobleck autoencoder slow tests

* remove TODO

* fast stable audio pipeline tests

* add slow tests

* make style

* add first version of docs

* wrap is_torchsde_available to the scheduler

* fix slow test

* test with input waveform

* add input waveform

* remove some todos

* create stableaudio gaussian projection + make style

* add pipeline to toctree

* fix copied from

* make quality

* refactor timestep_features->time_proj

* refactor joint_attention_kwargs->cross_attention_kwargs

* remove forward_chunk

* move StableAudioDitModel to transformers folder

* correct convert + remove partial rotary embed

* apply suggestions from yiyixuxu -> removing attn.kv_heads

* remove temb

* remove cross_attention_kwargs

* further removal of cross_attention_kwargs

* remove text encoder autocast to fp16

* continue removing autocast

* make style

* refactor how text and audio are embedded

* add paper

* update example code

* make style

* unify projection model forward + fix device placement

* make style

* remove fuse qkv

* apply suggestions from review

* Update src/diffusers/pipelines/stable_audio/pipeline_stable_audio.py

Co-authored-by: YiYi Xu <yixu310@gmail.com>

* make style

* smaller models in fast tests

* pass sequential offloading fast tests

* add docs for vae and autoencoder

* make style and update example

* remove useless import

* add cosine scheduler

* dummy classes

* cosine scheduler docs

* better description of scheduler

---------

Co-authored-by: YiYi Xu <yixu310@gmail.com>
 2024-12-23 13:02:14 +05:30
Sayak Paul
3566f4b18a [Docs] credit where it's due for Lumina and Latte. (#9000) 
credit where it's due for Lumina and Latte.
 2024-12-23 13:02:14 +05:30
Álvaro Somoza
edddf3d417 [Kolors] Add IP Adapter (#8901) 
* initial draft

* apply suggestions

* fix failing test

* added ipa to img2img

* add docs

* apply suggestions
 2024-12-23 13:02:14 +05:30
Aryan
b7ddd2bb99 [core] AnimateDiff SparseCtrl (#8897) 
* initial sparse control model draft

* remove unnecessary implementation

* copy animatediff pipeline

* remove deprecated callbacks

* update

* update pipeline implementation progress

* make style

* make fix-copies

* update progress

* add partially working pipeline

* remove debug prints

* add model docs

* dummy objects

* improve motion lora conversion script

* fix bugs

* update docstrings

* remove unnecessary model params; docs

* address review comment

* add copied from to zero_module

* copy animatediff test

* add fast tests

* update docs

* update

* update pipeline docs

* fix expected slice values

* fix license

* remove get_down_block usage

* remove temporal_double_self_attention from get_down_block

* update

* update docs with org and documentation images

* make from_unet work in sparsecontrolnetmodel

* add latest freeinit test from #8969

* make fix-copies

* LoraLoaderMixin -> StableDiffsuionLoraLoaderMixin
 2024-12-23 13:02:14 +05:30
RandomGamingDev
13ef7e1b98 Added accelerator based gradient accumulation for basic_example (#8966) 
added accelerator based gradient accumulation for basic_example

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:14 +05:30
Sayak Paul
6d11129c5a [Chore] add LoraLoaderMixin to the inits (#8981) 
* introduce  to promote reusability.

* up

* add more tests

* up

* remove comments.

* fix fuse_nan test

* clarify the scope of fuse_lora and unfuse_lora

* remove space

* rewrite fuse_lora a bit.

* feedback

* copy over load_lora_into_text_encoder.

* address dhruv's feedback.

* fix-copies

* fix issubclass.

* num_fused_loras

* fix

* fix

* remove mapping

* up

* fix

* style

* fix-copies

* change to SD3TransformerLoRALoadersMixin

* Apply suggestions from code review

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* up

* handle wuerstchen

* up

* move lora to lora_pipeline.py

* up

* fix-copies

* fix documentation.

* comment set_adapters().

* fix-copies

* fix set_adapters() at the model level.

* fix?

* fix

* loraloadermixin.

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:14 +05:30
YiYi Xu
a754d9071e Revert "[LoRA] introduce LoraBaseMixin to promote reusability." (#8976) 
Revert "[LoRA] introduce LoraBaseMixin to promote reusability. (#8774)"

This reverts commit 527430d0a4.
 2024-12-23 13:02:14 +05:30
Sayak Paul
82b37a4cc3 [LoRA] introduce LoraBaseMixin to promote reusability. (#8774) 
* introduce  to promote reusability.

* up

* add more tests

* up

* remove comments.

* fix fuse_nan test

* clarify the scope of fuse_lora and unfuse_lora

* remove space

* rewrite fuse_lora a bit.

* feedback

* copy over load_lora_into_text_encoder.

* address dhruv's feedback.

* fix-copies

* fix issubclass.

* num_fused_loras

* fix

* fix

* remove mapping

* up

* fix

* style

* fix-copies

* change to SD3TransformerLoRALoadersMixin

* Apply suggestions from code review

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* up

* handle wuerstchen

* up

* move lora to lora_pipeline.py

* up

* fix-copies

* fix documentation.

* comment set_adapters().

* fix-copies

* fix set_adapters() at the model level.

* fix?

* fix

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:14 +05:30
RandomGamingDev
7f10257852 Added Code for Gradient Accumulation to work for basic_training (#8961) 
added line allowing gradient accumulation to work for basic_training example
 2024-12-23 13:02:14 +05:30
Jiwook Han
eed4531f9c Reflect few contributions on ethical_guidelines.md that were not reflected on #8294 (#8914) 
fix_ethical_guidelines.md
 2024-12-23 13:02:14 +05:30
Sayak Paul
8e76f5b6b5 [Docs] small fixes to pag guide. (#8920) 
small fixes to pag guide.
 2024-12-23 13:02:14 +05:30
Seongsu Park
0e59db02d9 🌐 [i18n-KO] Translated docs to Korean (added 7 docs and etc) (#8804) 
* remove unused docs

* add ko-18n docs

* docs typo, edit etc

* reorder list, add `in translation` in toctree

* fix minor translation

* fix docs minor tone, etc
 2024-12-23 13:02:14 +05:30
Aryan
0f2c512fb6 [docs] pipeline docs for latte (#8844) 
* add pipeline docs for latte

* add inference time to latte docs

* apply review suggestions
 2024-12-23 13:02:14 +05:30