LLM/diffusers
1
0
Fork 0
mirror of https://github.com/huggingface/diffusers.git synced 2026-02-13 22:35:37 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,913 Commits 744 Branches 94 Tags
4c5082ca685d1dd0a24907890da170e258e5cd01
Commit Graph

4913 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Sam
0f599ee6b9 Update pipeline_flux_img2img.py (#9928) 
* Update pipeline_flux_img2img.py

Added FromSingleFileMixin to this pipeline loader like the other FLUX pipelines.

* Update pipeline_flux_img2img.py

typo

* modified:   src/diffusers/pipelines/flux/pipeline_flux_img2img.py
 2024-12-23 13:02:18 +05:30
Benjamin Paine
8731574e49 Fix Progress Bar Updates in SD 1.5 PAG Img2Img pipeline (#9925) 
fix progress bar updates in SD 1.5 PAG Img2Img pipeline
 2024-12-23 13:02:18 +05:30
Parag Ekbote
1022f6c2db Notebooks for Community Scripts Examples (#9905) 
* Add Notebooks on Community Scripts
 2024-12-23 13:02:18 +05:30
Eliseu Silva
aa71132aaf Feature IP Adapter Xformers Attention Processor (#9881) 
* Feature IP Adapter Xformers Attention Processor: this fix error loading incorrect attention processor when setting Xformers attn after load ip adapter scale, issues: #8863 #8872
 2024-12-23 13:02:18 +05:30
Sayak Paul
291db3e538 Revert "[Flux] reduce explicit device transfers and typecasting in flux." (#9896) 
Revert "[Flux] reduce explicit device transfers and typecasting in flux. (#9817)"

This reverts commit 5588725e8e.
 2024-12-23 13:02:18 +05:30
Sayak Paul
dbea93cb14 [Advanced LoRA v1.5] fix: gradient unscaling problem (#7018) 
fix: gradient unscaling problem

Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:18 +05:30
SahilCarterr
dd3e554b42 [FIX] Fix TypeError in DreamBooth SDXL when use_dora is False (#9879) 
* fix use_dora

* fix style and quality

* fix use_dora with peft version

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:18 +05:30
Michael Tkachuk
a27125d589 Enabling gradient checkpointing in eval() mode (#9878) 
* refactored
 2024-12-23 13:02:17 +05:30
SahilCarterr
55ec25ca08 [fix] Replaced shutil.copy with shutil.copyfile (#9885) 
fix shutil.copy
 2024-12-23 13:02:17 +05:30
Dhruv Nair
72e69ca811 Improve downloads of sharded variants (#9869) 
* update

* update

* update

* update

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul
cb7016cca3 [Flux] reduce explicit device transfers and typecasting in flux. (#9817) 
reduce explicit device transfers and typecasting in flux.
 2024-12-23 13:02:17 +05:30
Sayak Paul
e92bbf47c0 [Core] introduce controlnet module (#8768) 
* move vae flax module.

* controlnet module.

* prepare for PR.

* revert a commit

* gracefully deprecate controlnet deps.

* fix

* fix doc path

* fix-copies

* fix path

* style

* style

* conflicts

* fix

* fix-copies

* sparsectrl.

* updates

* fix

* updates

* updates

* updates

* fix

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
 2024-12-23 13:02:17 +05:30
SahilCarterr
221d6dbeba Updated _encode_prompt_with_clip and encode_prompt in train_dreamboth_sd3 (#9800) 
* updated encode prompt and clip encod prompt


---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Sookwan Han
f546404097 Add new community pipeline for 'Adaptive Mask Inpainting', introduced in [ECCV2024] ComA (#9228) 
* Add new community pipeline for 'Adaptive Mask Inpainting', introduced in [ECCV2024] Beyond the Contact: Discovering Comprehensive Affordance for 3D Objects from Pre-trained 2D Diffusion Models
 2024-12-23 13:02:17 +05:30
Vahid Askari
d1c42c626c Fix: Remove duplicated comma in distributed_inference.md (#9868) 
Fix: Remove duplicated comma

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
SahilCarterr
2861fc925b [Fix] Test of sd3 lora (#9843) 
* fix test

* fix test asser

* fix format

* Update test_lora_layers_sd3.py
 2024-12-23 13:02:17 +05:30
Aryan
939bb9e1d2 [core] Mochi T2V (#9769) 
* update

* udpate

* update transformer

* make style

* fix

* add conversion script

* update

* fix

* update

* fix

* update

* fixes

* make style

* update

* update

* update

* init

* update

* update

* add

* up

* up

* up

* update

* mochi transformer

* remove original implementation

* make style

* update inits

* update conversion script

* docs

* Update src/diffusers/pipelines/mochi/pipeline_mochi.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* Update src/diffusers/pipelines/mochi/pipeline_mochi.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* fix docs

* pipeline fixes

* make style

* invert sigmas in scheduler; fix pipeline

* fix pipeline num_frames

* flip proj and gate in swiglu

* make style

* fix

* make style

* fix tests

* latent mean and std fix

* update

* cherry-pick 1069d210e1

* remove additional sigma already handled by flow match scheduler

* fix

* remove hardcoded value

* replace conv1x1 with linear

* Update src/diffusers/pipelines/mochi/pipeline_mochi.py

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>

* framewise decoding and conv_cache

* make style

* Apply suggestions from code review

* mochi vae encoder changes

* rebase correctly

* Update scripts/convert_mochi_to_diffusers.py

* fix tests

* fixes

* make style

* update

* make style

* update

* add framewise and tiled encoding

* make style

* make original vae implementation behaviour the default; note: framewise encoding does not work

* remove framewise encoding implementation due to presence of attn layers

* fight test 1

* fight test 2

---------

Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: yiyixuxu <yixu310@gmail.com>
 2024-12-23 13:02:17 +05:30
RogerSinghChugh
a820e3a702 Refac training utils.py (#9815) 
* Refac training utils.py

* quality

---------

Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul
c1313968fc [feat] add load_lora_adapter() for compatible models (#9712) 
* add first draft.

* fix

* updates.

* updates.

* updates

* updates

* updates.

* fix-copies

* lora constants.

* add tests

* Apply suggestions from code review

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

* docstrings.

---------

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Dorsa Rohani
eb25f54a8b Add Diffusion Policy for Reinforcement Learning (#9824) 
* enable cpu ability

* model creation + comprehensive testing

* training + tests

* all tests working

* remove unneeded files + clarify docs

* update train tests

* update readme.md

* remove data from gitignore

* undo cpu enabled option

* Update README.md

* update readme

* code quality fixes

* diffusion policy example

* update readme

* add pretrained model weights + doc

* add comment

* add documentation

* add docstrings

* update comments

* update readme

* fix code quality

* Update examples/reinforcement_learning/README.md

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* Update examples/reinforcement_learning/diffusion_policy.py

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

* suggestions + safe globals for weights_only=True

* suggestions + safe weights loading

* fix code quality

* reformat file

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Leo Jiang
1a70054007 Reduce Memory Cost in Flux Training (#9829) 
* Improve NPU performance

* Improve NPU performance

* Improve NPU performance

* Improve NPU performance

* [bugfix] bugfix for npu free memory

* [bugfix] bugfix for npu free memory

* [bugfix] bugfix for npu free memory

* Reduce memory cost for flux training process

---------

Co-authored-by: 蒋硕 <jiangshuo9@h-partners.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Boseong Jeon
2a9727bdd1 Handling mixed precision for dreambooth flux lora training (#9565) 
Handling mixed precision and add unwarp

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
ScilenceForest
70bfeacc46 Update train_controlnet_flux.py,Fix size mismatch issue in validation (#9679) 
Update train_controlnet_flux.py

Fix the problem of inconsistency between size of image and size of validation_image which causes np.stack to report error.

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
SahilCarterr
ea68d7ccf4 Fixes EMAModel "from_pretrained" method (#9779) 
* fix from_pretrained and added test

* make style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Leo Jiang
ad754e6182 NPU Adaption for FLUX (#9751) 
* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

* NPU implementation for FLUX

---------

Co-authored-by: 蒋硕 <jiangshuo9@h-partners.com>
 2024-12-23 13:02:17 +05:30
Abhipsha Das
c538dea8fc [Model Card] standardize advanced diffusion training sd15 lora (#7613) 
* modelcard generation edit

* add missed tag

* fix param name

* fix var

* change str to dict

* add use_dora check

* use correct tags for lora

* make style && make quality

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:17 +05:30
YiYi Xu
1ef46d9d58 Revert "[LoRA] fix: lora loading when using with a device_mapped mode… (#9823) 
Revert "[LoRA] fix: lora loading when using with a device_mapped model. (#9449)"

This reverts commit 41e4779d98.
 2024-12-23 13:02:17 +05:30
Sayak Paul
bb6a324577 [LoRA] fix: lora loading when using with a device_mapped model. (#9449) 
* fix: lora loading when using with a device_mapped model.

* better attibutung

* empty

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

* Apply suggestions from code review

Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>

* minors

* better error messages.

* fix-copies

* add: tests, docs.

* add hardware note.

* quality

* Update docs/source/en/training/distributed_inference.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* fixes

* skip properly.

* fixes

---------

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul
18a46d12b2 [CI] add a big GPU marker to run memory-intensive tests separately on CI (#9691) 
* add a marker for big gpu tests

* update

* trigger on PRs temporarily.

* onnx

* fix

* total memory

* fixes

* reduce memory threshold.

* bigger gpu

* empty

* g6e

* Apply suggestions from code review

* address comments.

* fix

* fix

* fix

* fix

* fix

* okay

* further reduce.

* updates

* remove

* updates

* updates

* updates

* updates

* fixes

* fixes

* updates.

* fix

* workflow fixes.

---------

Co-authored-by: Aryan <aryan@huggingface.co>
 2024-12-23 13:02:17 +05:30
Sayak Paul
d143ba6478 [Tests] clean up and refactor gradient checkpointing tests (#9494) 
* check.

* fixes

* fixes

* updates

* fixes

* fixes
 2024-12-23 13:02:17 +05:30
Sayak Paul
2094e7a2b5 [training] use the lr when using 8bit adam. (#9796) 
* use the lr when using 8bit adam.

* remove lr as we pack it in params_to_optimize.

---------

Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul
dfbe972dd5 [training] fixes to the quantization training script and add AdEMAMix optimizer as an option (#9806) 
* fixes

* more fixes.
 2024-12-23 13:02:17 +05:30
Sayak Paul
bbbd1c0f99 [CI] add new runner for testing (#9699) 
new runner.
 2024-12-23 13:02:17 +05:30
Aryan
63c55c0c21 Allegro VAE fix (#9811) 
fix
 2024-12-23 13:02:17 +05:30
Aryan
c24688aab0 [core] Allegro T2V (#9736) 
* update

* refactor transformer part 1

* refactor part 2

* refactor part 3

* make style

* refactor part 4; modeling tests

* make style

* refactor part 5

* refactor part 6

* gradient checkpointing

* pipeline tests (broken atm)

* update

* add coauthor

Co-Authored-By: Huan Yang <hyang@fastmail.com>

* refactor part 7

* add docs

* make style

* add coauthor

Co-Authored-By: YiYi Xu <yixu310@gmail.com>

* make fix-copies

* undo unrelated change

* revert changes to embeddings, normalization, transformer

* refactor part 8

* make style

* refactor part 9

* make style

* fix

* apply suggestions from review

* Apply suggestions from code review

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* update example

* remove attention mask for self-attention

* update

* copied from

* update

* update

---------

Co-authored-by: Huan Yang <hyang@fastmail.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Raul Ciotescu
4690db221a adds the pipeline for pixart alpha controlnet (#8857) 
* add the controlnet pipeline for pixart alpha

---------

Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: junsongc <cjs1020440147@icloud.com>
 2024-12-23 13:02:17 +05:30
Linoy Tsaban
5905401d1e [flux dreambooth lora training] make LoRA target modules configurable + small bug fix (#9646) 
* make lora target modules configurable and change the default

* style

* make lora target modules configurable and change the default

* fix bug when using prodigy and training te

* fix mixed precision training as  proposed in https://github.com/huggingface/diffusers/pull/9565 for full dreambooth as well

* add test and notes

* style

* address sayaks comments

* style

* fix test

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Linoy Tsaban
ba31a14700 [SD 3.5 Dreambooth LoRA] support configurable training block & layers (#9762) 
* configurable layers

* configurable layers

* update README

* style

* add test

* style

* add layer test, update readme, add nargs

* readme

* test style

* remove print, change nargs

* test arg change

* style

* revert nargs 2/2

* address sayaks comments

* style

* address sayaks comments
 2024-12-23 13:02:17 +05:30
Biswaroop
dd6de12e1f [Fix] remove setting lr for T5 text encoder when using prodigy in flux dreambooth lora script (#9473) 
* fix: removed setting of text encoder lr for T5 as it's not being tuned

* fix: removed setting of text encoder lr for T5 as it's not being tuned

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Vinh H. Pham
2c6c9fc578 [Fix] train_dreambooth_lora_flux_advanced ValueError: unexpected save model: <class 'transformers.models.t5.modeling_t5.T5EncoderModel'> (#9777) 
fix save state te T5
 2024-12-23 13:02:17 +05:30
Sayak Paul
65a2db376c [research_projects] Update README.md to include a note about NF5 T5-xxl (#9775) 
Update README.md
 2024-12-23 13:02:17 +05:30
SahilCarterr
39f63d5746 Added Support of Xlabs controlnet to FluxControlNetInpaintPipeline (#9770) 
* added xlabs support
 2024-12-23 13:02:17 +05:30
Ina
969fa9f668 [refactor] enhance readability of flux related pipelines (#9711) 
* flux pipline: readability enhancement.
 2024-12-23 13:02:17 +05:30
Jingya HUANG
b85b6a74ef Add a doc for AWS Neuron in Diffusers (#9766) 
* start draft

* add doc

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

* bref intro of ON

* Update docs/source/en/optimization/neuron.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

---------

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
 2024-12-23 13:02:17 +05:30
Sayak Paul
56a2b8b9ad [research_projects] add flux training script with quantization (#9754) 
* add flux training script with quantization

* remove exclamation
 2024-12-23 13:02:17 +05:30
Leo Jiang
73a914ea68 [bugfix] bugfix for npu free memory (#9640) 
* Improve NPU performance

* Improve NPU performance

* Improve NPU performance

* Improve NPU performance

* [bugfix] bugfix for npu free memory

* [bugfix] bugfix for npu free memory

* [bugfix] bugfix for npu free memory

---------

Co-authored-by: 蒋硕 <jiangshuo9@h-partners.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30
Zhiyang Shen
8df1912b84 [Docs] fix docstring typo in SD3 pipeline (#9765) 
* fix docstring typo in SD3 pipeline

* fix docstring typo in SD3 pipeline
 2024-12-23 13:02:17 +05:30
Sayak Paul
876c8d76ef Some minor updates to the nightly and push workflows (#9759) 
* move lora integration tests to nightly./

* remove slow marker in the workflow where not needed.
 2024-12-23 13:02:17 +05:30
Rachit Shah
60d142d253 config attribute not foud error for FluxImagetoImage Pipeline for multi controlnet solved (#9586) 
Co-authored-by: YiYi Xu <yixu310@gmail.com>
 2024-12-23 13:02:17 +05:30
Linoy Tsaban
003676e961 [SD3-5 dreambooth lora] update model cards (#9749) 
* improve readme

* style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
 2024-12-23 13:02:17 +05:30