init

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+*.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,136 @@
+---
+base_model: black-forest-labs/FLUX.1-dev
+library_name: diffusers
+tags:
+- text-to-image
+- diffusers-training
+- diffusers
+- lora
+- flux
+- flux-diffusers
+widget:
+- text: "Serene raven-haired woman, moonlit lilies, swirling botanicals, yarn art style"
+  output:
+    url: >-
+      images/yarn_merged1.png
+- text: "a puppy in a pond, yarn art style"
+  output:
+    url: >-
+      images/yarn_merged2.png
+- text: "Ornate fox with a collar of autumn leaves and berries, amidst a tapestry of forest foliage, yarn art style"
+  output:
+    url: >-
+      images/yarn_merged3.png
+instance_prompt: null
+---
+# LoRA for FLUX.1-dev - Yarn Art Style
+
+This repository contains a LoRA (Low-Rank Adaptation) fine-tuned on `black-forest-labs/FLUX.1-dev` to generate images in the artistic style of Yarn Art. This work is part of the blog post, "Fine-Tuning FLUX.1-dev on consumer hardware and in FP8".
+
+<Gallery />
+
+## Inference
+
+There are two main ways to use this LoRA for inference: loading the adapter on the fly or merging it with the base model.
+
+### Option 1: Loading LoRA Adapters
+
+This approach offers flexibility, allowing you to easily switch between different LoRA styles.
+
+```python
+from diffusers import FluxPipeline
+import torch
+
+ckpt_id = "black-forest-labs/FLUX.1-dev"
+pipeline = FluxPipeline.from_pretrained(
+    ckpt_id, torch_dtype=torch.float16
+)
+pipeline.load_lora_weights("derekl35/yarn_qlora_flux", weight_name="pytorch_lora_weights.safetensors")
+pipeline.enable_model_cpu_offload()
+
+image = pipeline(
+    "a puppy in a pond, yarn art style",
+    num_inference_steps=28,
+    guidance_scale=3.5,
+    height=768,
+    generator=torch.manual_seed(0)
+).images[0]
+image.save("yarn_loaded.png")
+```
+
+### Option 2: Merging LoRA into Base Model
+
+Merging the LoRA into the base model can lead to slightly faster inference and is useful when you want to use a single style consistently.
+
+```python
+from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel
+import torch
+
+ckpt_id = "black-forest-labs/FLUX.1-dev"
+pipeline = FluxPipeline.from_pretrained(
+    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
+)
+pipeline.load_lora_weights("derekl35/yarn_qlora_flux", weight_name="pytorch_lora_weights.safetensors")
+pipeline.fuse_lora()
+pipeline.unload_lora_weights()
+
+# You can save the fused transformer for later use
+# pipeline.transformer.save_pretrained("fused_transformer")
+
+pipeline.enable_model_cpu_offload()
+image = pipeline(
+    "a puppy in a pond, yarn art style",
+    num_inference_steps=28,
+    guidance_scale=3.5,
+    height=768,
+    width=512,
+    generator=torch.manual_seed(0)
+).images[0]
+image.save("yarn_merged.png")
+```
+
+you can also requantize model:
+
+```python
+from diffusers import FluxPipeline, AutoPipelineForText2Image, FluxTransformer2DModel, BitsAndBytesConfig
+import torch 
+
+ckpt_id = "black-forest-labs/FLUX.1-dev"
+pipeline = FluxPipeline.from_pretrained(
+    ckpt_id, text_encoder=None, text_encoder_2=None, torch_dtype=torch.float16
+)
+pipeline.load_lora_weights("derekl35/yarn_qlora_flux", weight_name="pytorch_lora_weights.safetensors")
+pipeline.fuse_lora()
+pipeline.unload_lora_weights()
+
+pipeline.transformer.save_pretrained("fused_transformer")
+
+ckpt_id = "black-forest-labs/FLUX.1-dev"
+bnb_4bit_compute_dtype = torch.bfloat16
+
+nf4_config = BitsAndBytesConfig(
+    load_in_4bit=True,
+    bnb_4bit_quant_type="nf4",
+    bnb_4bit_compute_dtype=bnb_4bit_compute_dtype,
+)
+transformer = FluxTransformer2DModel.from_pretrained(
+    "fused_transformer",
+    quantization_config=nf4_config,
+    torch_dtype=bnb_4bit_compute_dtype,
+)
+
+pipeline = AutoPipelineForText2Image.from_pretrained(
+    ckpt_id, transformer=transformer, torch_dtype=bnb_4bit_compute_dtype
+)
+pipeline.enable_model_cpu_offload()
+
+image = pipeline(
+    "a puppy in a pond, yarn art style",
+    num_inference_steps=28,
+    guidance_scale=3.5,
+    height=768,
+    width=512,
+    generator=torch.manual_seed(0)
+).images[0]
+image.save("yarn_merged.png")
+```

pytorch_lora_weights.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a47c8786c9efc379b033b94dbc05b99061ddbaa22911bbd5361957d585451756
+size 18727592