Update README.md
Browse files
README.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
# Accuracy Recovery Adapters
|
| 2 |
|
| 3 |
This repo contains various accuracy recovery adapters (ARAs) that I have trained, primarialy for use with [AI Toolkit](https://github.com/ostris/ai-toolkit).
|
| 4 |
-
An ARA is a LoRA that is trained via student teacher training with the student being quantized
|
| 5 |
The goal is to have a side chain LoRA, at bfloat16, that runs parallel to highly quantized layers in a network to compensate for the loss in precision that happens
|
| 6 |
when weights are quantized. The training is done on a per layer basis in order to match the parent output as much as possible.
|
| 7 |
|
|
@@ -10,7 +10,7 @@ is now possible to train a LoRA on top of it, with 1 MP images, on a 24 GB GPU,
|
|
| 10 |
|
| 11 |
I have found the sweet spot, at least for [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image), is 3 bit quantization with a rank 16 adapter.
|
| 12 |
|
| 13 |
-
More info, links, training scripts, AI Toolkit example configs, and adapters to some soon.
|
| 14 |
|
| 15 |
## Qwen-Image 3 bit quantization
|
| 16 |
|
|
|
|
| 1 |
# Accuracy Recovery Adapters
|
| 2 |
|
| 3 |
This repo contains various accuracy recovery adapters (ARAs) that I have trained, primarialy for use with [AI Toolkit](https://github.com/ostris/ai-toolkit).
|
| 4 |
+
An ARA is a LoRA that is trained via student teacher training with the student being quantized down to a low precision and the teacher having a high precision.
|
| 5 |
The goal is to have a side chain LoRA, at bfloat16, that runs parallel to highly quantized layers in a network to compensate for the loss in precision that happens
|
| 6 |
when weights are quantized. The training is done on a per layer basis in order to match the parent output as much as possible.
|
| 7 |
|
|
|
|
| 10 |
|
| 11 |
I have found the sweet spot, at least for [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image), is 3 bit quantization with a rank 16 adapter.
|
| 12 |
|
| 13 |
+
More info, examples, links, training scripts, AI Toolkit example configs, and adapters to some soon.
|
| 14 |
|
| 15 |
## Qwen-Image 3 bit quantization
|
| 16 |
|