ostris commited on
Commit
7e670a7
·
verified ·
1 Parent(s): 6c57636

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -1,7 +1,7 @@
1
  # Accuracy Recovery Adapters
2
 
3
  This repo contains various accuracy recovery adapters (ARAs) that I have trained, primarialy for use with [AI Toolkit](https://github.com/ostris/ai-toolkit).
4
- An ARA is a LoRA that is trained via student teacher training with the student being quantized dow a low precision and the teacher having a high precision.
5
  The goal is to have a side chain LoRA, at bfloat16, that runs parallel to highly quantized layers in a network to compensate for the loss in precision that happens
6
  when weights are quantized. The training is done on a per layer basis in order to match the parent output as much as possible.
7
 
@@ -10,7 +10,7 @@ is now possible to train a LoRA on top of it, with 1 MP images, on a 24 GB GPU,
10
 
11
  I have found the sweet spot, at least for [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image), is 3 bit quantization with a rank 16 adapter.
12
 
13
- More info, links, training scripts, AI Toolkit example configs, and adapters to some soon.
14
 
15
  ## Qwen-Image 3 bit quantization
16
 
 
1
  # Accuracy Recovery Adapters
2
 
3
  This repo contains various accuracy recovery adapters (ARAs) that I have trained, primarialy for use with [AI Toolkit](https://github.com/ostris/ai-toolkit).
4
+ An ARA is a LoRA that is trained via student teacher training with the student being quantized down to a low precision and the teacher having a high precision.
5
  The goal is to have a side chain LoRA, at bfloat16, that runs parallel to highly quantized layers in a network to compensate for the loss in precision that happens
6
  when weights are quantized. The training is done on a per layer basis in order to match the parent output as much as possible.
7
 
 
10
 
11
  I have found the sweet spot, at least for [Qwen-Image](https://huggingface.co/Qwen/Qwen-Image), is 3 bit quantization with a rank 16 adapter.
12
 
13
+ More info, examples, links, training scripts, AI Toolkit example configs, and adapters to some soon.
14
 
15
  ## Qwen-Image 3 bit quantization
16