YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

Quantization made by Richard Erkhov.

Github

Discord

Request more models

Mistral-7B-v0.1-4bit-32rank - bnb 4bits

Original model description:

license: mit language: - en pipeline_tag: text-generation tags: - 'quantization ' - lora

LoftQ Initialization

| Paper | Code | PEFT Example |

LoftQ (LoRA-fine-tuning-aware Quantization) provides a quantized backbone Q and LoRA adapters A and B, given a full-precision pre-trained weight W.

This model, Mistral-7B-v0.1-4bit-32rank, is obtained from Mistral-7B-v0.1. The backbone is under LoftQ/Mistral-7B-v0.1-4bit-32rank and LoRA adapters are under the subfolder='loftq_init'.

Model Info

Backbone

  • Stored format: torch.bfloat16
  • Size: ~ 14 GiB
  • Loaded format: bitsandbytes nf4
  • Size loaded on GPU: ~3.5 GiB

LoRA adapters

  • rank: 32
  • lora_alpha: 16
  • target_modules: ["down_proj", "up_proj", "q_proj", "k_proj", "v_proj", "o_proj", "gate_proj"]

Usage

Training. Here's an example of loading this model and preparing for the LoRA fine-tuning.

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

MODEL_ID = "LoftQ/Mistral-7B-v0.1-4bit-32rank"

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID, 
    torch_dtype=torch.bfloat16,  # you may change it with different models
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_compute_dtype=torch.bfloat16,  # bfloat16 is recommended
        bnb_4bit_use_double_quant=False,
        bnb_4bit_quant_type='nf4',
    ),
)
peft_model = PeftModel.from_pretrained(
    base_model,
    MODEL_ID,
    subfolder="loftq_init",
    is_trainable=True,
)

# Do training with peft_model ...

See the full code at our Github Repo

Citation

@article{li2023loftq,
  title={Loftq: Lora-fine-tuning-aware quantization for large language models},
  author={Li, Yixiao and Yu, Yifan and Liang, Chen and He, Pengcheng and Karampatziakis, Nikos and Chen, Weizhu and Zhao, Tuo},
  journal={arXiv preprint arXiv:2310.08659},
  year={2023}
}
Downloads last month
2
Safetensors
Model size
3.86B params
Tensor type
F32
FP16
U8
Inference Providers NEW
This model is not currently available via any of the supported Inference Providers.
The model cannot be deployed to the HF Inference API: The model has no library tag.