cgus
/

Text Generation
English
Chinese
qwen2
medical
conversational
4-bit precision
exl2
cgus commited on
Commit
ac0d189
·
verified ·
1 Parent(s): c9308f0

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +17 -1
README.md CHANGED
@@ -7,12 +7,28 @@ language:
7
  - en
8
  - zh
9
  base_model:
10
- - Qwen/Qwen2.5-7B-Instruct
11
  pipeline_tag: text-generation
12
  tags:
13
  - medical
14
  ---
 
 
 
 
 
 
 
 
 
15
 
 
 
 
 
 
 
 
16
  <div align="center">
17
  <h1>
18
  HuatuoGPT-o1-7B
 
7
  - en
8
  - zh
9
  base_model:
10
+ - FreedomIntelligence/HuatuoGPT-o1-7B
11
  pipeline_tag: text-generation
12
  tags:
13
  - medical
14
  ---
15
+ # HuatuoGPT-o1-7B-exl2
16
+ Original model: [HuatuoGPT-o1-7B](https://huggingface.co/FreedomIntelligence/HuatuoGPT-o1-7B) made by [FreedomIntelligence](https://huggingface.co/FreedomIntelligence)
17
+ Based on: [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) by [Qwen](https://huggingface.co/Qwen)
18
+ ## Quants
19
+ [4bpw h6 (main)](https://huggingface.co/cgus/HuatuoGPT-o1-7B-exl2/tree/main)
20
+ [4.5bpw h6](https://huggingface.co/cgus/HuatuoGPT-o1-7B-exl2/tree/4.5bpw-h6)
21
+ [5bpw h6](https://huggingface.co/cgus/HuatuoGPT-o1-7B-exl2/tree/5bpw-h6)
22
+ [6bpw h6](https://huggingface.co/cgus/HuatuoGPT-o1-7B-exl2/tree/6bpw-h6)
23
+ [8bpw h8](https://huggingface.co/cgus/HuatuoGPT-o1-7B-exl2/tree/8bpw-h8)
24
 
25
+ ## Quantization notes
26
+ Made with Exllamav2 0.2.7 with default dataset.
27
+ Exl2 quants require Nvidia RTX on Windows or Nvidia RTX/AMD ROCm on Linux.
28
+ Model has to fully fit GPU as RAM offloading isn't supported natively.
29
+ It can be used with apps such as TabbyAPI, Text-Generation-WebUI, LoLLMs and others.
30
+
31
+ # Original model card
32
  <div align="center">
33
  <h1>
34
  HuatuoGPT-o1-7B