Spaces:
Sleeping
Sleeping
Update README.md
Browse files
README.md
CHANGED
@@ -10,7 +10,9 @@ pinned: false
|
|
10 |
license: apache-2.0
|
11 |
---
|
12 |
|
13 |
-
|
|
|
|
|
14 |
|
15 |
# ControlLLM
|
16 |
|
@@ -21,19 +23,16 @@ We present ControlLLM, a novel framework that enables large language models (LLM
|
|
21 |
|
22 |
## ๐ค Video Demo
|
23 |
|
24 |
-
<!-- <table>
|
25 |
-
<tr>
|
26 |
-
<td><img width="450" src="https://github.com/liu-zhy/graph-of-thought/assets/26198430/7fe7d1ec-e37e-4ea8-8201-dc639c82ba66" alt="Image 1"></td>
|
27 |
-
<td><img width="450" src="https://github.com/liu-zhy/graph-of-thought/assets/26198430/a8bc6644-368b-42e3-844a-9962fdc9bd01" alt="Image 2"></td>
|
28 |
-
</tr>
|
29 |
-
</table>
|
30 |
-
-->
|
31 |
|
32 |
-
|
|
|
|
|
|
|
33 |
|
34 |
## ๐ System Overview
|
35 |
|
36 |
-
|
|
|
37 |
|
38 |
## ๐ Major Features
|
39 |
- Image Perception
|
@@ -52,126 +51,6 @@ https://github.com/OpenGVLab/ControlLLM/assets/13723743/cf72861e-0e7b-4c15-89ee-
|
|
52 |
|
53 |
- [ ] Launch online demo
|
54 |
|
55 |
-
## ๐ ๏ธInstallation
|
56 |
-
|
57 |
-
### Basic requirements
|
58 |
-
|
59 |
-
* Linux
|
60 |
-
* Python 3.10+
|
61 |
-
* PyTorch 2.0+
|
62 |
-
* CUDA 11.8+
|
63 |
-
|
64 |
-
### Clone project
|
65 |
-
|
66 |
-
Execute the following command in the root directory:
|
67 |
-
|
68 |
-
```bash
|
69 |
-
git clone https://github.com/OpenGVLab/ControlLLM.git
|
70 |
-
```
|
71 |
-
|
72 |
-
### Install dependencies
|
73 |
-
|
74 |
-
Setup environment:
|
75 |
-
|
76 |
-
```bash
|
77 |
-
|
78 |
-
conda create -n cllm python=3.10
|
79 |
-
|
80 |
-
conda activate cllm
|
81 |
-
|
82 |
-
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.8 -c pytorch -c nvidia
|
83 |
-
```
|
84 |
-
|
85 |
-
Install [LLaVA](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file):
|
86 |
-
|
87 |
-
```bash
|
88 |
-
pip install git+https://github.com/haotian-liu/LLaVA.git
|
89 |
-
```
|
90 |
-
|
91 |
-
Then install other dependencies:
|
92 |
-
|
93 |
-
```bash
|
94 |
-
cd controlllm
|
95 |
-
|
96 |
-
pip install -r requirements.txt
|
97 |
-
```
|
98 |
-
|
99 |
-
## ๐จโ๐ซ Get Started
|
100 |
-
|
101 |
-
### Launch tool services
|
102 |
-
|
103 |
-
Please put your personal OpenAI Key and [Weather Key](https://www.visualcrossing.com/weather-api) into the corresponding environment variables.
|
104 |
-
```bash
|
105 |
-
|
106 |
-
cd ./controlllm
|
107 |
-
# openai key
|
108 |
-
export OPENAI_API_KEY="..."
|
109 |
-
# openai base
|
110 |
-
export OPENAI_BASE_URL="..."
|
111 |
-
# weather api key
|
112 |
-
export WEATHER_API_KEY="..."
|
113 |
-
|
114 |
-
python -m cllm.services.launch --port 10011 --host 0.0.0.0
|
115 |
-
```
|
116 |
-
|
117 |
-
### Launch ToG service
|
118 |
-
|
119 |
-
```bash
|
120 |
-
cd ./controlllm
|
121 |
-
|
122 |
-
export TOG_SERVICES_PORT=10011
|
123 |
-
export OPENAI_BASE_URL="..."
|
124 |
-
export OPENAI_API_KEY="..."
|
125 |
-
python -m cllm.services.tog.launch --port 10012 --host 0.0.0.0
|
126 |
-
|
127 |
-
```
|
128 |
-
|
129 |
-
### Launch gradio demo
|
130 |
-
|
131 |
-
Use `openssl` to generate the certificate:
|
132 |
-
```shell
|
133 |
-
mkdir certificate
|
134 |
-
|
135 |
-
openssl req -x509 -newkey rsa:4096 -keyout certificate/key.pem -out certificate/cert.pem -sha256 -days 365 -nodes
|
136 |
-
```
|
137 |
-
Launch gradio demo:
|
138 |
-
```bash
|
139 |
-
cd ./controlllm
|
140 |
-
|
141 |
-
export TOG_PORT=10012
|
142 |
-
export TOG_SERVICES_PORT=10011
|
143 |
-
export RESOURCE_ROOT="./client_resources"
|
144 |
-
export GRADIO_TEMP_DIR="$HOME/.tmp"
|
145 |
-
export OPENAI_BASE_URL="..."
|
146 |
-
export OPENAI_API_KEY="..."
|
147 |
-
|
148 |
-
python -m cllm.app.gradio --controller "cllm.agents.tog.Controller" --server_port 10024
|
149 |
-
|
150 |
-
```
|
151 |
-
|
152 |
-
### Tools as Services
|
153 |
-
|
154 |
-
Take image generation as an example, we first launch the service.
|
155 |
-
|
156 |
-
```bash
|
157 |
-
|
158 |
-
python -m cllm.services.image_generation.launch --port 10011 --host 0.0.0.0
|
159 |
-
|
160 |
-
```
|
161 |
-
|
162 |
-
Then, we call the services via python api.
|
163 |
-
|
164 |
-
```python
|
165 |
-
from cllm.services.image_generation.api import *
|
166 |
-
setup(port=10011)
|
167 |
-
text2image('A horse')
|
168 |
-
```
|
169 |
-
|
170 |
-
๐ฌ Launch all in one endpoint
|
171 |
-
|
172 |
-
```bash
|
173 |
-
python -m cllm.services.launch --port 10011 --host 0.0.0.0
|
174 |
-
```
|
175 |
|
176 |
## ๐ ๏ธ Support Tools
|
177 |
|
|
|
10 |
license: apache-2.0
|
11 |
---
|
12 |
|
13 |
+
|
14 |
+
<img src="https://cdn-uploads.huggingface.co/production/uploads/64564b0e4a7ffb7d5a47f412/T8e6gFBYtSxo4qPE7cRwK.png" width=10% align="left" />
|
15 |
+
|
16 |
|
17 |
# ControlLLM
|
18 |
|
|
|
23 |
|
24 |
## ๐ค Video Demo
|
25 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
26 |
|
27 |
+
|
28 |
+
|
29 |
+
<video controls autoplay src="https://cdn-uploads.huggingface.co/production/uploads/64564b0e4a7ffb7d5a47f412/vM9d412xQx2ajNkMew2ut.mp4"></video>
|
30 |
+
|
31 |
|
32 |
## ๐ System Overview
|
33 |
|
34 |
+
|
35 |
+
![image/png](https://cdn-uploads.huggingface.co/production/uploads/64564b0e4a7ffb7d5a47f412/0LB4HVZlt02Qhg5xXLCop.png)
|
36 |
|
37 |
## ๐ Major Features
|
38 |
- Image Perception
|
|
|
51 |
|
52 |
- [ ] Launch online demo
|
53 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
54 |
|
55 |
## ๐ ๏ธ Support Tools
|
56 |
|