Spaces:
Running
Running
David Ko
commited on
Commit
ยท
a2e8511
1
Parent(s):
8ed5ac1
Add login feature with Flask-Login
Browse files- README.md +25 -2
- api.py +157 -2
- requirements.txt +1 -0
README.md
CHANGED
@@ -10,7 +10,7 @@ license: gpl-3.0
|
|
10 |
|
11 |
# Vision LLM Agent - Object Detection with AI Assistant
|
12 |
|
13 |
-
A multi-model object detection and image classification demo with LLM-based AI assistant for answering questions about detected objects. This project uses YOLOv8, DETR, and ViT models for vision tasks, and TinyLlama for natural language processing.
|
14 |
|
15 |
## Project Architecture
|
16 |
|
@@ -86,14 +86,37 @@ This project follows a phased development approach:
|
|
86 |
- **ViT**: Vision Transformer for image classification
|
87 |
- **TinyLlama**: For natural language processing and question answering about detected objects
|
88 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
89 |
## API Endpoints
|
90 |
|
91 |
-
The Flask backend provides the following API endpoints:
|
92 |
|
93 |
- `GET /api/status` - Check the status of the API and available models
|
94 |
- `POST /api/detect/yolo` - Detect objects using YOLOv8
|
95 |
- `POST /api/detect/detr` - Detect objects using DETR
|
96 |
- `POST /api/classify/vit` - Classify images using ViT
|
|
|
|
|
|
|
|
|
|
|
97 |
|
98 |
All POST endpoints accept form data with an 'image' field containing the image file.
|
99 |
|
|
|
10 |
|
11 |
# Vision LLM Agent - Object Detection with AI Assistant
|
12 |
|
13 |
+
A multi-model object detection and image classification demo with LLM-based AI assistant for answering questions about detected objects. This project uses YOLOv8, DETR, and ViT models for vision tasks, and TinyLlama for natural language processing. The application includes a secure login system to protect access to the AI features.
|
14 |
|
15 |
## Project Architecture
|
16 |
|
|
|
86 |
- **ViT**: Vision Transformer for image classification
|
87 |
- **TinyLlama**: For natural language processing and question answering about detected objects
|
88 |
|
89 |
+
## Authentication
|
90 |
+
|
91 |
+
The application includes a secure login system to protect access to all features:
|
92 |
+
|
93 |
+
- **Default Credentials**:
|
94 |
+
- Username: `admin` / Password: `admin123`
|
95 |
+
- Username: `user` / Password: `user123`
|
96 |
+
|
97 |
+
- **Login Process**:
|
98 |
+
- All routes and API endpoints are protected with Flask-Login
|
99 |
+
- Users must authenticate before accessing any features
|
100 |
+
- Session management handles login state persistence
|
101 |
+
|
102 |
+
- **Security Features**:
|
103 |
+
- Password protection for all API endpoints and UI pages
|
104 |
+
- Session-based authentication with secure cookies
|
105 |
+
- Configurable secret key via environment variables
|
106 |
+
|
107 |
## API Endpoints
|
108 |
|
109 |
+
The Flask backend provides the following API endpoints (all require authentication):
|
110 |
|
111 |
- `GET /api/status` - Check the status of the API and available models
|
112 |
- `POST /api/detect/yolo` - Detect objects using YOLOv8
|
113 |
- `POST /api/detect/detr` - Detect objects using DETR
|
114 |
- `POST /api/classify/vit` - Classify images using ViT
|
115 |
+
- `POST /api/analyze` - Analyze images with LLM assistant
|
116 |
+
- `POST /api/similar-images` - Find similar images in the vector database
|
117 |
+
- `POST /api/add-to-collection` - Add images to the vector database
|
118 |
+
- `POST /api/add-detected-objects` - Add detected objects to the vector database
|
119 |
+
- `POST /api/search-similar-objects` - Search for similar objects in the vector database
|
120 |
|
121 |
All POST endpoints accept form data with an 'image' field containing the image file.
|
122 |
|
api.py
CHANGED
@@ -3,7 +3,7 @@
|
|
3 |
import os
|
4 |
os.environ['MPLCONFIGDIR'] = '/tmp/matplotlib'
|
5 |
|
6 |
-
from flask import Flask, request, jsonify, send_from_directory
|
7 |
import torch
|
8 |
from PIL import Image
|
9 |
import numpy as np
|
@@ -17,6 +17,7 @@ import time
|
|
17 |
from flask_cors import CORS
|
18 |
import json
|
19 |
import sys
|
|
|
20 |
|
21 |
# Fix for SQLite3 version compatibility with ChromaDB
|
22 |
try:
|
@@ -31,6 +32,34 @@ from chromadb.utils import embedding_functions
|
|
31 |
app = Flask(__name__, static_folder='static')
|
32 |
CORS(app) # Enable CORS for all routes
|
33 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
34 |
# Model initialization
|
35 |
print("Loading models... This may take a moment.")
|
36 |
|
@@ -434,6 +463,7 @@ def process_vit(image):
|
|
434 |
}
|
435 |
|
436 |
@app.route('/api/detect/yolo', methods=['POST'])
|
|
|
437 |
def yolo_detect():
|
438 |
if 'image' not in request.files:
|
439 |
return jsonify({"error": "No image provided"}), 400
|
@@ -445,6 +475,7 @@ def yolo_detect():
|
|
445 |
return jsonify(result)
|
446 |
|
447 |
@app.route('/api/detect/detr', methods=['POST'])
|
|
|
448 |
def detr_detect():
|
449 |
if 'image' not in request.files:
|
450 |
return jsonify({"error": "No image provided"}), 400
|
@@ -456,6 +487,7 @@ def detr_detect():
|
|
456 |
return jsonify(result)
|
457 |
|
458 |
@app.route('/api/classify/vit', methods=['POST'])
|
|
|
459 |
def vit_classify():
|
460 |
if 'image' not in request.files:
|
461 |
return jsonify({"error": "No image provided"}), 400
|
@@ -467,6 +499,7 @@ def vit_classify():
|
|
467 |
return jsonify(result)
|
468 |
|
469 |
@app.route('/api/analyze', methods=['POST'])
|
|
|
470 |
def analyze_with_llm():
|
471 |
# Check if required data is in the request
|
472 |
if not request.json:
|
@@ -508,6 +541,7 @@ def generate_image_embedding(image):
|
|
508 |
return None
|
509 |
|
510 |
@app.route('/api/similar-images', methods=['POST'])
|
|
|
511 |
def find_similar_images():
|
512 |
"""์ ์ฌ ์ด๋ฏธ์ง ๊ฒ์ API"""
|
513 |
if clip_model is None or clip_processor is None or image_collection is None:
|
@@ -570,6 +604,7 @@ def find_similar_images():
|
|
570 |
return jsonify({"error": str(e)}), 500
|
571 |
|
572 |
@app.route('/api/add-to-collection', methods=['POST'])
|
|
|
573 |
def add_to_collection():
|
574 |
"""์ด๋ฏธ์ง๋ฅผ ๋ฒกํฐ DB์ ์ถ๊ฐํ๋ API"""
|
575 |
if clip_model is None or clip_processor is None or image_collection is None:
|
@@ -629,6 +664,7 @@ def add_to_collection():
|
|
629 |
return jsonify({"error": str(e)}), 500
|
630 |
|
631 |
@app.route('/api/add-detected-objects', methods=['POST'])
|
|
|
632 |
def add_detected_objects():
|
633 |
"""๊ฐ์ฒด ์ธ์ ๊ฒฐ๊ณผ๋ฅผ ๋ฒกํฐ DB์ ์ถ๊ฐํ๋ API"""
|
634 |
if clip_model is None or object_collection is None:
|
@@ -782,6 +818,7 @@ def add_detected_objects():
|
|
782 |
return jsonify({"error": str(e)}), 500
|
783 |
|
784 |
@app.route('/api/search-similar-objects', methods=['POST'])
|
|
|
785 |
def search_similar_objects():
|
786 |
"""์ ์ฌํ ๊ฐ์ฒด ๊ฒ์ API"""
|
787 |
print("[DEBUG] Received request in search-similar-objects")
|
@@ -987,8 +1024,119 @@ def format_object_results(results):
|
|
987 |
|
988 |
return formatted_results
|
989 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
990 |
@app.route('/', defaults={'path': ''}, methods=['GET'])
|
991 |
@app.route('/<path:path>', methods=['GET'])
|
|
|
992 |
def serve_react(path):
|
993 |
"""Serve React frontend"""
|
994 |
if path != "" and os.path.exists(os.path.join(app.static_folder, path)):
|
@@ -997,21 +1145,25 @@ def serve_react(path):
|
|
997 |
return send_from_directory(app.static_folder, 'index.html')
|
998 |
|
999 |
@app.route('/similar-images', methods=['GET'])
|
|
|
1000 |
def similar_images_page():
|
1001 |
"""Serve similar images search page"""
|
1002 |
return send_from_directory(app.static_folder, 'similar-images.html')
|
1003 |
|
1004 |
@app.route('/object-detection-search', methods=['GET'])
|
|
|
1005 |
def object_detection_search_page():
|
1006 |
"""Serve object detection search page"""
|
1007 |
return send_from_directory(app.static_folder, 'object-detection-search.html')
|
1008 |
|
1009 |
@app.route('/model-vector-db', methods=['GET'])
|
|
|
1010 |
def model_vector_db_page():
|
1011 |
"""Serve model vector DB UI page"""
|
1012 |
return send_from_directory(app.static_folder, 'model-vector-db.html')
|
1013 |
|
1014 |
@app.route('/api/status', methods=['GET'])
|
|
|
1015 |
def status():
|
1016 |
return jsonify({
|
1017 |
"status": "online",
|
@@ -1020,9 +1172,12 @@ def status():
|
|
1020 |
"detr": detr_model is not None and detr_processor is not None,
|
1021 |
"vit": vit_model is not None and vit_processor is not None
|
1022 |
},
|
1023 |
-
"device": "GPU" if torch.cuda.is_available() else "CPU"
|
|
|
1024 |
})
|
1025 |
|
|
|
|
|
1026 |
def index():
|
1027 |
return send_from_directory('static', 'index.html')
|
1028 |
|
|
|
3 |
import os
|
4 |
os.environ['MPLCONFIGDIR'] = '/tmp/matplotlib'
|
5 |
|
6 |
+
from flask import Flask, request, jsonify, send_from_directory, redirect, url_for, session, render_template_string
|
7 |
import torch
|
8 |
from PIL import Image
|
9 |
import numpy as np
|
|
|
17 |
from flask_cors import CORS
|
18 |
import json
|
19 |
import sys
|
20 |
+
from flask_login import LoginManager, UserMixin, login_user, logout_user, login_required, current_user
|
21 |
|
22 |
# Fix for SQLite3 version compatibility with ChromaDB
|
23 |
try:
|
|
|
32 |
app = Flask(__name__, static_folder='static')
|
33 |
CORS(app) # Enable CORS for all routes
|
34 |
|
35 |
+
# ์ํฌ๋ฆฟ ํค ์ค์ (์ธ์
์ํธํ์ ์ฌ์ฉ)
|
36 |
+
app.config['SECRET_KEY'] = os.environ.get('SECRET_KEY', 'vision_llm_agent_secret_key')
|
37 |
+
|
38 |
+
# Flask-Login ์ค์
|
39 |
+
login_manager = LoginManager()
|
40 |
+
login_manager.init_app(app)
|
41 |
+
login_manager.login_view = 'login'
|
42 |
+
|
43 |
+
# ์ฌ์ฉ์ ํด๋์ค ์ ์
|
44 |
+
class User(UserMixin):
|
45 |
+
def __init__(self, id, username, password):
|
46 |
+
self.id = id
|
47 |
+
self.username = username
|
48 |
+
self.password = password
|
49 |
+
|
50 |
+
# ํ
์คํธ์ฉ ์ฌ์ฉ์ (์ค์ ํ๊ฒฝ์์๋ ๋ฐ์ดํฐ๋ฒ ์ด์ค ์ฌ์ฉ ๊ถ์ฅ)
|
51 |
+
users = {
|
52 |
+
'admin': User('1', 'admin', 'admin123'),
|
53 |
+
'user': User('2', 'user', 'user123')
|
54 |
+
}
|
55 |
+
|
56 |
+
@login_manager.user_loader
|
57 |
+
def load_user(user_id):
|
58 |
+
for user in users.values():
|
59 |
+
if user.id == user_id:
|
60 |
+
return user
|
61 |
+
return None
|
62 |
+
|
63 |
# Model initialization
|
64 |
print("Loading models... This may take a moment.")
|
65 |
|
|
|
463 |
}
|
464 |
|
465 |
@app.route('/api/detect/yolo', methods=['POST'])
|
466 |
+
@login_required
|
467 |
def yolo_detect():
|
468 |
if 'image' not in request.files:
|
469 |
return jsonify({"error": "No image provided"}), 400
|
|
|
475 |
return jsonify(result)
|
476 |
|
477 |
@app.route('/api/detect/detr', methods=['POST'])
|
478 |
+
@login_required
|
479 |
def detr_detect():
|
480 |
if 'image' not in request.files:
|
481 |
return jsonify({"error": "No image provided"}), 400
|
|
|
487 |
return jsonify(result)
|
488 |
|
489 |
@app.route('/api/classify/vit', methods=['POST'])
|
490 |
+
@login_required
|
491 |
def vit_classify():
|
492 |
if 'image' not in request.files:
|
493 |
return jsonify({"error": "No image provided"}), 400
|
|
|
499 |
return jsonify(result)
|
500 |
|
501 |
@app.route('/api/analyze', methods=['POST'])
|
502 |
+
@login_required
|
503 |
def analyze_with_llm():
|
504 |
# Check if required data is in the request
|
505 |
if not request.json:
|
|
|
541 |
return None
|
542 |
|
543 |
@app.route('/api/similar-images', methods=['POST'])
|
544 |
+
@login_required
|
545 |
def find_similar_images():
|
546 |
"""์ ์ฌ ์ด๋ฏธ์ง ๊ฒ์ API"""
|
547 |
if clip_model is None or clip_processor is None or image_collection is None:
|
|
|
604 |
return jsonify({"error": str(e)}), 500
|
605 |
|
606 |
@app.route('/api/add-to-collection', methods=['POST'])
|
607 |
+
@login_required
|
608 |
def add_to_collection():
|
609 |
"""์ด๋ฏธ์ง๋ฅผ ๋ฒกํฐ DB์ ์ถ๊ฐํ๋ API"""
|
610 |
if clip_model is None or clip_processor is None or image_collection is None:
|
|
|
664 |
return jsonify({"error": str(e)}), 500
|
665 |
|
666 |
@app.route('/api/add-detected-objects', methods=['POST'])
|
667 |
+
@login_required
|
668 |
def add_detected_objects():
|
669 |
"""๊ฐ์ฒด ์ธ์ ๊ฒฐ๊ณผ๋ฅผ ๋ฒกํฐ DB์ ์ถ๊ฐํ๋ API"""
|
670 |
if clip_model is None or object_collection is None:
|
|
|
818 |
return jsonify({"error": str(e)}), 500
|
819 |
|
820 |
@app.route('/api/search-similar-objects', methods=['POST'])
|
821 |
+
@login_required
|
822 |
def search_similar_objects():
|
823 |
"""์ ์ฌํ ๊ฐ์ฒด ๊ฒ์ API"""
|
824 |
print("[DEBUG] Received request in search-similar-objects")
|
|
|
1024 |
|
1025 |
return formatted_results
|
1026 |
|
1027 |
+
# ๋ก๊ทธ์ธ ํ์ด์ง HTML ํ
ํ๋ฆฟ
|
1028 |
+
LOGIN_TEMPLATE = '''
|
1029 |
+
<!DOCTYPE html>
|
1030 |
+
<html lang="ko">
|
1031 |
+
<head>
|
1032 |
+
<meta charset="UTF-8">
|
1033 |
+
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
1034 |
+
<title>Vision LLM Agent - ๋ก๊ทธ์ธ</title>
|
1035 |
+
<style>
|
1036 |
+
body {
|
1037 |
+
font-family: Arial, sans-serif;
|
1038 |
+
background-color: #f5f5f5;
|
1039 |
+
display: flex;
|
1040 |
+
justify-content: center;
|
1041 |
+
align-items: center;
|
1042 |
+
height: 100vh;
|
1043 |
+
margin: 0;
|
1044 |
+
}
|
1045 |
+
.login-container {
|
1046 |
+
background-color: white;
|
1047 |
+
padding: 2rem;
|
1048 |
+
border-radius: 8px;
|
1049 |
+
box-shadow: 0 4px 6px rgba(0, 0, 0, 0.1);
|
1050 |
+
width: 100%;
|
1051 |
+
max-width: 400px;
|
1052 |
+
}
|
1053 |
+
h1 {
|
1054 |
+
text-align: center;
|
1055 |
+
color: #4a6cf7;
|
1056 |
+
margin-bottom: 1.5rem;
|
1057 |
+
}
|
1058 |
+
.form-group {
|
1059 |
+
margin-bottom: 1rem;
|
1060 |
+
}
|
1061 |
+
label {
|
1062 |
+
display: block;
|
1063 |
+
margin-bottom: 0.5rem;
|
1064 |
+
font-weight: bold;
|
1065 |
+
}
|
1066 |
+
input {
|
1067 |
+
width: 100%;
|
1068 |
+
padding: 0.75rem;
|
1069 |
+
border: 1px solid #ddd;
|
1070 |
+
border-radius: 4px;
|
1071 |
+
font-size: 1rem;
|
1072 |
+
}
|
1073 |
+
button {
|
1074 |
+
width: 100%;
|
1075 |
+
padding: 0.75rem;
|
1076 |
+
background-color: #4a6cf7;
|
1077 |
+
color: white;
|
1078 |
+
border: none;
|
1079 |
+
border-radius: 4px;
|
1080 |
+
font-size: 1rem;
|
1081 |
+
cursor: pointer;
|
1082 |
+
margin-top: 1rem;
|
1083 |
+
}
|
1084 |
+
button:hover {
|
1085 |
+
background-color: #3a5cd8;
|
1086 |
+
}
|
1087 |
+
.error-message {
|
1088 |
+
color: #e74c3c;
|
1089 |
+
margin-top: 1rem;
|
1090 |
+
text-align: center;
|
1091 |
+
}
|
1092 |
+
</style>
|
1093 |
+
</head>
|
1094 |
+
<body>
|
1095 |
+
<div class="login-container">
|
1096 |
+
<h1>Vision LLM Agent</h1>
|
1097 |
+
<form action="/login" method="post">
|
1098 |
+
<div class="form-group">
|
1099 |
+
<label for="username">์ฌ์ฉ์ ID</label>
|
1100 |
+
<input type="text" id="username" name="username" required>
|
1101 |
+
</div>
|
1102 |
+
<div class="form-group">
|
1103 |
+
<label for="password">๋น๋ฐ๋ฒํธ</label>
|
1104 |
+
<input type="password" id="password" name="password" required>
|
1105 |
+
</div>
|
1106 |
+
<button type="submit">๋ก๊ทธ์ธ</button>
|
1107 |
+
{% if error %}
|
1108 |
+
<p class="error-message">{{ error }}</p>
|
1109 |
+
{% endif %}
|
1110 |
+
</form>
|
1111 |
+
</div>
|
1112 |
+
</body>
|
1113 |
+
</html>
|
1114 |
+
'''
|
1115 |
+
|
1116 |
+
@app.route('/login', methods=['GET', 'POST'])
|
1117 |
+
def login():
|
1118 |
+
error = None
|
1119 |
+
if request.method == 'POST':
|
1120 |
+
username = request.form.get('username')
|
1121 |
+
password = request.form.get('password')
|
1122 |
+
|
1123 |
+
if username in users and users[username].password == password:
|
1124 |
+
login_user(users[username])
|
1125 |
+
next_page = request.args.get('next')
|
1126 |
+
return redirect(next_page or url_for('index'))
|
1127 |
+
else:
|
1128 |
+
error = '์๋ชป๋ ์ฌ์ฉ์ ID ๋๋ ๋น๋ฐ๋ฒํธ์
๋๋ค.'
|
1129 |
+
|
1130 |
+
return render_template_string(LOGIN_TEMPLATE, error=error)
|
1131 |
+
|
1132 |
+
@app.route('/logout')
|
1133 |
+
def logout():
|
1134 |
+
logout_user()
|
1135 |
+
return redirect(url_for('login'))
|
1136 |
+
|
1137 |
@app.route('/', defaults={'path': ''}, methods=['GET'])
|
1138 |
@app.route('/<path:path>', methods=['GET'])
|
1139 |
+
@login_required
|
1140 |
def serve_react(path):
|
1141 |
"""Serve React frontend"""
|
1142 |
if path != "" and os.path.exists(os.path.join(app.static_folder, path)):
|
|
|
1145 |
return send_from_directory(app.static_folder, 'index.html')
|
1146 |
|
1147 |
@app.route('/similar-images', methods=['GET'])
|
1148 |
+
@login_required
|
1149 |
def similar_images_page():
|
1150 |
"""Serve similar images search page"""
|
1151 |
return send_from_directory(app.static_folder, 'similar-images.html')
|
1152 |
|
1153 |
@app.route('/object-detection-search', methods=['GET'])
|
1154 |
+
@login_required
|
1155 |
def object_detection_search_page():
|
1156 |
"""Serve object detection search page"""
|
1157 |
return send_from_directory(app.static_folder, 'object-detection-search.html')
|
1158 |
|
1159 |
@app.route('/model-vector-db', methods=['GET'])
|
1160 |
+
@login_required
|
1161 |
def model_vector_db_page():
|
1162 |
"""Serve model vector DB UI page"""
|
1163 |
return send_from_directory(app.static_folder, 'model-vector-db.html')
|
1164 |
|
1165 |
@app.route('/api/status', methods=['GET'])
|
1166 |
+
@login_required
|
1167 |
def status():
|
1168 |
return jsonify({
|
1169 |
"status": "online",
|
|
|
1172 |
"detr": detr_model is not None and detr_processor is not None,
|
1173 |
"vit": vit_model is not None and vit_processor is not None
|
1174 |
},
|
1175 |
+
"device": "GPU" if torch.cuda.is_available() else "CPU",
|
1176 |
+
"user": current_user.username
|
1177 |
})
|
1178 |
|
1179 |
+
@app.route('/index')
|
1180 |
+
@login_required
|
1181 |
def index():
|
1182 |
return send_from_directory('static', 'index.html')
|
1183 |
|
requirements.txt
CHANGED
@@ -11,6 +11,7 @@ timm>=0.9.0 # Vision Transformer support
|
|
11 |
# API dependencies
|
12 |
flask>=2.0.0
|
13 |
flask-cors>=3.0.0
|
|
|
14 |
matplotlib>=3.5.0
|
15 |
numpy>=1.20.0
|
16 |
|
|
|
11 |
# API dependencies
|
12 |
flask>=2.0.0
|
13 |
flask-cors>=3.0.0
|
14 |
+
flask-login>=0.6.2
|
15 |
matplotlib>=3.5.0
|
16 |
numpy>=1.20.0
|
17 |
|