Spaces:
Running
docs: Add comprehensive job submission requirements
Browse filesCritical Updates:
- Added HuggingFace Pro account requirement (/month with credit card)
- Clarified HF token needs Read + Write + Run Jobs permissions
- Updated all documentation to warn users about requirements
Documentation Screen (SMOLTRACE tab):
- Added prominent Requirements section before job configuration
- Listed HF Pro account requirement with pricing link
- Specified token permissions needed (Read + Write + Run Jobs)
- Added hardware pricing estimates (~/usr/bin/bash.05/hr to ~.00/hr)
- Warned that read-only tokens will NOT work
Settings Screen:
- Updated HF token label to indicate job submission requirement
- Added warning in info text about Pro account requirement
- Expanded HF token documentation with required permissions checklist
- Added links to HF pricing and billing pages
README:
- Split Prerequisites into viewing vs job submission
- Added Requirements for Job Submission section with clear warnings
- Added hardware options and pricing information
- Added step-by-step guide for submitting jobs
- Clarified what happens during job execution
- Updated HF token guidance for both viewing and job submission
Why These Changes:
- Users were not aware HF Pro account is required for jobs
- Many users have read-only tokens which fail silently
- Credit card requirement was not mentioned
- Pricing information was missing, users couldn't estimate costs
Impact:
- Sets clear expectations before users attempt job submission
- Prevents frustration from failed submissions due to permissions
- Helps users budget for compute costs
- Improves transparency about platform requirements
- README.md +79 -6
- screens/documentation.py +34 -6
- screens/settings.py +14 -5
|
@@ -81,9 +81,17 @@ TraceMind demonstrates enterprise MCP client usage by connecting to [TraceMind-m
|
|
| 81 |
## Quick Start
|
| 82 |
|
| 83 |
### Prerequisites
|
|
|
|
|
|
|
| 84 |
- Python 3.10+
|
| 85 |
- HuggingFace account (for authentication)
|
| 86 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
### Installation
|
| 89 |
|
|
@@ -149,13 +157,11 @@ Both APIs have generous free tiers:
|
|
| 149 |
- Click "Get API Key" โ Create project โ Generate key
|
| 150 |
- **Free tier**: 1,500 requests/day (sufficient for evaluation)
|
| 151 |
|
| 152 |
-
**HuggingFace Token
|
| 153 |
- Visit: https://huggingface.co/settings/tokens
|
| 154 |
-
- Click "New token" โ Name it (e.g., "TraceMind
|
| 155 |
- **Permissions**:
|
| 156 |
-
- Select "Read" for viewing datasets (sufficient for
|
| 157 |
-
- Select "Write" if you want to use synthetic dataset generation & push features (via MCP server)
|
| 158 |
-
- **Recommended**: Use "Write" permissions for full functionality
|
| 159 |
- **Free tier**: No rate limits for public dataset access
|
| 160 |
|
| 161 |
### Default Configuration (Without Your Keys)
|
|
@@ -172,6 +178,73 @@ If you don't configure your own keys:
|
|
| 172 |
โ
**Not exposed via API**: Settings forms use `api_name=False`
|
| 173 |
โ
**HTTPS encryption**: All API calls over secure connections
|
| 174 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 175 |
## Configuration
|
| 176 |
|
| 177 |
Create a `.env` file with the following variables:
|
|
|
|
| 81 |
## Quick Start
|
| 82 |
|
| 83 |
### Prerequisites
|
| 84 |
+
|
| 85 |
+
**For Viewing Leaderboard & Analysis:**
|
| 86 |
- Python 3.10+
|
| 87 |
- HuggingFace account (for authentication)
|
| 88 |
+
|
| 89 |
+
**For Submitting Evaluation Jobs:**
|
| 90 |
+
- โ ๏ธ **HuggingFace Pro account** ($9/month) with credit card
|
| 91 |
+
- HuggingFace token with **Read + Write + Run Jobs** permissions
|
| 92 |
+
- API keys for model providers (OpenAI, Anthropic, etc.)
|
| 93 |
+
|
| 94 |
+
> **Note**: Job submission requires a paid HuggingFace Pro account to access compute infrastructure. Viewing existing results is free.
|
| 95 |
|
| 96 |
### Installation
|
| 97 |
|
|
|
|
| 157 |
- Click "Get API Key" โ Create project โ Generate key
|
| 158 |
- **Free tier**: 1,500 requests/day (sufficient for evaluation)
|
| 159 |
|
| 160 |
+
**HuggingFace Token** (for viewing):
|
| 161 |
- Visit: https://huggingface.co/settings/tokens
|
| 162 |
+
- Click "New token" โ Name it (e.g., "TraceMind Viewer")
|
| 163 |
- **Permissions**:
|
| 164 |
+
- Select "Read" for viewing datasets (sufficient for browsing leaderboard)
|
|
|
|
|
|
|
| 165 |
- **Free tier**: No rate limits for public dataset access
|
| 166 |
|
| 167 |
### Default Configuration (Without Your Keys)
|
|
|
|
| 178 |
โ
**Not exposed via API**: Settings forms use `api_name=False`
|
| 179 |
โ
**HTTPS encryption**: All API calls over secure connections
|
| 180 |
|
| 181 |
+
## ๐ Submitting Evaluation Jobs
|
| 182 |
+
|
| 183 |
+
TraceMind-AI allows you to submit evaluation jobs directly from the UI to HuggingFace Jobs infrastructure.
|
| 184 |
+
|
| 185 |
+
### โ ๏ธ Requirements for Job Submission
|
| 186 |
+
|
| 187 |
+
**IMPORTANT**: To submit evaluation jobs, you need:
|
| 188 |
+
|
| 189 |
+
1. **HuggingFace Pro Account** ($9/month)
|
| 190 |
+
- Sign up at: https://huggingface.co/pricing
|
| 191 |
+
- **Credit card required** to pay for compute usage
|
| 192 |
+
- Free accounts cannot submit jobs
|
| 193 |
+
|
| 194 |
+
2. **HuggingFace Token with Enhanced Permissions**
|
| 195 |
+
- Visit: https://huggingface.co/settings/tokens
|
| 196 |
+
- Create token with these permissions:
|
| 197 |
+
- โ
**Read** (view datasets)
|
| 198 |
+
- โ
**Write** (upload results)
|
| 199 |
+
- โ
**Run Jobs** (submit evaluation jobs)
|
| 200 |
+
- โ ๏ธ Read-only tokens will NOT work
|
| 201 |
+
|
| 202 |
+
3. **Model Provider API Keys**
|
| 203 |
+
- OpenAI, Anthropic, Google, etc.
|
| 204 |
+
- Configure in Settings โ LLM Provider API Keys
|
| 205 |
+
- Passed securely as job secrets
|
| 206 |
+
|
| 207 |
+
### Hardware Options & Pricing
|
| 208 |
+
|
| 209 |
+
TraceMind auto-selects hardware based on your model:
|
| 210 |
+
|
| 211 |
+
- **cpu-basic**: API models (OpenAI, Anthropic) - ~$0.05/hr
|
| 212 |
+
- **t4-small**: Small models (4B-8B parameters) - ~$0.60/hr
|
| 213 |
+
- **a10g-small**: Medium models (7B-13B) - ~$1.10/hr
|
| 214 |
+
- **a100-large**: Large models (70B+) - ~$3.00/hr
|
| 215 |
+
|
| 216 |
+
Full pricing: https://huggingface.co/pricing#spaces-pricing
|
| 217 |
+
|
| 218 |
+
### How to Submit a Job
|
| 219 |
+
|
| 220 |
+
1. **Configure API Keys** (Settings tab):
|
| 221 |
+
- Add HF Token (with Run Jobs permission)
|
| 222 |
+
- Add Modal API credentials (optional, for Modal execution)
|
| 223 |
+
- Add LLM provider keys (OpenAI, Anthropic, etc.)
|
| 224 |
+
|
| 225 |
+
2. **Create Evaluation** (New Evaluation tab):
|
| 226 |
+
- Select infrastructure: HuggingFace Jobs or Modal
|
| 227 |
+
- Choose model and agent type
|
| 228 |
+
- Configure hardware (or use "auto")
|
| 229 |
+
- Set timeout (default: 1h)
|
| 230 |
+
- Click "Submit Evaluation"
|
| 231 |
+
|
| 232 |
+
3. **Monitor Job**:
|
| 233 |
+
- View job ID and status
|
| 234 |
+
- Track at: https://huggingface.co/jobs
|
| 235 |
+
- Results automatically appear in leaderboard when complete
|
| 236 |
+
|
| 237 |
+
### What Happens During a Job
|
| 238 |
+
|
| 239 |
+
1. Job starts on HuggingFace infrastructure
|
| 240 |
+
2. SMOLTRACE evaluates your model with OpenTelemetry tracing
|
| 241 |
+
3. Results uploaded to 4 HuggingFace datasets:
|
| 242 |
+
- Leaderboard entry (summary stats)
|
| 243 |
+
- Results dataset (test case details)
|
| 244 |
+
- Traces dataset (OTEL spans)
|
| 245 |
+
- Metrics dataset (GPU metrics, CO2 emissions)
|
| 246 |
+
4. Results appear in TraceMind leaderboard automatically
|
| 247 |
+
|
| 248 |
## Configuration
|
| 249 |
|
| 250 |
Create a `.env` file with the following variables:
|
|
@@ -984,7 +984,29 @@ SMOLTRACE automatically detects available resources:
|
|
| 984 |
|
| 985 |
## ๐๏ธ Integration with HuggingFace Jobs
|
| 986 |
|
| 987 |
-
SMOLTRACE works seamlessly with HuggingFace Jobs
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 988 |
|
| 989 |
```yaml
|
| 990 |
# job.yaml
|
|
@@ -1005,15 +1027,21 @@ command: |
|
|
| 1005 |
--leaderboard-repo huggingface/smolagents-leaderboard
|
| 1006 |
```
|
| 1007 |
|
| 1008 |
-
|
| 1009 |
-
|
| 1010 |
-
-
|
| 1011 |
-
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1012 |
|
| 1013 |
-
**Benefits:**
|
| 1014 |
- ๐ **Automatic Upload**: Results โ HuggingFace datasets
|
| 1015 |
- ๐ **Reproducible**: Same environment every time
|
| 1016 |
- โก **Optimized Compute**: Right hardware for your model size
|
|
|
|
| 1017 |
|
| 1018 |
---
|
| 1019 |
|
|
|
|
| 984 |
|
| 985 |
## ๐๏ธ Integration with HuggingFace Jobs
|
| 986 |
|
| 987 |
+
SMOLTRACE works seamlessly with HuggingFace Jobs for running evaluations on cloud infrastructure.
|
| 988 |
+
|
| 989 |
+
### โ ๏ธ Requirements to Submit Jobs
|
| 990 |
+
|
| 991 |
+
**IMPORTANT**: To submit jobs via TraceMind UI or HF CLI, you must:
|
| 992 |
+
|
| 993 |
+
1. **๐ HuggingFace Pro Account**
|
| 994 |
+
- You must be a HuggingFace Pro user
|
| 995 |
+
- **Credit card required** to pay for compute usage
|
| 996 |
+
- Sign up at: https://huggingface.co/pricing
|
| 997 |
+
|
| 998 |
+
2. **๐ซ HuggingFace Token Permissions**
|
| 999 |
+
- Your HF token needs **Read + Write** permissions
|
| 1000 |
+
- Token must have **"Run Jobs"** permission enabled
|
| 1001 |
+
- Create/update token at: https://huggingface.co/settings/tokens
|
| 1002 |
+
- โ ๏ธ Read-only tokens will **NOT** work for job submission
|
| 1003 |
+
|
| 1004 |
+
3. **๐ณ Billing**
|
| 1005 |
+
- You will be charged for compute usage
|
| 1006 |
+
- Pricing: https://huggingface.co/pricing#spaces-pricing
|
| 1007 |
+
- Monitor usage at: https://huggingface.co/settings/billing
|
| 1008 |
+
|
| 1009 |
+
### Example Job Configuration
|
| 1010 |
|
| 1011 |
```yaml
|
| 1012 |
# job.yaml
|
|
|
|
| 1027 |
--leaderboard-repo huggingface/smolagents-leaderboard
|
| 1028 |
```
|
| 1029 |
|
| 1030 |
+
### Hardware Selection
|
| 1031 |
+
|
| 1032 |
+
- ๐ง **cpu-basic**: API models (OpenAI, Anthropic via LiteLLM) - ~$0.05/hr
|
| 1033 |
+
- ๐ฎ **t4-small**: Small models (4B-8B) - ~$0.60/hr
|
| 1034 |
+
- ๐ง **a10g-small**: Medium models (7B-13B) - ~$1.10/hr
|
| 1035 |
+
- ๐ **a100-large**: Large models (70B+) - ~$3.00/hr
|
| 1036 |
+
|
| 1037 |
+
**Pricing**: See https://huggingface.co/pricing#spaces-pricing
|
| 1038 |
+
|
| 1039 |
+
### Benefits
|
| 1040 |
|
|
|
|
| 1041 |
- ๐ **Automatic Upload**: Results โ HuggingFace datasets
|
| 1042 |
- ๐ **Reproducible**: Same environment every time
|
| 1043 |
- โก **Optimized Compute**: Right hardware for your model size
|
| 1044 |
+
- ๐ฐ **Pay-per-use**: Only pay for actual compute time
|
| 1045 |
|
| 1046 |
---
|
| 1047 |
|
|
@@ -78,11 +78,11 @@ def create_settings_screen():
|
|
| 78 |
with gr.Row():
|
| 79 |
with gr.Column(scale=4):
|
| 80 |
hf_token = gr.Textbox(
|
| 81 |
-
label="HuggingFace Token",
|
| 82 |
placeholder="Enter your HF token (starts with 'hf_...')",
|
| 83 |
type="password",
|
| 84 |
value=os.environ.get("HF_TOKEN", ""),
|
| 85 |
-
info="
|
| 86 |
)
|
| 87 |
with gr.Column(scale=1):
|
| 88 |
hf_status = gr.Markdown("โช Not configured")
|
|
@@ -167,13 +167,22 @@ LITELLM_API_KEY=...""",
|
|
| 167 |
|
| 168 |
### HuggingFace Token
|
| 169 |
|
|
|
|
|
|
|
| 170 |
1. Go to [HuggingFace Settings](https://huggingface.co/settings/tokens)
|
| 171 |
2. Click "New token"
|
| 172 |
-
3. Give it a name (e.g., "TraceMind
|
| 173 |
-
4. Select
|
|
|
|
|
|
|
|
|
|
| 174 |
5. Create and copy the token (starts with `hf_...`)
|
| 175 |
|
| 176 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 177 |
|
| 178 |
---
|
| 179 |
|
|
|
|
| 78 |
with gr.Row():
|
| 79 |
with gr.Column(scale=4):
|
| 80 |
hf_token = gr.Textbox(
|
| 81 |
+
label="HuggingFace Token (Required for Job Submission)",
|
| 82 |
placeholder="Enter your HF token (starts with 'hf_...')",
|
| 83 |
type="password",
|
| 84 |
value=os.environ.get("HF_TOKEN", ""),
|
| 85 |
+
info="โ ๏ธ Token needs: Read + Write + Run Jobs permissions | Pro account required"
|
| 86 |
)
|
| 87 |
with gr.Column(scale=1):
|
| 88 |
hf_status = gr.Markdown("โช Not configured")
|
|
|
|
| 167 |
|
| 168 |
### HuggingFace Token
|
| 169 |
|
| 170 |
+
**For Job Submission (Required):**
|
| 171 |
+
|
| 172 |
1. Go to [HuggingFace Settings](https://huggingface.co/settings/tokens)
|
| 173 |
2. Click "New token"
|
| 174 |
+
3. Give it a name (e.g., "TraceMind Job Submission")
|
| 175 |
+
4. Select these permissions:
|
| 176 |
+
- โ
**Read** (view datasets)
|
| 177 |
+
- โ
**Write** (upload results)
|
| 178 |
+
- โ
**Run Jobs** (submit evaluation jobs)
|
| 179 |
5. Create and copy the token (starts with `hf_...`)
|
| 180 |
|
| 181 |
+
**โ ๏ธ IMPORTANT Requirements:**
|
| 182 |
+
- You must have a **HuggingFace Pro account** ($9/month)
|
| 183 |
+
- **Credit card required** to pay for compute usage
|
| 184 |
+
- Read-only tokens will NOT work for job submission
|
| 185 |
+
- Sign up for Pro: https://huggingface.co/pricing
|
| 186 |
|
| 187 |
---
|
| 188 |
|