kshitijthakkar commited on
Commit
4449927
ยท
1 Parent(s): 4db4e9d

docs: Add comprehensive job submission requirements

Browse files

Critical Updates:
- Added HuggingFace Pro account requirement (/month with credit card)
- Clarified HF token needs Read + Write + Run Jobs permissions
- Updated all documentation to warn users about requirements

Documentation Screen (SMOLTRACE tab):
- Added prominent Requirements section before job configuration
- Listed HF Pro account requirement with pricing link
- Specified token permissions needed (Read + Write + Run Jobs)
- Added hardware pricing estimates (~/usr/bin/bash.05/hr to ~.00/hr)
- Warned that read-only tokens will NOT work

Settings Screen:
- Updated HF token label to indicate job submission requirement
- Added warning in info text about Pro account requirement
- Expanded HF token documentation with required permissions checklist
- Added links to HF pricing and billing pages

README:
- Split Prerequisites into viewing vs job submission
- Added Requirements for Job Submission section with clear warnings
- Added hardware options and pricing information
- Added step-by-step guide for submitting jobs
- Clarified what happens during job execution
- Updated HF token guidance for both viewing and job submission

Why These Changes:
- Users were not aware HF Pro account is required for jobs
- Many users have read-only tokens which fail silently
- Credit card requirement was not mentioned
- Pricing information was missing, users couldn't estimate costs

Impact:
- Sets clear expectations before users attempt job submission
- Prevents frustration from failed submissions due to permissions
- Helps users budget for compute costs
- Improves transparency about platform requirements

Files changed (3) hide show
  1. README.md +79 -6
  2. screens/documentation.py +34 -6
  3. screens/settings.py +14 -5
README.md CHANGED
@@ -81,9 +81,17 @@ TraceMind demonstrates enterprise MCP client usage by connecting to [TraceMind-m
81
  ## Quick Start
82
 
83
  ### Prerequisites
 
 
84
  - Python 3.10+
85
  - HuggingFace account (for authentication)
86
- - HuggingFace token (optional, for private datasets)
 
 
 
 
 
 
87
 
88
  ### Installation
89
 
@@ -149,13 +157,11 @@ Both APIs have generous free tiers:
149
  - Click "Get API Key" โ†’ Create project โ†’ Generate key
150
  - **Free tier**: 1,500 requests/day (sufficient for evaluation)
151
 
152
- **HuggingFace Token**:
153
  - Visit: https://huggingface.co/settings/tokens
154
- - Click "New token" โ†’ Name it (e.g., "TraceMind Access")
155
  - **Permissions**:
156
- - Select "Read" for viewing datasets (sufficient for most features)
157
- - Select "Write" if you want to use synthetic dataset generation & push features (via MCP server)
158
- - **Recommended**: Use "Write" permissions for full functionality
159
  - **Free tier**: No rate limits for public dataset access
160
 
161
  ### Default Configuration (Without Your Keys)
@@ -172,6 +178,73 @@ If you don't configure your own keys:
172
  โœ… **Not exposed via API**: Settings forms use `api_name=False`
173
  โœ… **HTTPS encryption**: All API calls over secure connections
174
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
175
  ## Configuration
176
 
177
  Create a `.env` file with the following variables:
 
81
  ## Quick Start
82
 
83
  ### Prerequisites
84
+
85
+ **For Viewing Leaderboard & Analysis:**
86
  - Python 3.10+
87
  - HuggingFace account (for authentication)
88
+
89
+ **For Submitting Evaluation Jobs:**
90
+ - โš ๏ธ **HuggingFace Pro account** ($9/month) with credit card
91
+ - HuggingFace token with **Read + Write + Run Jobs** permissions
92
+ - API keys for model providers (OpenAI, Anthropic, etc.)
93
+
94
+ > **Note**: Job submission requires a paid HuggingFace Pro account to access compute infrastructure. Viewing existing results is free.
95
 
96
  ### Installation
97
 
 
157
  - Click "Get API Key" โ†’ Create project โ†’ Generate key
158
  - **Free tier**: 1,500 requests/day (sufficient for evaluation)
159
 
160
+ **HuggingFace Token** (for viewing):
161
  - Visit: https://huggingface.co/settings/tokens
162
+ - Click "New token" โ†’ Name it (e.g., "TraceMind Viewer")
163
  - **Permissions**:
164
+ - Select "Read" for viewing datasets (sufficient for browsing leaderboard)
 
 
165
  - **Free tier**: No rate limits for public dataset access
166
 
167
  ### Default Configuration (Without Your Keys)
 
178
  โœ… **Not exposed via API**: Settings forms use `api_name=False`
179
  โœ… **HTTPS encryption**: All API calls over secure connections
180
 
181
+ ## ๐Ÿš€ Submitting Evaluation Jobs
182
+
183
+ TraceMind-AI allows you to submit evaluation jobs directly from the UI to HuggingFace Jobs infrastructure.
184
+
185
+ ### โš ๏ธ Requirements for Job Submission
186
+
187
+ **IMPORTANT**: To submit evaluation jobs, you need:
188
+
189
+ 1. **HuggingFace Pro Account** ($9/month)
190
+ - Sign up at: https://huggingface.co/pricing
191
+ - **Credit card required** to pay for compute usage
192
+ - Free accounts cannot submit jobs
193
+
194
+ 2. **HuggingFace Token with Enhanced Permissions**
195
+ - Visit: https://huggingface.co/settings/tokens
196
+ - Create token with these permissions:
197
+ - โœ… **Read** (view datasets)
198
+ - โœ… **Write** (upload results)
199
+ - โœ… **Run Jobs** (submit evaluation jobs)
200
+ - โš ๏ธ Read-only tokens will NOT work
201
+
202
+ 3. **Model Provider API Keys**
203
+ - OpenAI, Anthropic, Google, etc.
204
+ - Configure in Settings โ†’ LLM Provider API Keys
205
+ - Passed securely as job secrets
206
+
207
+ ### Hardware Options & Pricing
208
+
209
+ TraceMind auto-selects hardware based on your model:
210
+
211
+ - **cpu-basic**: API models (OpenAI, Anthropic) - ~$0.05/hr
212
+ - **t4-small**: Small models (4B-8B parameters) - ~$0.60/hr
213
+ - **a10g-small**: Medium models (7B-13B) - ~$1.10/hr
214
+ - **a100-large**: Large models (70B+) - ~$3.00/hr
215
+
216
+ Full pricing: https://huggingface.co/pricing#spaces-pricing
217
+
218
+ ### How to Submit a Job
219
+
220
+ 1. **Configure API Keys** (Settings tab):
221
+ - Add HF Token (with Run Jobs permission)
222
+ - Add Modal API credentials (optional, for Modal execution)
223
+ - Add LLM provider keys (OpenAI, Anthropic, etc.)
224
+
225
+ 2. **Create Evaluation** (New Evaluation tab):
226
+ - Select infrastructure: HuggingFace Jobs or Modal
227
+ - Choose model and agent type
228
+ - Configure hardware (or use "auto")
229
+ - Set timeout (default: 1h)
230
+ - Click "Submit Evaluation"
231
+
232
+ 3. **Monitor Job**:
233
+ - View job ID and status
234
+ - Track at: https://huggingface.co/jobs
235
+ - Results automatically appear in leaderboard when complete
236
+
237
+ ### What Happens During a Job
238
+
239
+ 1. Job starts on HuggingFace infrastructure
240
+ 2. SMOLTRACE evaluates your model with OpenTelemetry tracing
241
+ 3. Results uploaded to 4 HuggingFace datasets:
242
+ - Leaderboard entry (summary stats)
243
+ - Results dataset (test case details)
244
+ - Traces dataset (OTEL spans)
245
+ - Metrics dataset (GPU metrics, CO2 emissions)
246
+ 4. Results appear in TraceMind leaderboard automatically
247
+
248
  ## Configuration
249
 
250
  Create a `.env` file with the following variables:
screens/documentation.py CHANGED
@@ -984,7 +984,29 @@ SMOLTRACE automatically detects available resources:
984
 
985
  ## ๐Ÿ—๏ธ Integration with HuggingFace Jobs
986
 
987
- SMOLTRACE works seamlessly with HuggingFace Jobs:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
988
 
989
  ```yaml
990
  # job.yaml
@@ -1005,15 +1027,21 @@ command: |
1005
  --leaderboard-repo huggingface/smolagents-leaderboard
1006
  ```
1007
 
1008
- **Hardware Selection:**
1009
- - ๐Ÿ”ง **gpu-a10**: Perfect for 7B-13B models (cost-effective)
1010
- - ๐Ÿš€ **gpu-h200**: Use for 70B+ models (high performance)
1011
- - ๐Ÿ’ป **cpu-basic**: API models (OpenAI, Anthropic via LiteLLM)
 
 
 
 
 
 
1012
 
1013
- **Benefits:**
1014
  - ๐Ÿ“Š **Automatic Upload**: Results โ†’ HuggingFace datasets
1015
  - ๐Ÿ”„ **Reproducible**: Same environment every time
1016
  - โšก **Optimized Compute**: Right hardware for your model size
 
1017
 
1018
  ---
1019
 
 
984
 
985
  ## ๐Ÿ—๏ธ Integration with HuggingFace Jobs
986
 
987
+ SMOLTRACE works seamlessly with HuggingFace Jobs for running evaluations on cloud infrastructure.
988
+
989
+ ### โš ๏ธ Requirements to Submit Jobs
990
+
991
+ **IMPORTANT**: To submit jobs via TraceMind UI or HF CLI, you must:
992
+
993
+ 1. **๐Ÿ”‘ HuggingFace Pro Account**
994
+ - You must be a HuggingFace Pro user
995
+ - **Credit card required** to pay for compute usage
996
+ - Sign up at: https://huggingface.co/pricing
997
+
998
+ 2. **๐ŸŽซ HuggingFace Token Permissions**
999
+ - Your HF token needs **Read + Write** permissions
1000
+ - Token must have **"Run Jobs"** permission enabled
1001
+ - Create/update token at: https://huggingface.co/settings/tokens
1002
+ - โš ๏ธ Read-only tokens will **NOT** work for job submission
1003
+
1004
+ 3. **๐Ÿ’ณ Billing**
1005
+ - You will be charged for compute usage
1006
+ - Pricing: https://huggingface.co/pricing#spaces-pricing
1007
+ - Monitor usage at: https://huggingface.co/settings/billing
1008
+
1009
+ ### Example Job Configuration
1010
 
1011
  ```yaml
1012
  # job.yaml
 
1027
  --leaderboard-repo huggingface/smolagents-leaderboard
1028
  ```
1029
 
1030
+ ### Hardware Selection
1031
+
1032
+ - ๐Ÿ”ง **cpu-basic**: API models (OpenAI, Anthropic via LiteLLM) - ~$0.05/hr
1033
+ - ๐ŸŽฎ **t4-small**: Small models (4B-8B) - ~$0.60/hr
1034
+ - ๐Ÿ”ง **a10g-small**: Medium models (7B-13B) - ~$1.10/hr
1035
+ - ๐Ÿš€ **a100-large**: Large models (70B+) - ~$3.00/hr
1036
+
1037
+ **Pricing**: See https://huggingface.co/pricing#spaces-pricing
1038
+
1039
+ ### Benefits
1040
 
 
1041
  - ๐Ÿ“Š **Automatic Upload**: Results โ†’ HuggingFace datasets
1042
  - ๐Ÿ”„ **Reproducible**: Same environment every time
1043
  - โšก **Optimized Compute**: Right hardware for your model size
1044
+ - ๐Ÿ’ฐ **Pay-per-use**: Only pay for actual compute time
1045
 
1046
  ---
1047
 
screens/settings.py CHANGED
@@ -78,11 +78,11 @@ def create_settings_screen():
78
  with gr.Row():
79
  with gr.Column(scale=4):
80
  hf_token = gr.Textbox(
81
- label="HuggingFace Token",
82
  placeholder="Enter your HF token (starts with 'hf_...')",
83
  type="password",
84
  value=os.environ.get("HF_TOKEN", ""),
85
- info="Get your token at: https://huggingface.co/settings/tokens"
86
  )
87
  with gr.Column(scale=1):
88
  hf_status = gr.Markdown("โšช Not configured")
@@ -167,13 +167,22 @@ LITELLM_API_KEY=...""",
167
 
168
  ### HuggingFace Token
169
 
 
 
170
  1. Go to [HuggingFace Settings](https://huggingface.co/settings/tokens)
171
  2. Click "New token"
172
- 3. Give it a name (e.g., "TraceMind Access")
173
- 4. Select "Read" permissions
 
 
 
174
  5. Create and copy the token (starts with `hf_...`)
175
 
176
- **Note**: Read-only access is sufficient for viewing datasets
 
 
 
 
177
 
178
  ---
179
 
 
78
  with gr.Row():
79
  with gr.Column(scale=4):
80
  hf_token = gr.Textbox(
81
+ label="HuggingFace Token (Required for Job Submission)",
82
  placeholder="Enter your HF token (starts with 'hf_...')",
83
  type="password",
84
  value=os.environ.get("HF_TOKEN", ""),
85
+ info="โš ๏ธ Token needs: Read + Write + Run Jobs permissions | Pro account required"
86
  )
87
  with gr.Column(scale=1):
88
  hf_status = gr.Markdown("โšช Not configured")
 
167
 
168
  ### HuggingFace Token
169
 
170
+ **For Job Submission (Required):**
171
+
172
  1. Go to [HuggingFace Settings](https://huggingface.co/settings/tokens)
173
  2. Click "New token"
174
+ 3. Give it a name (e.g., "TraceMind Job Submission")
175
+ 4. Select these permissions:
176
+ - โœ… **Read** (view datasets)
177
+ - โœ… **Write** (upload results)
178
+ - โœ… **Run Jobs** (submit evaluation jobs)
179
  5. Create and copy the token (starts with `hf_...`)
180
 
181
+ **โš ๏ธ IMPORTANT Requirements:**
182
+ - You must have a **HuggingFace Pro account** ($9/month)
183
+ - **Credit card required** to pay for compute usage
184
+ - Read-only tokens will NOT work for job submission
185
+ - Sign up for Pro: https://huggingface.co/pricing
186
 
187
  ---
188