Upload folder using huggingface_hub

.gitignore

@@ -0,0 +1,3 @@
+__pycache__/
+venv/
+.env

.gradio/certificate.pem

@@ -0,0 +1,31 @@
+-----BEGIN CERTIFICATE-----
+MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+-----END CERTIFICATE-----

DAG_VISUALIZATION_README.md

@@ -0,0 +1,235 @@
+# DAG Visualization for QA_LLM_Module
+
+## Overview
+
+This enhancement adds DAG (Directed Acyclic Graph) visualization capability to the QA_LLM_Module, allowing users to see visual representations of robot task dependencies generated by the LLM.
+
+## Features
+
+- **Automatic DAG Generation**: Visualizations are generated automatically when the LLM outputs valid task JSON
+- **IEEE-Style Formatting**: Professional, publication-ready graphs
+- **Hierarchical Layout**: Tasks are organized by dependency levels
+- **Color-Coded Nodes**:
+  - 🔴 Red: Start tasks (no dependencies)
+  - 🟣 Purple: End tasks (no dependents)
+  - 🟠 Orange: Intermediate tasks
+- **Detailed Information**: Each task shows function name, assigned robots, and object keywords
+- **Two Visualization Modes**: Detailed and simplified versions
+- **🔒 Safety Confirmation Workflow**: 
+  - Task plans require operator approval before deployment
+  - "Validate & Deploy Task Plan" button for safety compliance
+  - Visual status updates (Pending → Approved & Deployed)
+  - Prevents accidental execution of unverified plans
+
+## Installation
+
+The required dependencies are already added to `requirements.txt`:
+
+```bash
+pip install matplotlib==3.9.4
+pip install networkx==3.4
+```
+
+## Usage
+
+### Automatic Integration with Safety Workflow
+
+When you run the main application and input a query that generates task JSON:
+
+1. **DAG Generation**: The visualization appears automatically with "Pending Approval" status
+2. **Safety Review**: Operator reviews the task dependency graph
+3. **Confirmation**: Click "🔒 Validate & Deploy Task Plan" to send to construction site
+4. **Deployment**: Graph updates to "APPROVED & DEPLOYED" status
+
+```bash
+python3 main.py
+```
+
+### Enhanced Safety Workflow Steps
+
+1. 🤖 **LLM generates task plan** → DAG shows "Pending Approval"
+2. 👨‍💼 **Operator reviews visualization** → Confirms task safety and correctness
+3. 📝 **[Optional] Edit Task Plan** → Manual JSON editing if modifications needed
+4. 🔄 **[Optional] Update DAG Visualization** → Regenerate graph with edits
+5. 🔒 **Validate & Deploy Task Plan** → Final approval and deployment to robots
+6. ✅ **Confirmation displayed** → DAG shows "APPROVED & DEPLOYED"
+
+### Three-Button IEEE-Style Workflow
+
+- **📝 Edit Task Plan**: Opens JSON editor for manual modifications
+- **🔄 Update DAG Visualization**: Regenerates graph from edited JSON
+- **🔒 Validate & Deploy Task Plan**: Final safety approval and deployment
+
+### Manual Testing
+
+You can test the DAG visualization independently:
+
+```bash
+python3 test_dag_visualization.py
+python3 dag_demo.py
+```
+
+## API Reference
+
+### DAGVisualizer Class
+
+```python
+from dag_visualizer import DAGVisualizer
+
+visualizer = DAGVisualizer()
+```
+
+#### Methods
+
+##### `create_dag_visualization(task_data, title="Robot Task Dependency Graph")`
+Creates a detailed DAG visualization with full task information.
+
+**Parameters:**
+- `task_data` (dict): Task data in the expected JSON format
+- `title` (str): Title for the graph
+
+**Returns:**
+- `str`: Path to the generated PNG image, or `None` if failed
+
+##### `create_simplified_dag_visualization(task_data, title="Robot Task Graph")`
+Creates a simplified DAG visualization suitable for smaller displays.
+
+**Parameters:**
+- `task_data` (dict): Task data in the expected JSON format
+- `title` (str): Title for the graph
+
+**Returns:**
+- `str`: Path to the generated PNG image, or `None` if failed
+
+## Expected JSON Format
+
+The visualizer expects task data in the following format:
+
+```json
+{
+  "tasks": [
+    {
+      "task": "task_name",
+      "instruction_function": {
+        "name": "function_name",
+        "robot_ids": ["robot_id_1", "robot_id_2"],
+        "dependencies": ["dependency_task_name"],
+        "object_keywords": ["object1", "object2"]
+      }
+    }
+  ]
+}
+```
+
+## Examples
+
+### Single Task
+```json
+{
+  "tasks": [
+    {
+      "task": "target_area_for_specific_robots_1",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+```
+
+### Multiple Tasks with Dependencies
+```json
+{
+  "tasks": [
+    {
+      "task": "excavate_soil",
+      "instruction_function": {
+        "name": "excavate_soil_from_pile",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "object_keywords": ["soil_pile"]
+      }
+    },
+    {
+      "task": "transport_soil",
+      "instruction_function": {
+        "name": "transport_soil_to_pit",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": ["excavate_soil"],
+        "object_keywords": ["pit"]
+      }
+    }
+  ]
+}
+```
+
+## File Structure
+
+```
+QA_LLM_Module/
+├── dag_visualizer.py          # Main visualization module
+├── test_dag_visualization.py  # Test suite
+├── dag_demo.py               # Demo script
+├── gradio_llm_interface.py   # Updated with DAG integration
+├── main.py                   # Updated Gradio interface
+└── requirements.txt          # Updated dependencies
+```
+
+## Integration Details
+
+### Modified Files
+
+1. **gradio_llm_interface.py**:
+   - Added `DAGVisualizer` import
+   - Modified `predict()` method to generate visualizations
+   - Returns additional DAG image output
+
+2. **main.py**:
+   - Added `gr.Image` component for DAG display
+   - Updated input/output mappings for DAG visualization
+
+3. **requirements.txt**:
+   - Added `matplotlib==3.9.4`
+   - Added `networkx==3.4`
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"No module named 'matplotlib'"**
+   - Solution: `pip install matplotlib networkx`
+
+2. **"No tasks found or invalid graph structure"**
+   - Solution: Ensure your JSON follows the expected format
+
+3. **Images not displaying in Gradio**
+   - Solution: Check that the image paths are accessible and files exist
+
+### Debug Mode
+
+Enable detailed logging by setting the log level:
+
+```python
+from loguru import logger
+logger.add("dag_debug.log", level="DEBUG")
+```
+
+## Performance Notes
+
+- Visualizations are generated in temporary directories
+- Images are automatically cleaned up by the system
+- Large graphs (>20 nodes) may take longer to render
+- Memory usage scales with graph complexity
+
+## Future Enhancements
+
+Potential improvements for future versions:
+
+1. **Interactive Graphs**: Clickable nodes with detailed information
+2. **Real-time Updates**: Live visualization updates as tasks execute
+3. **Export Options**: Support for PDF, SVG, and other formats
+4. **Custom Styling**: User-configurable colors and layouts
+5. **Animation**: Animated execution flow visualization

EDITING_WORKFLOW_GUIDE.md

@@ -0,0 +1,136 @@
+# 📝 Enhanced Editing Workflow Guide
+
+## 🔧 Fixed Issues
+
+### ✅ Persistent Editing Problem Solved
+- **Issue**: After updating DAG, editor showed empty `{"tasks": []}` on subsequent edits
+- **Solution**: Enhanced state management to preserve task plans across edit cycles
+- **Result**: Task plans now persist through the complete workflow
+
+## 🚀 Complete Workflow
+
+### 1️⃣ **LLM Generation Phase**
+```
+User Input → LLM Response → DAG Generated → Buttons Appear
+```
+**Buttons shown**: `📝 Edit Task Plan` + `🔒 Validate & Deploy Task Plan`
+
+### 2️⃣ **Manual Editing Phase** (Optional)
+```
+Click "📝 Edit Task Plan" → JSON Editor Opens → Make Changes
+```
+**Buttons shown**: `🔄 Update DAG Visualization`
+
+### 3️⃣ **DAG Update Phase**
+```
+Click "🔄 Update DAG Visualization" → New DAG Generated → Review Changes
+```
+**Buttons shown**: `🔒 Validate & Deploy Task Plan`
+
+### 4️⃣ **Deployment Phase**
+```
+Click "🔒 Validate & Deploy Task Plan" → Send to ROS Topics → Confirmation
+```
+**Status**: `✅ Task Plan Successfully Deployed`
+
+### 5️⃣ **Re-editing Capability**
+```
+Click "📝 Edit Task Plan" Again → Previous Plan Loads → Continue Editing
+```
+**Feature**: Deployed plans can be re-edited and updated
+
+## 🎯 Key Features
+
+### 🔄 **Persistent State Management**
+- ✅ Task plans persist across edit cycles
+- ✅ Deployed plans remain available for re-editing
+- ✅ Intelligent fallback to templates when needed
+- ✅ Proper error handling for malformed data
+
+### 📋 **Smart Editor Behavior**
+- **With existing plan**: Shows current task JSON for editing
+- **After deployment**: Shows deployed plan for potential re-editing
+- **Empty state**: Shows example template with proper structure
+- **Malformed data**: Gracefully falls back to template
+
+### 🛡️ **Error Handling**
+- **JSON Syntax Errors**: Shows clear error messages and keeps editor open
+- **Missing Fields**: Validates required structure (`tasks` field)
+- **Empty Tasks**: Handles empty task arrays gracefully
+- **State Corruption**: Recovers with template when state is invalid
+
+## 📊 Example JSON Structure
+
+```json
+{
+  "tasks": [
+    {
+      "task": "excavate_soil_from_pile",
+      "instruction_function": {
+        "name": "excavate_soil_from_pile",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "object_keywords": ["soil_pile"]
+      }
+    },
+    {
+      "task": "transport_soil_to_pit",
+      "instruction_function": {
+        "name": "transport_soil_to_pit",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": ["excavate_soil_from_pile"],
+        "object_keywords": ["pit"]
+      }
+    }
+  ]
+}
+```
+
+## 🔍 Testing Commands
+
+### Run the Application
+```bash
+cd /root/share/QA_LLM_Module
+python3 main.py
+```
+
+### Monitor ROS Topics
+```bash
+./monitor_topics.sh
+```
+
+### Test Editing Workflow
+```bash
+python3 test_persistent_editing.py
+```
+
+## 🎉 Success Indicators
+
+### ✅ Working Correctly When:
+1. **First Edit**: LLM plan loads in editor
+2. **Update DAG**: New visualization appears
+3. **Deploy**: Confirmation message shows
+4. **Re-edit**: Previous plan content loads (not empty template)
+5. **Multiple Cycles**: Can edit → update → deploy repeatedly
+
+### 🔧 Troubleshooting
+
+#### Problem: Editor shows empty tasks
+- **Cause**: State management issue
+- **Solution**: Fixed with enhanced persistence logic
+
+#### Problem: DAG not updating
+- **Cause**: JSON syntax error
+- **Solution**: Check error message, fix JSON, try again
+
+#### Problem: Cannot deploy
+- **Cause**: ROS node not initialized
+- **Solution**: Restart application, check ROS connection
+
+## 🏆 Benefits
+
+1. **Enhanced Safety**: Multiple review opportunities before deployment
+2. **Flexibility**: Manual fine-tuning of LLM-generated plans
+3. **Persistence**: No loss of work during editing cycles
+4. **User-Friendly**: Clear status messages and error handling
+5. **IEEE Compliant**: Professional terminology and workflow

config.py

@@ -0,0 +1,459 @@
+# Description of robots
+DESCRIPTION_ROBOT = {
+    "excavator": {'width': 100, 'height': 100, 'functions': ["Excavation", "Unloading"]},
+    "dump_truck": {'width': 50, 'height': 100, 'functions': ["Loading", "Unloading"]},
+}
+
+# Navigation functions
+NAVIGATION_FUNCTIONS = [
+    "avoid_areas_for_all_robots",
+    "avoid_areas_for_specific_robots",
+    "target_area_for_all_robots",
+    "target_area_for_specific_robots",
+    "allow_areas_for_all_robots",
+    "allow_areas_for_specific_robots",
+    "return_to_start_for_all_robots",
+    "return_to_start_for_specific_robots"
+]
+
+ROBOT_SPECIFIC_FUNCTIONS = [
+    "Excavation", 
+    "ExcavatorUnloading", 
+    "DumpUnloading", 
+    "DumpLoading"
+]
+
+
+# Robot names
+ROBOT_NAMES = {
+    "robot_dump_truck_01": {"id": "robot_dump_truck_01", "type": "dump_truck"},
+    "robot_dump_truck_02": {"id": "robot_dump_truck_02", "type": "dump_truck"},
+    # "robot_dump_truck_03": {"id": "robot_dump_truck_03", "type": "dump_truck"},
+    # "robot_dump_truck_04": {"id": "robot_dump_truck_04", "type": "dump_truck"},
+    # "robot_dump_truck_05": {"id": "robot_dump_truck_05", "type": "dump_truck"},
+    # "robot_dump_truck_06": {"id": "robot_dump_truck_06", "type": "dump_truck"},
+    "robot_excavator_01": {"id": "robot_excavator_01", "type": "excavator"},
+    # "robot_excavator_02": {"id": "robot_excavator_02", "type": "excavator"},
+}
+
+# Get robot functions
+def get_robot_functions(robot_name):
+    robot_type = ROBOT_NAMES[robot_name]["type"]
+    specific_functions = DESCRIPTION_ROBOT[robot_type]["functions"]
+    return NAVIGATION_FUNCTIONS + specific_functions
+
+# Get all functions description for planner mode
+def get_all_functions_description():
+    description = "Here are the common navigation functions for all robots:\n\n"
+    description += ", ".join(NAVIGATION_FUNCTIONS) + "\n\n"
+    description += "Here are the robots in the system and their specific functions:\n\n"
+    for robot_name, robot_info in ROBOTS_CONFIG['robot_names'].items():
+        specific_functions = [func for func in ROBOTS_CONFIG["get_robot_functions"](robot_name) if func not in NAVIGATION_FUNCTIONS]
+        functions = ", ".join(specific_functions)
+        description += f"- {robot_name}: {robot_info['type']} ({functions})\n"
+    return description
+
+# Robots configuration
+ROBOTS_CONFIG = {
+    "description_robot": DESCRIPTION_ROBOT,
+    "robot_names": ROBOT_NAMES,
+    "get_robot_functions": get_robot_functions,
+    "get_all_functions_description": get_all_functions_description
+}
+
+# Model configuration
+MODEL_CONFIG = {
+    "model_options": ["gpt-4o", "gpt-3.5-turbo", "gpt-4-turbo", "claude-3-haiku-20240307", "claude-3-5-sonnet-20240620", "claude-3-opus-20240229", "llama-3.3-70b-versatile", "llama-3.1-8b-instant", "llama3.3:70b-instruct-q4_K_M", "llama3.1:8b"],
+    "default_model": "gpt-4o",
+    "model_type": "openai",
+    "provider": "openai",
+    "max_tokens": 2048,
+    "temperature": 0,
+    "frequency_penalty": 0,
+}
+
+# Flag to determine whether confirmation is required
+CONFIRMATION_REQUIRED = False  # Set to False if confirmation is not required
+
+# Initial messages configuration
+if CONFIRMATION_REQUIRED:
+    INITIAL_MESSAGES_CONFIG = {
+        "system": (
+            "You are a confident and pattern-following assistant that assists the operator in managing multiple construction robots on a construction site. "
+            "In regular conversations, do not generate or send JSON commands. "
+            "When it becomes necessary to control the robot swarm, generate the appropriate JSON command but do not send it immediately. "
+            "You must ask the operator for confirmation before sending the JSON command. "
+            "When asking for confirmation, provide a detailed summary of the command's purpose and expected actions, but **do not include the actual JSON code**. "
+            "Use the following format for confirmation: "
+            "'I am ready to send a command to [target] on the construction site. The command will [brief description of action]. "
+            "Key details: [list important parameters or actions]. Do you agree to proceed with this command?' "
+            "It is crucial that you follow this instruction strictly to avoid including any JSON in the confirmation message, "
+            "while still providing a clear and detailed description of the command's intent and effects. "
+            "After receiving confirmation (e.g., 'yes', 'proceed', 'agreed'), immediately send the JSON command without further questions or explanations. "
+            "If the confirmation is negative or unclear, ask for clarification or await further instructions without sending the command."
+        ),
+        "user_intro": {
+            "default": (
+                "I would like you to assist in managing multiple construction robots on a construction site. "
+                "In most cases, you should engage in regular conversation without generating or sending JSON commands. "
+                "However, when it becomes necessary to control the robot swarm, you should write JSON to do so, but only after confirming with me. "
+                "When confirming, **do not include the JSON code in your confirmation response under any circumstances**. "
+                "Instead, provide a detailed summary of the command's purpose and expected actions using the following format: "
+                "'I am ready to send a command to [target] on the construction site. The command will [brief description of action]. "
+                "Key details: [list important parameters or actions]. Do you agree to proceed with this command?' "
+                "It is essential to adhere to this format, providing a clear description of the command's intent and effects, "
+                "while avoiding the inclusion of any JSON in the confirmation message. "
+                "Once I confirm (e.g., by saying 'yes', 'proceed', or 'agreed'), immediately send the JSON command without asking any more questions. "
+                "If I don't confirm or my response is unclear, ask for clarification or wait for further instructions. "
+                "Pay attention to patterns that appear in the given context code. "
+                "Be thorough and thoughtful in your JSON. Do not include any import statements. Do not repeat my question. Do not provide any text explanation. "
+                "Note that x is back to front, y is left to right, and z is bottom to up.\n\n"
+                "Only use functions from the following library:\n\n\n{library}\n\n\n"
+                "Consider the following environmental objects in the scene:\n\n\n```{env_objects}```\n\n\n"
+                "The available robots on the construction site are:\n\n\n```{robot_names}```\n\n\n"
+                "Here are some examples of how to format the JSON based on previous queries:\n\n\n```{fewshot_examples}```\n\n\n"
+            ),
+            "task_2_commannd_prompt": "You are working on decomposing tasks for the robots.",
+            "dart": "You are working on decomposing tasks for the robots.",
+            "task_decomposer": "You are working on decomposing tasks for the robots.",
+            "composer": "You are composing high-level tasks for the robots.",
+            "instruction_translator": "You are translating instructions for the robots.",
+            "planner": "You are planning tasks for the robots. Please use both navigation functions and robot-specific functions from the following list:\n\n{functions_description}\n\n"
+        },
+        "assistant": (
+            "Understood. I will generate the JSON and seek your confirmation by providing a detailed summary of the command's purpose and expected actions, without including the actual JSON code. "
+            "I will use the format: 'I am ready to send a command to [target] on the construction site. The command will [brief description of action]. Key details: [list important parameters or actions]. Do you agree to proceed with this command?' "
+            "This confirmation message will not include any JSON code but will give you a clear understanding of the command's intent and effects. "
+            "Once you confirm with a positive response like 'yes', 'proceed', or 'agreed', I will immediately send the JSON command without asking any further questions. "
+            "If your response is negative or unclear, I will seek clarification or await further instructions before proceeding."
+        )
+    }
+else:
+    INITIAL_MESSAGES_CONFIG = {
+        "system": (
+            "You are a confident and pattern-following assistant that pays attention to the user's instructions and writes good JSON for controlling multiple construction robots in a construction site. "
+            "Please ensure that the JSON code is properly formatted with consistent indentation and alignment for better readability and ease of use when copying."
+
+        ),
+        "user_intro": {
+            "default": (
+                "I would like you to help me write JSON to control multiple construction robots in a construction site. "
+                "Please complete the JSON code every time when I give you a new query. Pay attention to patterns that appear in the given context code. "
+                "Be thorough and thoughtful in your JSON. Do not include any import statement. Do not repeat my question. Do not provide any text explanation. "
+                "Note that x is back to front, y is left to right, and z is bottom to up.\n\n"
+                "Only use functions from the following library:\n\n\n{library}\n\n\n"
+                "Consider the following environmental objects in the scene:\n\n\n```{env_objects}```\n\n\n"
+                "The available robots in the construction site are:\n\n\n```{robot_names}```\n\n\n"
+                "Here are some examples of how to format the JSON based on previous queries:\n\n\n```{fewshot_examples}```\n\n\n"
+            ),
+            "task_2_commannd_prompt": "You are working on decomposing tasks for the robots.",
+            "dart": "You are working on decomposing tasks for the robots.",
+            "task_decomposer": "You are working on decomposing tasks for the robots.",
+            "composer": "You are composing high-level tasks for the robots.",
+            "instruction_translator": "You are translating instructions for the robots.",
+            "planner": "You are planning tasks for the robots. Please use both navigation functions and robot-specific functions from the following list:\n\n{functions_description}\n\n"
+        },
+        "assistant": "Got it. I will complete the JSON you provide next."
+    }
+
+
+# Mode configuration
+MODE_CONFIG = {
+    "task_2_commannd_prompt": {
+        "display_name": "Task to Command",
+        "prompt_file": "./prompts/swarm/task_2_commannd_prompt.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "gpt-4o",
+        # "model_version": "claude-3-haiku-20240307",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_gpt_4o": {
+        "display_name": "gpt-4o",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "gpt-4o",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_gpt_4_turbo": {
+        "display_name": "gpt-4-turbo",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "gpt-4-turbo",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_gpt_3_5_turbo": {
+        "display_name": "gpt-3.5-turbo",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "gpt-3.5-turbo",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_claude_3_haiku": {
+        "display_name": "claude-3-haiku",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "claude-3-haiku-20240307",
+        "provider": "anthropic",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_claude_3_sonnet": {
+        "display_name": "claude-3-5-sonnet",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "claude-3-5-sonnet-20240620",
+        "provider": "anthropic",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    }, 
+    "dart_claude_3_opus": {
+        "display_name": "claude-3-opus",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "claude-3-opus-20240229",
+        "provider": "anthropic",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    }, 
+    "dart_llama_3_3_70b": {
+        "display_name": "llama-3.3-70b-versatile",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "llama-3.3-70b-versatile",
+        "provider": "groq",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    }, 
+    "dart_llama_3_1_8b": {
+        "display_name": "llama-3.1-8b-instant",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "llama-3.1-8b-instant",
+        "provider": "groq",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_ollama_llama3_1_8b": {
+        "display_name": "ollama-llama3.1:8b",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "llama3.1:8b",
+        "provider": "ollama",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "dart_ollama_llama3_3_70b": {
+        "display_name": "ollama-llama3.3:70b",
+        "prompt_file": "./prompts/swarm/dart.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "llama3.3:70b-instruct-q4_K_M",
+        "provider": "ollama",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": [
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+
+    "task_decomposer": {
+        "display_name": "Task Decomposer",
+        "prompt_file": "./prompts/swarm/task_decomposer_prompt.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "gpt-4o",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": ["tasks_topic"],
+        "json_keys": {"tasks": "tasks_topic"},
+        "functions_description": ""
+    },
+    "composer": {
+        "display_name": "Composer",
+        "prompt_file": "./prompts/swarm/composer_prompt.txt",
+        "type": "GRADIO_MESSAGE_MODES",
+        "model_version": "gpt-4o",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": [],
+        "output_topics": ["tasks_topic"],
+        "json_keys": {"tasks": "tasks_topic"},
+        "functions_description": ""
+    },
+    "instruction_translator": {
+        "display_name": "Instruction Translator (Developer Mode)",
+        "prompt_file": "./prompts/swarm/instruction_translator_prompt.txt",
+        "type": "ROS_MESSAGE_MODE",
+        "model_version": "gpt-4o",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": ["tasks_topic"],
+        "output_topics": [
+            "robovla_instruction_translator_out",
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ""
+    },
+    "planner": {
+        "display_name": "Planner (Developer Mode)",
+        "prompt_file": "./prompts/swarm/planner_prompt.txt",
+        "type": "ROS_MESSAGE_MODE",
+        "model_version": "gpt-4o",
+        "provider": "openai",
+        "max_tokens": 2048,
+        "temperature": 0,
+        "frequency_penalty": 0,
+        "input_topics": ["tasks_topic"],
+        "output_topics": [
+            "robovla_instruction_translator_out",
+            "instruction_topic",
+            "keywords_topic"
+        ],
+        "json_keys": {
+            "instruction_function": "instruction_topic",
+            "clip_keywords": "keywords_topic"
+        },
+        "functions_description": ROBOTS_CONFIG["get_all_functions_description"]()
+    }
+}
+
+# Default modes to display in the UI
+# GRADIO_MESSAGE_MODES = ["task_2_commannd_prompt", "task_decomposer", "instruction_translator", "composer", "planner"]
+GRADIO_MESSAGE_MODES = ["dart_gpt_4o", "dart_gpt_3_5_turbo", "dart_gpt_4_turbo", "dart_claude_3_haiku", "dart_claude_3_sonnet", "dart_claude_3_opus", "dart_llama_3_3_70b","dart_llama_3_1_8b", "dart_ollama_llama3_3_70b", "dart_ollama_llama3_1_8b", "task_2_commannd_prompt"]
+ROS_MESSAGE_MODE = "instruction_translator"

dag_demo.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""
+Demo script showing DAG visualization integration with QA_LLM_Module
+"""
+
+import json
+from dag_visualizer import DAGVisualizer
+
+def create_demo_task_data():
+    """Create sample task data for demonstration"""
+    return {
+        "tasks": [
+            {
+                "task": "excavate_soil_from_pile",
+                "instruction_function": {
+                    "name": "excavate_soil_from_pile",
+                    "robot_ids": ["robot_excavator_01"],
+                    "dependencies": [],
+                    "object_keywords": ["soil_pile"]
+                }
+            },
+            {
+                "task": "transport_soil_to_pit",
+                "instruction_function": {
+                    "name": "transport_soil_to_pit",
+                    "robot_ids": ["robot_dump_truck_01"],
+                    "dependencies": ["excavate_soil_from_pile"],
+                    "object_keywords": ["pit"]
+                }
+            },
+            {
+                "task": "avoid_puddle_areas",
+                "instruction_function": {
+                    "name": "avoid_areas_for_all_robots",
+                    "robot_ids": ["robot_excavator_01", "robot_dump_truck_01"],
+                    "dependencies": [],
+                    "object_keywords": ["puddle1", "puddle2"]
+                }
+            },
+            {
+                "task": "coordinate_dump_operation",
+                "instruction_function": {
+                    "name": "coordinate_dump_operation",
+                    "robot_ids": ["robot_dump_truck_01", "robot_excavator_01"],
+                    "dependencies": ["transport_soil_to_pit", "avoid_puddle_areas"],
+                    "object_keywords": ["pit", "dumping_zone"]
+                }
+            }
+        ]
+    }
+
+def main():
+    print("🎯 DAG Visualization Demo for QA_LLM_Module")
+    print("=" * 50)
+    
+    # Create sample task data
+    task_data = create_demo_task_data()
+    print("Sample task data:")
+    print(json.dumps(task_data, indent=2))
+    print()
+    
+    # Initialize visualizer
+    visualizer = DAGVisualizer()
+    
+    # Create detailed visualization
+    print("Creating detailed DAG visualization...")
+    detailed_image = visualizer.create_dag_visualization(
+        task_data, 
+        title="DART-LLM: Robot Task Dependency Graph"
+    )
+    
+    if detailed_image:
+        print(f"✅ Detailed visualization saved: {detailed_image}")
+    else:
+        print("❌ Failed to create detailed visualization")
+    
+    # Create simplified visualization
+    print("\nCreating simplified DAG visualization...")
+    simple_image = visualizer.create_simplified_dag_visualization(
+        task_data,
+        title="DART-LLM: Task Dependencies"
+    )
+    
+    if simple_image:
+        print(f"✅ Simplified visualization saved: {simple_image}")
+    else:
+        print("❌ Failed to create simplified visualization")
+    
+    print("\n" + "=" * 50)
+    print("🚀 Integration Instructions:")
+    print("1. Install requirements: pip install -r requirements.txt")
+    print("2. Run the main application: python3 main.py")
+    print("3. Enter a query that generates task JSON")
+    print("4. The DAG visualization will appear automatically!")
+    print("\n📝 Example queries:")
+    print("- 'Move half of the soil from the soil pile to the pit.'")
+    print("- 'Excavate material and transport it while avoiding obstacles.'")
+
+if __name__ == "__main__":
+    main()

dag_visualizer.py

@@ -0,0 +1,334 @@
+import matplotlib.pyplot as plt
+import matplotlib
+matplotlib.use('Agg')  # Use non-interactive backend for server environments
+import networkx as nx
+import json
+import numpy as np
+from loguru import logger
+import os
+import tempfile
+from datetime import datetime
+
+class DAGVisualizer:
+    def __init__(self):
+        # Configure Matplotlib to use IEEE-style parameters
+        plt.rcParams.update({
+            'font.family': 'DejaVu Sans',  # Use available font instead of Times New Roman
+            'font.size': 10,
+            'axes.linewidth': 1.2,
+            'axes.labelsize': 12,
+            'xtick.labelsize': 10,
+            'ytick.labelsize': 10,
+            'legend.fontsize': 10,
+            'figure.titlesize': 14
+        })
+    
+    def create_dag_from_tasks(self, task_data):
+        """
+        Create a directed graph from task data.
+        
+        Args:
+            task_data: Dictionary containing tasks with structure like:
+                {
+                    "tasks": [
+                        {
+                            "task": "task_name",
+                            "instruction_function": {
+                                "name": "function_name",
+                                "robot_ids": ["robot1", "robot2"],
+                                "dependencies": ["dependency_task"],
+                                "object_keywords": ["object1", "object2"]
+                            }
+                        }
+                    ]
+                }
+        
+        Returns:
+            NetworkX DiGraph object
+        """
+        if not task_data or "tasks" not in task_data:
+            logger.warning("Invalid task data structure")
+            return None
+        
+        # Create a directed graph
+        G = nx.DiGraph()
+        
+        # Add nodes and store mapping from task name to ID
+        task_mapping = {}
+        for i, task in enumerate(task_data["tasks"]):
+            task_id = i + 1
+            task_name = task["task"]
+            task_mapping[task_name] = task_id
+            
+            # Add node with attributes
+            G.add_node(task_id, 
+                      name=task_name,
+                      function=task["instruction_function"]["name"],
+                      robots=task["instruction_function"].get("robot_ids", []),
+                      objects=task["instruction_function"].get("object_keywords", []))
+        
+        # Add dependency edges
+        for i, task in enumerate(task_data["tasks"]):
+            task_id = i + 1
+            dependencies = task["instruction_function"]["dependencies"]
+            for dep in dependencies:
+                if dep in task_mapping:
+                    dep_id = task_mapping[dep]
+                    G.add_edge(dep_id, task_id)
+        
+        return G
+    
+    def calculate_layout(self, G):
+        """
+        Calculate hierarchical layout for the graph based on dependencies.
+        """
+        if not G:
+            return {}
+        
+        # Calculate layers based on dependencies
+        layers = {}
+        
+        def get_layer(node_id, visited=None):
+            if visited is None:
+                visited = set()
+            if node_id in visited:
+                return 0
+            visited.add(node_id)
+            
+            predecessors = list(G.predecessors(node_id))
+            if not predecessors:
+                return 0
+            return max(get_layer(pred, visited.copy()) for pred in predecessors) + 1
+        
+        for node in G.nodes():
+            layer = get_layer(node)
+            layers.setdefault(layer, []).append(node)
+        
+        # Calculate positions by layer
+        pos = {}
+        layer_height = 3.0
+        node_width = 4.0
+        
+        for layer_idx, nodes in layers.items():
+            y = layer_height * (len(layers) - 1 - layer_idx)
+            start_x = -(len(nodes) - 1) * node_width / 2
+            for i, node in enumerate(sorted(nodes)):
+                pos[node] = (start_x + i * node_width, y)
+        
+        return pos
+    
+    def create_dag_visualization(self, task_data, title="Robot Task Dependency Graph"):
+        """
+        Create a DAG visualization from task data and return the image path.
+        
+        Args:
+            task_data: Task data dictionary
+            title: Title for the graph
+        
+        Returns:
+            str: Path to the generated image file
+        """
+        try:
+            # Create graph
+            G = self.create_dag_from_tasks(task_data)
+            if not G or len(G.nodes()) == 0:
+                logger.warning("No tasks found or invalid graph structure")
+                return None
+            
+            # Calculate layout
+            pos = self.calculate_layout(G)
+            
+            # Create figure
+            fig, ax = plt.subplots(1, 1, figsize=(max(12, len(G.nodes()) * 2), 8))
+            
+            # Draw edges with arrows
+            nx.draw_networkx_edges(G, pos, 
+                                  edge_color='#2E86AB',
+                                  arrows=True,
+                                  arrowsize=20,
+                                  arrowstyle='->',
+                                  width=2,
+                                  alpha=0.8,
+                                  connectionstyle="arc3,rad=0.1")
+            
+            # Color nodes based on their position in the graph
+            node_colors = []
+            for node in G.nodes():
+                if G.in_degree(node) == 0:  # Start nodes
+                    node_colors.append('#F24236')
+                elif G.out_degree(node) == 0:  # End nodes
+                    node_colors.append('#A23B72')
+                else:  # Intermediate nodes
+                    node_colors.append('#F18F01')
+            
+            # Draw nodes
+            nx.draw_networkx_nodes(G, pos,
+                                  node_color=node_colors,
+                                  node_size=3500,
+                                  alpha=0.9,
+                                  edgecolors='black',
+                                  linewidths=2)
+            
+            # Label nodes with task IDs
+            node_labels = {node: f"T{node}" for node in G.nodes()}
+            nx.draw_networkx_labels(G, pos, node_labels,
+                                   font_size=18,
+                                   font_weight='bold',
+                                   font_color='white')
+            
+            # Add detailed info text boxes for each task
+            for i, node in enumerate(G.nodes()):
+                x, y = pos[node]
+                function_name = G.nodes[node]['function']
+                robots = G.nodes[node]['robots']
+                objects = G.nodes[node]['objects']
+                
+                # Create info text content
+                info_text = f"Task {node}: {function_name.replace('_', ' ').title()}\n"
+                if robots:
+                    robot_text = ", ".join([r.replace('robot_', '').replace('_', ' ').title() for r in robots])
+                    info_text += f"Robots: {robot_text}\n"
+                if objects:
+                    object_text = ", ".join(objects)
+                    info_text += f"Objects: {object_text}"
+                
+                # Calculate offset based on node position to avoid overlaps
+                offset_x = 2.2 if i % 2 == 0 else -2.2
+                offset_y = 0.5 if i % 4 < 2 else -0.5
+                
+                # Choose alignment based on offset direction
+                h_align = 'left' if offset_x > 0 else 'right'
+                
+                # Draw text box
+                bbox_props = dict(boxstyle="round,pad=0.4", 
+                                 facecolor='white', 
+                                 edgecolor='gray',
+                                 alpha=0.95,
+                                 linewidth=1)
+                
+                ax.text(x + offset_x, y + offset_y, info_text,
+                       bbox=bbox_props,
+                       fontsize=12,
+                       verticalalignment='center',
+                       horizontalalignment=h_align,
+                       weight='bold')
+                
+                # Draw dashed connector line from node to text box
+                ax.plot([x, x + offset_x], [y, y + offset_y], 
+                        linestyle='--', color='gray', alpha=0.6, linewidth=1)
+            
+            # Expand axis limits to fit everything
+            x_vals = [coord[0] for coord in pos.values()]
+            y_vals = [coord[1] for coord in pos.values()]
+            ax.set_xlim(min(x_vals) - 4.0, max(x_vals) + 4.0)
+            ax.set_ylim(min(y_vals) - 2.0, max(y_vals) + 2.0)
+            
+            # Set overall figure properties
+            ax.set_title(title, fontsize=16, fontweight='bold', pad=20)
+            ax.set_aspect('equal')
+            ax.margins(0.2)
+            ax.axis('off')
+            
+            # Add legend for node types
+            legend_elements = [
+                plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='#F24236', 
+                           markersize=10, label='Start Tasks', markeredgecolor='black'),
+                plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='#A23B72', 
+                           markersize=10, label='End Tasks', markeredgecolor='black'),
+                plt.Line2D([0], [0], marker='o', color='w', markerfacecolor='#F18F01', 
+                           markersize=10, label='Intermediate Tasks', markeredgecolor='black'),
+                plt.Line2D([0], [0], color='#2E86AB', linewidth=2, label='Dependencies')
+            ]
+            ax.legend(handles=legend_elements, loc='upper left', bbox_to_anchor=(1.05, 1.05))
+            
+            # Adjust layout and save
+            plt.tight_layout()
+            
+            # Create temporary file for saving the image
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            temp_dir = tempfile.gettempdir()
+            image_path = os.path.join(temp_dir, f'dag_visualization_{timestamp}.png')
+            
+            plt.savefig(image_path, dpi=400, bbox_inches='tight', 
+                        pad_inches=0.1, facecolor='white', edgecolor='none')
+            plt.close(fig)  # Close figure to free memory
+            
+            logger.info(f"DAG visualization saved to: {image_path}")
+            return image_path
+            
+        except Exception as e:
+            logger.error(f"Error creating DAG visualization: {e}")
+            return None
+    
+    def create_simplified_dag_visualization(self, task_data, title="Robot Task Graph"):
+        """
+        Create a simplified DAG visualization suitable for smaller displays.
+        
+        Args:
+            task_data: Task data dictionary
+            title: Title for the graph
+        
+        Returns:
+            str: Path to the generated image file
+        """
+        try:
+            # Create graph
+            G = self.create_dag_from_tasks(task_data)
+            if not G or len(G.nodes()) == 0:
+                logger.warning("No tasks found or invalid graph structure")
+                return None
+            
+            # Calculate layout
+            pos = self.calculate_layout(G)
+            
+            # Create figure for simplified graph
+            fig, ax = plt.subplots(1, 1, figsize=(10, 6))
+            
+            # Draw edges
+            nx.draw_networkx_edges(G, pos, 
+                                  edge_color='black',
+                                  arrows=True,
+                                  arrowsize=15,
+                                  arrowstyle='->',
+                                  width=1.5)
+            
+            # Draw nodes
+            nx.draw_networkx_nodes(G, pos,
+                                  node_color='lightblue',
+                                  node_size=3000,
+                                  edgecolors='black',
+                                  linewidths=1.5)
+            
+            # Add node labels with simplified names
+            labels = {}
+            for node in G.nodes():
+                function_name = G.nodes[node]['function']
+                simplified_name = function_name.replace('_', ' ').title()
+                if len(simplified_name) > 15:
+                    simplified_name = simplified_name[:12] + "..."
+                labels[node] = f"T{node}\n{simplified_name}"
+            
+            nx.draw_networkx_labels(G, pos, labels,
+                                   font_size=11,
+                                   font_weight='bold')
+            
+            ax.set_title(title, fontsize=14, fontweight='bold')
+            ax.axis('off')
+            
+            # Adjust layout and save
+            plt.tight_layout()
+            
+            # Create temporary file for saving the image
+            timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+            temp_dir = tempfile.gettempdir()
+            image_path = os.path.join(temp_dir, f'simple_dag_{timestamp}.png')
+            
+            plt.savefig(image_path, dpi=400, bbox_inches='tight')
+            plt.close(fig)  # Close figure to free memory
+            
+            logger.info(f"Simplified DAG visualization saved to: {image_path}")
+            return image_path
+            
+        except Exception as e:
+            logger.error(f"Error creating simplified DAG visualization: {e}")
+            return None

gradio_llm_interface.py

@@ -0,0 +1,298 @@
+import rclpy
+import gradio as gr
+from loguru import logger
+from llm_request_handler import LLMRequestHandler
+from ros_node_publisher import RosNodePublisher
+from json_processor import JsonProcessor
+from dag_visualizer import DAGVisualizer
+from config import ROBOTS_CONFIG, MODEL_CONFIG, MODE_CONFIG
+
+class GradioLlmInterface:
+    def __init__(self):
+        self.node_publisher = None
+        self.received_tasks = []
+        self.json_processor = JsonProcessor()
+        self.dag_visualizer = DAGVisualizer()
+
+    def initialize_interface(self, mode):
+        if not rclpy.ok():
+            rclpy.init()
+        self.node_publisher = RosNodePublisher(mode)
+        mode_config = MODE_CONFIG[mode]
+        # Use provider from mode_config and model version
+        model_version = mode_config.get("model_version", MODEL_CONFIG["default_model"])
+        provider = mode_config.get("provider", "openai")
+        llm_handler = LLMRequestHandler(
+            model_name=model_version,
+            provider=provider,
+            max_tokens=mode_config.get("max_tokens", MODEL_CONFIG["max_tokens"]),
+            temperature=mode_config.get("temperature", MODEL_CONFIG["temperature"]),
+            frequency_penalty=mode_config.get("frequency_penalty", MODEL_CONFIG["frequency_penalty"]),
+            list_navigation_once=True
+        )
+        file_path = mode_config["prompt_file"]
+        initial_messages = llm_handler.build_initial_messages(file_path, mode)
+        # Don't create chatbot here, return state data only
+        state_data = {
+            "file_path": file_path,
+            "initial_messages": initial_messages,
+            "mode": mode,
+            # store config dict with provider
+            "llm_config": llm_handler.get_config_dict()
+        }
+        return state_data
+
+    async def predict(self, input, state):
+        if not self.node_publisher.is_initialized():
+            mode = state.get('mode')
+            self.node_publisher.initialize_node(mode)
+        
+        initial_messages = state['initial_messages']
+        full_history = initial_messages + state.get('history', [])
+        user_input = f"# Query: {input}"
+        full_history.append({"role": "user", "content": user_input})
+
+        mode_config = MODE_CONFIG[state.get('mode')]
+        if mode_config["type"] == 'complex' and self.received_tasks:
+            for task in self.received_tasks:
+                task_prompt = f"# Task: {task}"
+                full_history.append({"role": "user", "content": task_prompt})
+            self.received_tasks = []
+
+        # Create a new LLMRequestHandler instance for each request
+        llm_config = state['llm_config']
+        llm_handler = LLMRequestHandler.create_from_config_dict(llm_config)
+
+        response = await llm_handler.make_completion(full_history)
+        if response:
+            full_history.append({"role": "assistant", "content": response})
+        else:
+            response = "Error: Unable to get response."
+            full_history.append({"role": "assistant", "content": response})
+        
+        response_json = self.json_processor.process_response(response)
+        
+        # Store the task plan for approval workflow
+        state.update({'pending_task_plan': response_json})
+
+        # Generate DAG visualization if valid task data is available
+        dag_image_path = None
+        confirm_button_visible = False
+        if response_json and "tasks" in response_json:
+            try:
+                dag_image_path = self.dag_visualizer.create_dag_visualization(
+                    response_json, 
+                    title="Robot Task Dependency Graph - Pending Approval"
+                )
+                logger.info(f"DAG visualization generated: {dag_image_path}")
+                confirm_button_visible = True
+            except Exception as e:
+                logger.error(f"Failed to generate DAG visualization: {e}")
+
+        # Modify the messages format to match the "messages" type
+        messages = [{"role": message["role"], "content": message["content"]} for message in full_history[len(initial_messages):]]
+        updated_history = state.get('history', []) + [{"role": "user", "content": input}, {"role": "assistant", "content": response}]
+        state.update({'history': updated_history})
+        return messages, state, dag_image_path, gr.update(visible=confirm_button_visible)
+    
+    def clear_chat(self, state):
+        state['history'] = []
+    
+    def show_task_plan_editor(self, state):
+        """
+        Show the current task plan in JSON format for manual editing.
+        """
+        # Check for pending plan first, then deployed plan as fallback
+        pending_plan = state.get('pending_task_plan')
+        deployed_plan = state.get('deployed_task_plan')
+        
+        # Use pending plan if available, otherwise use deployed plan
+        current_plan = pending_plan if pending_plan else deployed_plan
+        
+        if current_plan and "tasks" in current_plan and len(current_plan["tasks"]) > 0:
+            import json
+            # Format JSON for better readability
+            formatted_json = json.dumps(current_plan, indent=2, ensure_ascii=False)
+            plan_status = "pending" if pending_plan else "deployed"
+            logger.info(f"📝 Task plan editor opened with {plan_status} plan")
+            
+            # Set pending plan for editing (copy from deployed if needed)
+            if not pending_plan and deployed_plan:
+                state.update({'pending_task_plan': deployed_plan})
+            
+            return (
+                gr.update(visible=True, value=formatted_json),  # Show editor with current JSON
+                gr.update(visible=True),  # Show Update DAG button
+                gr.update(visible=False),  # Hide Validate & Deploy button
+                f"📝 **Task Plan Editor Opened**\n\nYou can now manually edit the task plan JSON below. {plan_status.title()} plan loaded for editing."
+            )
+        else:
+            # Provide a better template with example structure
+            template_json = """{
+  "tasks": [
+    {
+      "task": "example_task_1",
+      "instruction_function": {
+        "name": "example_function_name",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": [],
+        "object_keywords": ["object1", "object2"]
+      }
+    }
+  ]
+}"""
+            logger.info("📝 Task plan editor opened with template")
+            return (
+                gr.update(visible=True, value=template_json),  # Show template
+                gr.update(visible=True),  # Show Update DAG button  
+                gr.update(visible=False),  # Hide Validate & Deploy button
+                "⚠️ **No Task Plan Available**\n\nStarting with example template. Please edit the JSON structure and update."
+            )
+    
+    def update_dag_from_editor(self, edited_json, state):
+        """
+        Update DAG visualization from manually edited JSON.
+        """
+        try:
+            import json
+            # Parse the edited JSON
+            edited_plan = json.loads(edited_json)
+            
+            # Validate the JSON structure
+            if "tasks" not in edited_plan:
+                raise ValueError("JSON must contain 'tasks' field")
+            
+            # Store the edited plan
+            state.update({'pending_task_plan': edited_plan})
+            
+            # Generate updated DAG visualization
+            dag_image_path = self.dag_visualizer.create_dag_visualization(
+                edited_plan, 
+                title="Robot Task Dependency Graph - EDITED & PENDING APPROVAL"
+            )
+            
+            logger.info("🔄 DAG updated from manual edits")
+            
+            return (
+                dag_image_path,
+                gr.update(visible=True),  # Show Validate & Deploy button
+                gr.update(visible=False),  # Hide editor
+                gr.update(visible=False),  # Hide Update DAG button
+                "✅ **DAG Updated Successfully**\n\nTask plan has been updated with your edits. Please review the visualization and click 'Validate & Deploy' to proceed.",
+                state
+            )
+            
+        except json.JSONDecodeError as e:
+            error_msg = f"❌ **JSON Parsing Error**\n\nInvalid JSON format: {str(e)}\n\nPlease fix the JSON syntax and try again."
+            return (
+                None,
+                gr.update(visible=False),  # Hide Validate & Deploy button
+                gr.update(visible=True),   # Keep editor visible
+                gr.update(visible=True),   # Keep Update DAG button visible
+                error_msg,
+                state
+            )
+        except Exception as e:
+            error_msg = f"❌ **Update Failed**\n\nError: {str(e)}"
+            logger.error(f"Failed to update DAG from editor: {e}")
+            return (
+                None,
+                gr.update(visible=False),  # Hide Validate & Deploy button
+                gr.update(visible=True),   # Keep editor visible
+                gr.update(visible=True),   # Keep Update DAG button visible
+                error_msg,
+                state
+            )
+        
+    def validate_and_deploy_task_plan(self, state):
+        """
+        Validate and deploy the task plan to the construction site.
+        This function implements the safety confirmation workflow.
+        """
+        pending_plan = state.get('pending_task_plan')
+        if pending_plan:
+            try:
+                # Deploy the approved task plan to ROS
+                self.node_publisher.publish_response(pending_plan)
+                
+                # Update DAG visualization to show approved status
+                approved_image_path = None
+                if "tasks" in pending_plan:
+                    approved_image_path = self.dag_visualizer.create_dag_visualization(
+                        pending_plan, 
+                        title="Robot Task Dependency Graph - APPROVED & DEPLOYED"
+                    )
+                
+                # Keep the deployed plan for potential re-editing, but mark as deployed
+                state.update({'deployed_task_plan': pending_plan, 'pending_task_plan': None})
+                
+                logger.info("✅ Task plan validated and deployed to construction site")
+                
+                # Return confirmation message and updated visualization
+                confirmation_msg = "✅ **Task Plan Successfully Deployed**\n\nThe validated task dependency graph has been sent to the construction site robots. All safety protocols confirmed."
+                
+                return (
+                    confirmation_msg,
+                    approved_image_path,
+                    gr.update(visible=False),  # Hide confirmation button
+                    state
+                )
+                
+            except Exception as e:
+                logger.error(f"Failed to deploy task plan: {e}")
+                error_msg = f"❌ **Deployment Failed**\n\nError: {str(e)}"
+                return (
+                    error_msg,
+                    None,
+                    gr.update(visible=True),  # Keep button visible for retry
+                    state
+                )
+        else:
+            warning_msg = "⚠️ **No Task Plan to Deploy**\n\nPlease generate a task plan first."
+            return (
+                warning_msg,
+                None,
+                gr.update(visible=False),
+                state
+            )
+
+    def update_chatbot(self, mode, state):
+        # Destroy and reinitialize the ROS node
+        self.node_publisher.destroy_node()
+        if not rclpy.ok():
+            rclpy.init()
+        self.node_publisher = RosNodePublisher(mode)
+        self.json_processor = JsonProcessor()
+
+        # Update llm_handler with the new model settings
+        mode_config = MODE_CONFIG[mode]
+        model_version = mode_config["model_version"]
+        model_type = mode_config.get("model_type", "openai")  # Ensure the correct model_type is used
+        provider = mode_config.get("provider", MODEL_CONFIG["provider"])
+
+        # Re-instantiate LLMRequestHandler with the new model_version and model_type
+        llm_handler = LLMRequestHandler(
+            model_version=model_version,
+            provider=provider,
+            max_tokens=mode_config.get("max_tokens", MODEL_CONFIG["max_tokens"]),
+            temperature=mode_config.get("temperature", MODEL_CONFIG["temperature"]),
+            frequency_penalty=mode_config.get("frequency_penalty", MODEL_CONFIG["frequency_penalty"]),
+            list_navigation_once=True,
+            model_type=model_type
+        )
+
+        # Update the prompt file and initial messages
+        file_path = mode_config["prompt_file"]
+        initial_messages = llm_handler.build_initial_messages(file_path, mode)
+
+        # Update state with the new handler and reset history
+        logger.info(f"Updating chatbot with {file_path}, model {model_version}, provider {provider}")
+        state['file_path'] = file_path
+        state['initial_messages'] = initial_messages
+        state['history'] = []
+        state['mode'] = mode
+        state['llm_config'] = llm_handler.get_config_dict()  # Update the state with the new handler
+
+        logger.info(f"\033[33mMode updated to {mode}\033[0m")
+        return gr.update(value=[]), state
+

json_processor.py

@@ -0,0 +1,33 @@
+import json
+import re
+from loguru import logger
+
+class JsonProcessor:
+    def process_response(self, response):
+        try:
+            # Search for JSON string in the response
+            json_str_match = re.search(r'\{.*\}', response, re.DOTALL)
+            if json_str_match:
+                # Get the matched JSON string
+                json_str = json_str_match.group()
+                logger.debug(f"Full JSON string: {json_str}")  
+
+                # Replace escape characters and remove trailing commas
+                json_str = json_str.replace("\\", "")
+                json_str = json_str.replace(r'\\_', '_')
+                json_str = re.sub(r',\s*}', '}', json_str)
+                json_str = re.sub(r',\s*\]', ']', json_str)
+
+                # Parse the JSON string
+                response_json = json.loads(json_str)
+                return response_json
+            else:
+                logger.error("No JSON string match found in response.")
+                return None
+
+        except json.JSONDecodeError as e:
+            logger.error(f"JSONDecodeError: {e}")
+        except Exception as e:
+            logger.error(f"Unexpected error: {e}")
+
+        return None

llm_request_handler.py

@@ -0,0 +1,278 @@
+import os
+import json
+import asyncio
+from typing import List, Optional, Dict, Any
+from loguru import logger
+from pydantic import BaseModel
+
+from langchain_core.prompts import ChatPromptTemplate
+from langchain_core.messages import SystemMessage, HumanMessage, AIMessage
+from langchain_openai import ChatOpenAI
+from langchain_anthropic import ChatAnthropic
+from langchain_groq import ChatGroq
+from langchain_ollama import ChatOllama
+
+from config import MODEL_CONFIG, INITIAL_MESSAGES_CONFIG, MODE_CONFIG, NAVIGATION_FUNCTIONS, ROBOT_SPECIFIC_FUNCTIONS, ROBOT_NAMES
+
+class LLMRequestConfig(BaseModel):
+    model_name: str = MODEL_CONFIG["default_model"]
+    max_tokens: int = MODEL_CONFIG["max_tokens"]
+    temperature: float = MODEL_CONFIG["temperature"]
+    frequency_penalty: float = MODEL_CONFIG["frequency_penalty"]
+    list_navigation_once: bool = True
+    provider: str = "openai"
+    
+    # Resolve Pydantic namespace conflicts
+    model_config = {"protected_namespaces": ()}
+
+    def to_dict(self):
+        return {
+            "model_name": self.model_name,
+            "max_tokens": self.max_tokens,
+            "temperature": self.temperature,
+            "frequency_penalty": self.frequency_penalty,
+            "list_navigation_once": self.list_navigation_once,
+            "provider": self.provider
+        }
+
+    @classmethod
+    def from_dict(cls, config_dict):
+        return cls(**config_dict)
+
+class LLMRequestHandler:
+    class Message(BaseModel):
+        role: str
+        content: str
+
+    def __init__(self, 
+                 # Support both old and new parameter names for backward compatibility
+                 model_version: str = None, 
+                 model_name: str = None,
+                 max_tokens: int = None, 
+                 temperature: float = None, 
+                 frequency_penalty: float = None, 
+                 list_navigation_once: bool = None, 
+                 model_type: str = None,
+                 provider: str = None,
+                 config: Optional[LLMRequestConfig] = None):
+                 
+        # Initialize with config or from individual parameters
+        if config:
+            self.config = config
+        else:
+            # Create config from individual parameters, giving priority to new names
+            self.config = LLMRequestConfig(
+                model_name=model_name or model_version or MODEL_CONFIG["default_model"],
+                max_tokens=max_tokens or MODEL_CONFIG["max_tokens"],
+                temperature=temperature or MODEL_CONFIG["temperature"],
+                frequency_penalty=frequency_penalty or MODEL_CONFIG["frequency_penalty"],
+                list_navigation_once=list_navigation_once if list_navigation_once is not None else True,
+                provider=provider or model_type or "openai"
+            )
+        
+        # Store parameters for easier access
+        self.model_name = self.config.model_name
+        self.model_version = self.model_name  # Alias for backward compatibility
+        self.max_tokens = self.config.max_tokens
+        self.temperature = self.config.temperature
+        self.frequency_penalty = self.config.frequency_penalty
+        self.list_navigation_once = self.config.list_navigation_once
+        self.provider = self.config.provider
+        self.model_type = self.provider  # Alias for backward compatibility
+
+        # Store API keys
+        self.openai_api_key = os.getenv("OPENAI_API_KEY")
+        self.anthropic_api_key = os.getenv("ANTHROPIC_API_KEY")
+        self.groq_api_key = os.getenv("GROQ_API_KEY")
+        
+        # Create the appropriate LangChain LLM based on provider
+        self._setup_llm()
+    
+    def _setup_llm(self):
+        """Initialize the appropriate LangChain LLM based on provider."""
+        if "anthropic" in self.provider or "claude" in self.model_name:
+            self.llm = ChatAnthropic(
+                api_key=self.anthropic_api_key,
+                model_name=self.model_name,
+                max_tokens=self.max_tokens,
+                temperature=self.temperature
+            )
+        elif "ollama" in self.provider or "ollama" in self.model_name:
+            self.llm = ChatOllama(
+            model=self.model_name,
+            max_tokens=self.max_tokens,
+            temperature=self.temperature,
+            base_url="http://host.docker.internal:11434"
+        )
+        elif "groq" in self.provider or "llama" in self.model_name:
+            self.llm = ChatGroq(
+                api_key=self.groq_api_key,
+                model_name=self.model_name,
+                max_tokens=self.max_tokens,
+                temperature=self.temperature,
+                frequency_penalty=self.frequency_penalty
+            )
+        else:  # Default to OpenAI
+            self.llm = ChatOpenAI(
+                api_key=self.openai_api_key,
+                model_name=self.model_name,
+                max_tokens=self.max_tokens,
+                temperature=self.temperature,
+                frequency_penalty=self.frequency_penalty
+            )
+
+    def get_config_dict(self):
+        """Get a serializable configuration dictionary"""
+        return self.config.to_dict()
+
+    @staticmethod
+    def create_from_config_dict(config_dict):
+        """Create a new handler instance from a config dictionary"""
+        config = LLMRequestConfig.from_dict(config_dict)
+        return LLMRequestHandler(config=config)
+        
+    def load_object_data(self) -> Dict[str, Any]:
+        """Load environment information (E) from a JSON file"""
+        json_path = os.path.abspath(os.path.join(os.path.dirname(__file__), '..', 'ros2_ws', 'src', 'breakdown_function_handler', 'object_database', 'object_database.json'))
+        
+        with open(json_path, 'r') as json_file:
+            data = json.load(json_file)
+        
+        return self.format_env_object(data)
+    
+    def format_env_object(self, data: List[Dict[str, Any]]) -> Dict[str, Any]:
+        """Format the environment data (E) for use in the prompt"""
+        formatted_env_object = {}
+        for obj in data:
+            object_name = obj['object_name']
+            target_position = obj['target_position']
+            shape = obj['shape']
+            formatted_env_object[object_name] = {
+                "position": {
+                    "x": target_position["x"],
+                    "y": target_position["y"]
+                },
+                "shape": shape
+            }
+        return formatted_env_object
+
+    def build_initial_messages(self, file_path: str, mode: str) -> List[Dict[str, str]]:
+        """Build the initial prompt (P = (I, E, R, S))"""
+        with open(file_path, 'r', encoding='utf-8') as file:
+            user1 = file.read()  # Example user instructions for few-shot learning (optional)
+
+        system = INITIAL_MESSAGES_CONFIG["system"]
+
+        # Load environment information (E)
+        env_objects = self.load_object_data()
+
+        # Create the user introduction with robot set (R), skills (S), and environment (E)
+        user_intro = INITIAL_MESSAGES_CONFIG["user_intro"]["default"] + INITIAL_MESSAGES_CONFIG["user_intro"].get(mode, "")
+        functions_description = MODE_CONFIG[mode].get("functions_description", "")
+
+        # Format user introduction with the instruction (I), robot set (R), skills (S), and environment (E)
+        user_intro = user_intro.format(
+            library=NAVIGATION_FUNCTIONS+ROBOT_SPECIFIC_FUNCTIONS, 
+            env_objects=env_objects, 
+            robot_names=ROBOT_NAMES, 
+            fewshot_examples=user1, 
+            functions_description=functions_description
+        )
+
+        assistant1 = INITIAL_MESSAGES_CONFIG["assistant"]
+
+        # Construct the messages (system, user, assistant)
+        messages = [
+            {"role": "system", "content": system},
+            {"role": "user", "content": user_intro},
+            {"role": "assistant", "content": assistant1}
+        ]
+        return messages
+
+    def add_user_message(self, messages: List[Dict[str, str]], content: str) -> None:
+        """Add a user message with natural language instruction (I)"""
+        user_message = self.Message(role="user", content=content)
+        messages.append(user_message.model_dump())
+
+    def _convert_to_langchain_messages(self, full_history: List[Dict[str, str]]):
+        """Convert traditional message format to LangChain message objects"""
+        lc_messages = []
+        for msg in full_history:
+            if msg["role"] == "system":
+                lc_messages.append(SystemMessage(content=msg["content"]))
+            elif msg["role"] == "user":
+                lc_messages.append(HumanMessage(content=msg["content"]))
+            elif msg["role"] == "assistant":
+                lc_messages.append(AIMessage(content=msg["content"]))
+        return lc_messages
+
+    async def make_completion(self, full_history: List[Dict[str, str]]) -> Optional[str]:
+        """Make a completion request to the selected model using LangChain"""
+        logger.debug(f"Using model: {self.model_name}")
+        try:
+            # Convert traditional messages to LangChain message format
+            lc_messages = self._convert_to_langchain_messages(full_history)
+            
+            # Create a chat prompt template
+            chat_prompt = ChatPromptTemplate.from_messages(lc_messages)
+            
+            # Get the response
+            chain = chat_prompt | self.llm
+            response = await chain.ainvoke({})
+            
+            # Extract the content from the response
+            return response.content if hasattr(response, 'content') else str(response)
+            
+        except Exception as e:
+            logger.error(f"Error making completion: {e}")
+            return None
+
+
+if __name__ == "__main__":
+    async def main():
+        selected_model_index = 3  # 0 for OpenAI, 1 for Anthropic, 2 for LLaMA, 3 for Ollama
+
+        model_options = MODEL_CONFIG["model_options"]
+
+        # Choose the model based on selected_model_index
+        if selected_model_index == 0:
+            model = model_options[0]
+            provider = "openai"
+        elif selected_model_index == 1:
+            model = model_options[4]
+            provider = "anthropic"
+        elif selected_model_index == 2:
+            model = model_options[6]
+            provider = "groq"
+        elif selected_model_index == 3:
+            model = "llama3"
+            provider = "ollama"
+        else:
+            raise ValueError("Invalid selected_model_index")
+
+        logger.debug("Starting test llm_request_handler with LangChain...")
+        config = LLMRequestConfig(
+            model_name=model,
+            list_navigation_once=True,
+            provider=provider
+        )
+        handler = LLMRequestHandler(config=config)
+
+        # Build initial messages based on the selected model
+        if selected_model_index == 0:
+            messages = handler.build_initial_messages("/root/share/QA_LLM_Module/prompts/swarm/dart.txt", "dart_gpt_4o")
+        elif selected_model_index == 1:
+            messages = handler.build_initial_messages("/root/share/QA_LLM_Module/prompts/swarm/dart.txt", "dart_claude_3_sonnet")
+        elif selected_model_index == 2:
+            messages = handler.build_initial_messages("/root/share/QA_LLM_Module/prompts/swarm/dart.txt", "dart_llama_3_3_70b")
+        elif selected_model_index == 3:
+            messages = handler.build_initial_messages("/root/share/QA_LLM_Module/prompts/swarm/dart.txt", "dart_ollama_llama3_1_8b")
+
+        # Add a natural language instruction (I) to the prompt
+        handler.add_user_message(messages, "Excavator 1 performs excavation, then excavator 2 performs, then dump 1 performs unload.")
+
+        # Request completion from the model
+        response = await handler.make_completion(messages)
+        logger.debug(f"Response from make_completion: {response}")
+
+    asyncio.run(main())

main.py

@@ -0,0 +1,348 @@
+import gradio as gr
+from loguru import logger
+from gradio_llm_interface import GradioLlmInterface
+from config import GRADIO_MESSAGE_MODES, MODE_CONFIG
+import openai
+import os
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv()
+
+# Speech-to-text function using OpenAI Whisper
+def audio_to_text(audio):
+    if audio is None:
+        return "No audio file provided."
+    
+    try:
+        # Get OpenAI API key from environment variable
+        openai_api_key = os.getenv("OPENAI_API_KEY")
+        if not openai_api_key:
+            return "Error: OpenAI API key not found. Please set OPENAI_API_KEY environment variable."
+        
+        # Initialize OpenAI client
+        client = openai.OpenAI(api_key=openai_api_key)
+        
+        # Open and transcribe the audio file
+        with open(audio, "rb") as audio_file:
+            transcript = client.audio.transcriptions.create(
+                model="whisper-1",
+                file=audio_file
+            )
+        
+        return transcript.text
+        
+    except FileNotFoundError:
+        return "Error: Audio file not found."
+    except openai.AuthenticationError:
+        return "Error: Invalid OpenAI API key."
+    except openai.RateLimitError:
+        return "Error: OpenAI API rate limit exceeded."
+    except Exception as e:
+        logger.error(f"Speech-to-text error: {str(e)}")
+        return f"Error during speech recognition: {str(e)}"
+
+def main():
+    gradio_ros_interface = GradioLlmInterface()
+
+    title_markdown = ("""
+    # 🌋 DART-LLM: Dependency-Aware Multi-Robot Task Decomposition and Execution using Large Language Models
+    [[Project Page](https://wyd0817.github.io/project-dart-llm/)] [[Code](https://github.com/wyd0817/gradio_gpt_interface)] [[Model](https://artificialanalysis.ai/)] | 📚 [[RoboQA](https://www.overleaf.com/project/6614a987ae2994cae02efcb2)] 
+    """)
+
+    with gr.Blocks(css="""
+        #text-input, #audio-input {
+            height: 100px; /* Unified height */
+            max-height: 100px;
+            width: 100%; /* Full container width */
+            margin: 0;
+        }
+        .input-container {
+            display: flex; /* Flex layout */
+            gap: 10px; /* Spacing */
+            align-items: center; /* Vertical alignment */
+        }
+        #voice-input-container {
+            display: flex;
+            align-items: center;
+            gap: 15px;
+            margin: 15px 0;
+            padding: 15px;
+            background: linear-gradient(135deg, #ffeef8 0%, #fff5f5 100%);
+            border-radius: 20px;
+            border: 1px solid #ffe4e6;
+        }
+        #voice-btn {
+            width: 50px !important;
+            height: 50px !important;
+            border-radius: 50% !important;
+            font-size: 20px !important;
+            background: linear-gradient(135deg, #ff6b9d 0%, #c44569 100%) !important;
+            color: white !important;
+            border: none !important;
+            box-shadow: 0 4px 15px rgba(255, 107, 157, 0.3) !important;
+            transition: all 0.3s ease !important;
+        }
+        #voice-btn:hover {
+            transform: scale(1.05) !important;
+            box-shadow: 0 6px 20px rgba(255, 107, 157, 0.4) !important;
+        }
+        #voice-btn:active {
+            transform: scale(0.95) !important;
+        }
+        .voice-recording {
+            background: linear-gradient(135deg, #ff4757 0%, #ff3742 100%) !important;
+            animation: pulse 1.5s infinite !important;
+        }
+        @keyframes pulse {
+            0% { box-shadow: 0 4px 15px rgba(255, 71, 87, 0.3); }
+            50% { box-shadow: 0 4px 25px rgba(255, 71, 87, 0.6); }
+            100% { box-shadow: 0 4px 15px rgba(255, 71, 87, 0.3); }
+        }
+        #voice-status {
+            color: #ff6b9d;
+            font-size: 14px;
+            font-weight: 500;
+            text-align: center;
+            margin-top: 10px;
+        }
+        /* Enhanced layout for left-right split */
+        .gradio-container .gradio-row {
+            gap: 20px; /* Add spacing between columns */
+        }
+        .gradio-column {
+            padding: 10px;
+            border-radius: 8px;
+            background-color: var(--panel-background-fill);
+        }
+        /* Chat interface styling */
+        .chat-column {
+            border: 1px solid var(--border-color-primary);
+        }
+        /* DAG visualization column styling */
+        .dag-column {
+            border: 1px solid var(--border-color-primary);
+        }
+    """) as demo:
+        gr.Markdown(title_markdown)
+
+        mode_choices = [MODE_CONFIG[mode]["display_name"] for mode in GRADIO_MESSAGE_MODES]
+        mode_selector = gr.Radio(choices=mode_choices, label="Backend model", value=mode_choices[0])
+        clear_button = gr.Button("Clear Chat")
+
+        logger.info("Starting Gradio GPT Interface...")
+        
+        initial_mode = GRADIO_MESSAGE_MODES[0]
+        
+        def update_mode(selected_mode, state):
+            mode_key = [key for key, value in MODE_CONFIG.items() if value["display_name"] == selected_mode][0]
+            return gradio_ros_interface.update_chatbot(mode_key, state)
+
+        # Main content area with left-right layout
+        with gr.Row():
+            # Left column: Chat interface
+            with gr.Column(scale=1, elem_classes=["chat-column"]):
+                gr.Markdown("### 🤖 DART-LLM Chat Interface")
+                # Create chatbot component in the left column
+                chatbot_container = gr.Chatbot(label="DART-LLM", type="messages")
+                
+                # Initialize the interface and get state data
+                state_data = gradio_ros_interface.initialize_interface(initial_mode)
+                state = gr.State(state_data)
+                
+                # Add input area in the left column
+                with gr.Row(elem_id="input-container"):
+                    txt = gr.Textbox(show_label=False, placeholder="Enter text and press enter", elem_id="text-input", container=False)
+                    
+                with gr.Row(elem_id="voice-input-container"):
+                    with gr.Column(scale=4):
+                        # Hidden audio component
+                        audio_input = gr.Audio(
+                            sources=["microphone"], 
+                            type="filepath", 
+                            elem_id="audio-input", 
+                            show_label=False,
+                            interactive=True,
+                            streaming=False,
+                            visible=False
+                        )
+                        # Voice input status display
+                        voice_status = gr.Markdown("", elem_id="voice-status", visible=False)
+                        
+                    with gr.Column(scale=1, min_width=80):
+                        # Main voice button
+                        voice_btn = gr.Button(
+                            "🎙️", 
+                            elem_id="voice-btn", 
+                            variant="secondary",
+                            size="sm",
+                            scale=1
+                        )
+                
+                # Example prompts in the left column
+                gr.Examples(
+                    examples=[
+                        "Dump truck 1 goes to the puddle for inspection, after which all robots avoid the puddle",
+                        "Send Excavator 1 and Dump Truck 1 to the soil area; Excavator 1 will excavate and unload, followed by Dump Truck 1 proceeding to the puddle for unloading."
+                    ], 
+                    inputs=txt
+                )
+                
+            # Right column: DAG visualization and controls
+            with gr.Column(scale=1, elem_classes=["dag-column"]):
+                gr.Markdown("### 📊 Task Dependency Visualization")
+                # DAG visualization display
+                dag_image = gr.Image(label="Task Dependency Graph", visible=True, height=600)
+                
+                # Task plan editing section
+                task_editor = gr.Code(
+                    label="Task Plan JSON Editor",
+                    language="json",
+                    visible=False,
+                    lines=15,
+                    interactive=True
+                )
+                
+                # Control buttons section
+                with gr.Row():
+                    with gr.Column(scale=2):
+                        deployment_status = gr.Markdown("", visible=True)
+                    with gr.Column(scale=1):
+                        with gr.Row():
+                            edit_task_btn = gr.Button(
+                                "📝 Edit Task Plan",
+                                variant="secondary",
+                                visible=False,
+                                size="sm"
+                            )
+                            update_dag_btn = gr.Button(
+                                "🔄 Update DAG Visualization", 
+                                variant="secondary",
+                                visible=False,
+                                size="sm"
+                            )
+                            validate_deploy_btn = gr.Button(
+                                "🔒 Validate & Deploy Task Plan", 
+                                variant="primary", 
+                                visible=False,
+                                size="sm"
+                            )
+        
+        mode_selector.change(update_mode, inputs=[mode_selector, state], outputs=[chatbot_container, state])
+        clear_button.click(gradio_ros_interface.clear_chat, inputs=[state], outputs=[chatbot_container])
+
+        # Handle text input submission
+        async def handle_text_submit(text, state):
+            messages, state, dag_image_path, validate_btn_update = await gradio_ros_interface.predict(text, state)
+            # Show edit button when task plan is generated
+            edit_btn_visible = validate_btn_update.get('visible', False)
+            return (
+                "",  # Clear the text input after submission
+                messages, 
+                state, 
+                dag_image_path, 
+                validate_btn_update,
+                gr.update(visible=edit_btn_visible)  # Show edit button
+            )
+        
+        txt.submit(handle_text_submit, [txt, state], [txt, chatbot_container, state, dag_image, validate_deploy_btn, edit_task_btn])
+
+        # Voice input state management
+        voice_recording = gr.State(False)
+        
+        # Voice button click handler
+        def handle_voice_input(audio, is_recording):
+            logger.info(f"Voice button clicked, current recording state: {is_recording}")
+            
+            if not is_recording:
+                # Start recording state
+                logger.info("Starting recording...")
+                return (
+                    gr.update(value="🔴", elem_classes=["voice-recording"]),  # Change button style
+                    "💬 Recording in progress...",  # Status message
+                    gr.update(visible=True),  # Show status
+                    gr.update(visible=True),  # Show audio component
+                    True,  # Update recording state
+                    ""     # Clear text box
+                )
+            else:
+                # Stop recording and transcribe
+                logger.info("Stopping recording, starting transcription...")
+                if audio is not None and audio != "":
+                    try:
+                        text = audio_to_text(audio)
+                        logger.info(f"Transcription completed: {text}")
+                        return (
+                            gr.update(value="🎙️", elem_classes=[]),  # Restore button style
+                            "✨ Transcription completed!",  # Success message
+                            gr.update(visible=True),   # Show status
+                            gr.update(visible=False),  # Hide audio component
+                            False,  # Reset recording state
+                            text    # Fill in transcribed text
+                        )
+                    except Exception as e:
+                        logger.error(f"Transcription error: {e}")
+                        return (
+                            gr.update(value="🎙️", elem_classes=[]),  # Restore button style
+                            f"❌ Transcription failed: {str(e)}",
+                            gr.update(visible=True),
+                            gr.update(visible=False),
+                            False,
+                            ""
+                        )
+                else:
+                    logger.warning("No audio detected")
+                    return (
+                        gr.update(value="🎙️", elem_classes=[]),  # Restore button style
+                        "⚠️ No audio detected, please record again",
+                        gr.update(visible=True),
+                        gr.update(visible=False),
+                        False,
+                        ""
+                    )
+
+        # Voice button event handling
+        voice_btn.click(
+            handle_voice_input,
+            inputs=[audio_input, voice_recording],
+            outputs=[voice_btn, voice_status, voice_status, audio_input, voice_recording, txt]
+        )
+        
+        # Audio state change listener - automatic prompt
+        def on_audio_change(audio):
+            if audio is not None:
+                logger.info("Audio file detected")
+                return "🎵 Audio detected, you can click the button to complete transcription"
+            return ""
+        
+        audio_input.change(
+            on_audio_change,
+            inputs=[audio_input],
+            outputs=[voice_status]
+        )
+        
+        # Handle task plan editing
+        edit_task_btn.click(
+            gradio_ros_interface.show_task_plan_editor,
+            inputs=[state],
+            outputs=[task_editor, update_dag_btn, validate_deploy_btn, deployment_status]
+        )
+        
+        # Handle DAG update from editor
+        update_dag_btn.click(
+            gradio_ros_interface.update_dag_from_editor,
+            inputs=[task_editor, state],
+            outputs=[dag_image, validate_deploy_btn, task_editor, update_dag_btn, deployment_status, state]
+        )
+        
+        # Handle validation and deployment
+        validate_deploy_btn.click(
+            gradio_ros_interface.validate_and_deploy_task_plan, 
+            inputs=[state], 
+            outputs=[deployment_status, dag_image, validate_deploy_btn, state]
+        )
+
+    demo.launch(server_port=8080, share=True)
+
+if __name__ == "__main__":
+    main()

monitor_topics.sh

@@ -0,0 +1,44 @@
+#!/bin/bash
+
+echo "🔍 ROS Topic Monitor for QA_LLM_Module"
+echo "====================================="
+
+# Check if ROS2 is running
+if ! command -v ros2 &> /dev/null; then
+    echo "❌ ROS2 not found. Please source your ROS2 environment."
+    exit 1
+fi
+
+echo "📋 Active Topics:"
+ros2 topic list | grep -E "(instruction_topic|keywords_topic|tasks_topic|robovla)" || echo "  No relevant topics found"
+
+echo ""
+echo "🤖 Active Nodes:"
+ros2 node list | grep robovla || echo "  No robovla nodes found"
+
+echo ""
+echo "🎯 Topic Information:"
+for topic in "/instruction_topic" "/keywords_topic" "/tasks_topic"; do
+    if ros2 topic list | grep -q "^${topic}$"; then
+        echo "  Topic: $topic"
+        ros2 topic info $topic
+        echo ""
+    fi
+done
+
+echo "📊 Real-time Monitoring (Press Ctrl+C to stop):"
+echo "Monitoring key topics for messages..."
+
+# Monitor in background and display messages
+{
+    ros2 topic echo /instruction_topic &
+    PID1=$!
+    ros2 topic echo /keywords_topic &
+    PID2=$!
+    ros2 topic echo /tasks_topic &
+    PID3=$!
+
+    # Wait for user interrupt
+    trap "kill $PID1 $PID2 $PID3 2>/dev/null; exit" INT
+    wait
+}

prompts/VoxPoser/composer_prompt.txt

@@ -0,0 +1,120 @@
+import numpy as np
+from env_utils import execute, reset_to_default_pose
+from perception_utils import parse_query_obj
+from plan_utils import get_affordance_map, get_avoidance_map, get_velocity_map, get_rotation_map, get_gripper_map
+
+# Query: move ee forward for 10cm.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map(f'a point 10cm in front of {movable.position}')
+execute(movable, affordance_map)
+
+# Query: go back to default.
+reset_to_default_pose()
+
+# Query: move the gripper behind the bowl, and slow down when near the bowl.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 15cm behind the bowl')
+avoidance_map = get_avoidance_map('10cm near the bowl')
+velocity_map = get_velocity_map('slow down when near the bowl')
+execute(movable, affordance_map=affordance_map, avoidance_map=avoidance_map, velocity_map=velocity_map)
+
+# Query: move to the back side of the table while staying at least 5cm from the blue block.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point on the back side of the table')
+avoidance_map = get_avoidance_map('5cm from the blue block')
+execute(movable, affordance_map=affordance_map, avoidance_map=avoidance_map)
+
+# Query: move to the top of the plate and face the plate.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 10cm above the plate')
+rotation_map = get_rotation_map('face the plate')
+execute(movable, affordance_map=affordance_map, rotation_map=rotation_map)
+
+# Query: drop the toy inside container.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 15cm above the container')
+gripper_map = get_gripper_map('close everywhere but open when on top of the container')
+execute(movable, affordance_map=affordance_map, gripper_map=gripper_map)
+
+# Query: push close the topmost drawer.
+movable = parse_query_obj('topmost drawer handle')
+affordance_map = get_affordance_map('a point 30cm into the topmost drawer handle')
+execute(movable, affordance_map=affordance_map)
+
+# Query: push the second to the left block along the red line.
+movable = parse_query_obj('second to the left block')
+affordance_map = get_affordance_map('the red line')
+execute(movable, affordance_map=affordance_map)
+
+# Query: grasp the blue block from the table at a quarter of the speed.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point at the center of blue block')
+velocity_map = get_velocity_map('quarter of the speed')
+gripper_map = get_gripper_map('open everywhere except 1cm around the blue block')
+execute(movable, affordance_map=affordance_map, velocity_map=velocity_map, gripper_map=gripper_map)
+
+# Query: move to the left of the brown block.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 10cm to the left of the brown block')
+execute(movable, affordance_map=affordance_map)
+
+# Query: move to the top of the tray that contains the lemon.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 10cm above the tray that contains the lemon')
+execute(movable, affordance_map=affordance_map)
+
+# Query: close drawer by 5cm.
+movable = parse_query_obj('drawer handle')
+affordance_map = get_affordance_map('a point 5cm into the drawer handle')
+execute(movable, affordance_map=affordance_map)
+
+# Query: move to 5cm on top of the soda can, at 0.5x speed when within 20cm of the wooden mug, and keep at least 15cm away from the wooden mug.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 5cm above the soda can')
+avoidance_map = get_avoidance_map('15cm from the wooden mug')
+velocity_map = get_velocity_map('0.5x speed when within 20cm of the wooden mug')
+execute(movable, affordance_map=affordance_map, avoidance_map=avoidance_map, velocity_map=velocity_map)
+
+# Query: wipe the red dot but avoid the blue block.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('the red dot')
+avoidance_map = get_avoidance_map('10cm from the blue block')
+execute(movable, affordance_map=affordance_map, avoidance_map=avoidance_map)
+
+# Query: grasp the mug from the shelf.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point at the center of the mug handle')
+gripper_map = get_gripper_map('open everywhere except 1cm around the mug handle')
+execute(movable, affordance_map=affordance_map, gripper_map=gripper_map)
+
+# Query: move to 10cm on top of the soup bowl, and 5cm to the left of the soup bowl, while away from the glass, at 0.75x speed.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point 10cm above and 5cm to the left of the soup bowl')
+avoidance_map = get_avoidance_map('10cm from the glass')
+velocity_map = get_velocity_map('0.75x speed')
+execute(movable, affordance_map=affordance_map, avoidance_map=avoidance_map, velocity_map=velocity_map)
+
+# Query: open gripper.
+movable = parse_query_obj('gripper')
+gripper_map = get_gripper_map('open everywhere')
+execute(movable, gripper_map=gripper_map)
+
+# Query: turn counter-clockwise by 180 degrees.
+movable = parse_query_obj('gripper')
+rotation_map = get_rotation_map('turn counter-clockwise by 180 degrees')
+execute(movable, rotation_map=rotation_map)
+
+# Query: sweep all particles to the left side of the table.
+particles = parse_query_obj('particles')
+for particle in particles:
+    movable = particle
+    affordance_map = get_affordance_map('a point on the left side of the table')
+    execute(particle, affordance_map=affordance_map)
+
+# Query: grasp the bottom drawer handle while moving at 0.5x speed.
+movable = parse_query_obj('gripper')
+affordance_map = get_affordance_map('a point at the center of the bottom drawer handle')
+velocity_map = get_velocity_map('0.5x speed')
+rotation_map = get_rotation_map('face the bottom drawer handle')
+gripper_map = get_gripper_map('open everywhere except 1cm around the bottom drawer handle')
+execute(movable, affordance_map=affordance_map, velocity_map=velocity_map, rotation_map=rotation_map, gripper_map=gripper_map)

prompts/VoxPoser/get_affordance_map_prompt.txt

@@ -0,0 +1,153 @@
+import numpy as np
+from perception_utils import parse_query_obj
+from plan_utils import get_empty_affordance_map, set_voxel_by_radius, cm2index
+
+# Query: a point 10cm in front of [10, 15, 60].
+affordance_map = get_empty_affordance_map()
+# 10cm in front of so we add to x-axis
+x = 10 + cm2index(10, 'x')
+y = 15
+z = 60
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point on the right side of the table.
+affordance_map = get_empty_affordance_map()
+table = parse_query_obj('table')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = table.aabb
+center_x, center_y, center_z = table.position
+# right side so y = max_y
+x = center_x
+y = max_y
+z = center_z
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point 20cm on top of the container.
+affordance_map = get_empty_affordance_map()
+container = parse_query_obj('container')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = container.aabb
+center_x, center_y, center_z = container.position
+# 20cm on top of so we add to z-axis
+x = center_x
+y = center_y
+z = max_z + cm2index(20, 'z')
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point 1cm to the left of the brown block.
+affordance_map = get_empty_affordance_map()
+brown_block = parse_query_obj('brown block')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = brown_block.aabb
+center_x, center_y, center_z = brown_block.position
+# 1cm to the left of so we subtract from y-axis
+x = center_x
+y = min_y - cm2index(1, 'y')
+z = center_z
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: anywhere within 20cm of the right most block.
+affordance_map = get_empty_affordance_map()
+right_most_block = parse_query_obj('the right most block')
+set_voxel_by_radius(affordance_map, right_most_block.position, radius_cm=20, value=1)
+
+# Query: a point on the back side of the table.
+affordance_map = get_empty_affordance_map()
+table = parse_query_obj('table')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = table.aabb
+center_x, center_y, center_z = table.position
+# back side so x = min_x
+x = min_x
+y = center_y
+z = center_z
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point on the front right corner of the table.
+affordance_map = get_empty_affordance_map()
+table = parse_query_obj('table')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = table.aabb
+center_x, center_y, center_z = table.position
+# front right corner so x = max_x and y = max_y
+x = max_x
+y = max_y
+z = center_z
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point 30cm into the topmost drawer handle.
+affordance_map = get_empty_affordance_map()
+top_handle = parse_query_obj('topmost drawer handle')
+# negative normal because we are moving into the handle.
+moving_dir = -top_handle.normal
+affordance_xyz = top_handle.position + cm2index(30, moving_dir)
+affordance_map[affordance_xyz[0], affordance_xyz[1], affordance_xyz[2]] = 1
+ret_val = affordance_map
+
+# Query: a point 5cm above the blue block.
+affordance_map = get_empty_affordance_map()
+blue_block = parse_query_obj('blue block')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = blue_block.aabb
+center_x, center_y, center_z = blue_block.position
+# 5cm above so we add to z-axis
+x = center_x
+y = center_y
+z = max_z + cm2index(5, 'z')
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point 20cm away from the leftmost block.
+affordance_map = get_empty_affordance_map()
+leftmost_block = parse_query_obj('leftmost block')
+# positive normal because we are moving away from the block.
+moving_dir = leftmost_block.normal
+affordance_xyz = leftmost_block.position + cm2index(20, moving_dir)
+affordance_map[affordance_xyz[0], affordance_xyz[1], affordance_xyz[2]] = 1
+ret_val = affordance_map
+
+# Query: a point 4cm to the left of and 10cm on top of the tray that contains the lemon.
+affordance_map = get_empty_affordance_map()
+tray_with_lemon = parse_query_obj('tray that contains the lemon')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = tray_with_lemon.aabb
+center_x, center_y, center_z = tray_with_lemon.position
+# 4cm to the left of so we subtract from y-axis, and 10cm on top of so we add to z-axis
+x = center_x
+y = min_y - cm2index(4, 'y')
+z = max_z + cm2index(10, 'z')
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point 10cm to the right of [45 49 66], and 5cm above it.
+affordance_map = get_empty_affordance_map()
+# 10cm to the right of so we add to y-axis, and 5cm above it so we add to z-axis
+x = 45
+y = 49 + cm2index(10, 'y')
+z = 66 + cm2index(5, 'z')
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: the blue circle.
+affordance_map = get_empty_affordance_map()
+blue_circle = parse_query_obj('blue circle')
+affordance_map = blue_circle.occupancy_map
+ret_val = affordance_map
+
+# Query: a point at the center of the blue circle.
+affordance_map = get_empty_affordance_map()
+blue_circle = parse_query_obj('blue circle')
+x, y, z = blue_block.position
+affordance_map[x, y, z] = 1
+ret_val = affordance_map
+
+# Query: a point 10cm above and 5cm to the left of the yellow bowl.
+affordance_map = get_empty_affordance_map()
+yellow_bowl = parse_query_obj('yellow bowl')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = yellow_bowl.aabb
+center_x, center_y, center_z = yellow_bowl.position
+# 10cm above so we add to z-axis, and 5cm to the left of so we subtract from y-axis
+x = center_x
+y = min_y - cm2index(5, 'y')
+z = max_z + cm2index(10, 'z')
+affordance_map[x, y, z] = 1
+ret_val = affordance_mapS

prompts/VoxPoser/get_avoidance_map_prompt.txt

@@ -0,0 +1,36 @@
+import numpy as np
+from perception_utils import parse_query_obj
+from plan_utils import get_empty_avoidance_map, set_voxel_by_radius, cm2index
+
+# Query: 10cm from the bowl.
+avoidance_map = get_empty_avoidance_map()
+bowl = parse_query_obj('bowl')
+set_voxel_by_radius(avoidance_map, bowl.position, radius_cm=10, value=1)
+ret_val = avoidance_map
+
+# Query: 20cm near the mug.
+avoidance_map = get_empty_avoidance_map()
+mug = parse_query_obj('mug')
+set_voxel_by_radius(avoidance_map, mug.position, radius_cm=20, value=1)
+ret_val = avoidance_map
+
+# Query: 20cm around the mug and 10cm around the bowl.
+avoidance_map = get_empty_avoidance_map()
+mug = parse_query_obj('mug')
+set_voxel_by_radius(avoidance_map, mug.position, radius_cm=20, value=1)
+bowl = parse_query_obj('bowl')
+set_voxel_by_radius(avoidance_map, bowl.position, radius_cm=10, value=1)
+ret_val = avoidance_map
+
+# Query: 10cm from anything fragile.
+avoidance_map = get_empty_avoidance_map()
+fragile_objects = parse_query_obj('anything fragile')
+for obj in fragile_objects:
+    set_voxel_by_radius(avoidance_map, obj.position, radius_cm=10, value=1)
+ret_val = avoidance_map
+
+# Query: 10cm from the blue circle.
+avoidance_map = get_empty_avoidance_map()
+blue_circle = parse_query_obj('blue circle')
+set_voxel_by_radius(avoidance_map, blue_circle.position, radius_cm=10, value=1)
+ret_val = avoidance_map

prompts/VoxPoser/get_gripper_map_prompt.txt

@@ -0,0 +1,48 @@
+import numpy as np
+from perception_utils import parse_query_obj
+from plan_utils import get_empty_gripper_map, set_voxel_by_radius, cm2index
+
+# Query: open everywhere except 1cm around the green block.
+gripper_map = get_empty_gripper_map()
+# open everywhere
+gripper_map[:, :, :] = 1
+# close when 1cm around the green block
+green_block = parse_query_obj('green block')
+set_voxel_by_radius(gripper_map, green_block.position, radius_cm=1, value=0)
+ret_val = gripper_map
+
+# Query: close everywhere but open when on top of the back left corner of the table.
+gripper_map = get_empty_gripper_map()
+# close everywhere
+gripper_map[:, :, :] = 0
+# open when on top of the back left corner of the table
+table = parse_query_obj('table')
+(min_x, min_y, min_z), (max_x, max_y, max_z) = table.aabb
+center_x, center_y, center_z = table.position
+# back so x = min_x, left so y = min_y, top so we add to z
+x = min_x
+y = min_y
+z = max_z + cm2index(15, 'z')
+set_voxel_by_radius(gripper_map, (x, y, z), radius_cm=10, value=1)
+ret_val = gripper_map
+
+# Query: always open except when you are on the right side of the table.
+gripper_map = get_empty_gripper_map()
+# always open
+gripper_map[:, :, :] = 1
+# close when you are on the right side of the table
+table = parse_query_obj('table')
+center_x, center_y, center_z = table.position
+# right side so y is greater than center_y
+gripper_map[:, center_y:, :] = 0
+
+# Query: always close except when you are on the back side of the table.
+gripper_map = get_empty_gripper_map()
+# always close
+gripper_map[:, :, :] = 0
+# open when you are on the back side of the table
+table = parse_query_obj('table')
+center_x, center_y, center_z = table.position
+# back side so x is less than center_x
+gripper_map[:center_x, :, :] = 1
+ret_val = gripper_map

prompts/VoxPoser/get_rotation_map_prompt.txt

@@ -0,0 +1,53 @@
+import numpy as np
+from plan_utils import get_empty_rotation_map, set_voxel_by_radius, cm2index, vec2quat
+from perception_utils import parse_query_obj
+from transforms3d.euler import euler2quat, quat2euler
+from transforms3d.quaternions import qmult, qinverse
+
+# Query: face the support surface of the bowl.
+rotation_map = get_empty_rotation_map()
+bowl = parse_query_obj('bowl')
+target_rotation = vec2quat(-bowl.normal)
+rotation_map[:, :, :] = target_rotation
+ret_val = rotation_map
+
+# Query: face the table when within 30cm from table center.
+rotation_map = get_empty_rotation_map()
+table = parse_query_obj('table')
+table_center = table.position
+target_rotation = vec2quat(-table.normal)
+set_voxel_by_radius(rotation_map, table_center, radius_cm=30, value=target_rotation)
+ret_val = rotation_map
+
+# Query: face the blue bowl.
+rotation_map = get_empty_rotation_map()
+blue_bowl = parse_query_obj('brown block')
+target_rotation = vec2quat(-blue_bowl.normal)
+rotation_map[:, :, :] = target_rotation
+ret_val = rotation_map
+
+# Query: turn clockwise by 45 degrees when at the center of the beer cap.
+rotation_map = get_empty_rotation_map()
+beer_cap = parse_query_obj('beer cap')
+(x, y, z) = beer_cap.position
+curr_rotation = rotation_map[x, y, z]
+rotation_delta = euler2quat(0, 0, np.pi / 4)
+rotation_map[x, y, z] = qmult(curr_rotation, rotation_delta)
+ret_val = rotation_map
+
+# Query: turn counter-clockwise by 30 degrees.
+rotation_map = get_empty_rotation_map()
+curr_rotation = rotation_map[0, 0, 0]
+rotation_delta = euler2quat(0, 0, -np.pi / 6)
+rotation_map[:, :, :] = qmult(curr_rotation, rotation_delta)
+ret_val = rotation_map
+
+# Query: rotate the gripper to be 45 degrees slanted relative to the plate.
+rotation_map = get_empty_rotation_map()
+plate = parse_query_obj('plate')
+face_plate_quat = vec2quat(-plate.normal)
+# rotate 45 degrees around the x-axis
+rotation_delta = euler2quat(-np.pi / 4, 0, 0)
+target_rotation = qmult(face_plate_quat, rotation_delta)
+rotation_map[:, :, :] = target_rotation
+ret_val = rotation_map

prompts/VoxPoser/get_velocity_map_prompt.txt

@@ -0,0 +1,31 @@
+import numpy as np
+from plan_utils import get_empty_velocity_map, set_voxel_by_radius, cm2index
+from perception_utils import parse_query_obj
+
+# Query: faster when on the right side of the table and slower when on the left side of the table.
+velocity_map = get_empty_velocity_map()
+table = parse_query_obj('table')
+center_x, center_y, center_z = table.position
+# faster on right side so 1.5 when y > center_y, slower on left side so 0.5 when y < center_y
+velocity_map[:, center_y:, :] = 1.5
+velocity_map[:, :center_y, :] = 0.5
+ret_val = velocity_map
+
+# Query: slow down by a quarter.
+velocity_map = get_empty_velocity_map()
+velocity_map[:] = 0.75
+ret_val = velocity_map
+
+# Query: slow down by a half when you're near anything fragile (objects: ['block', 'fork', 'mug', 'bowl', 'chips']).
+velocity_map = get_empty_velocity_map()
+mug = parse_query_obj('mug')
+set_voxel_by_radius(velocity_map, mug.position, radius_cm=10, value=0.5)
+bowl = parse_query_obj('bowl')
+set_voxel_by_radius(velocity_map, bowl.position, radius_cm=10, value=0.5)
+ret_val = velocity_map
+
+# Query: quarter of the speed when within 9cm from the yellow line.
+velocity_map = get_empty_velocity_map()
+yellow_line = parse_query_obj('yellow_line')
+set_voxel_by_radius(velocity_map, yellow_line.position, radius_cm=9, value=0.25)
+ret_val = velocity_map

prompts/VoxPoser/parse_query_obj_prompt.txt

@@ -0,0 +1,66 @@
+import numpy as np
+from perception_utils import detect
+
+objects = ['green block', 'cardboard box']
+# Query: gripper.
+gripper = detect('gripper')
+ret_val = gripper
+
+objects = ['handle1', 'handle2', 'egg1', 'egg2', 'plate']
+# Query: topmost handle.
+handle1 = detect('handle1')
+handle2 = detect('handle2')
+if handle1.position[2] > handle2.position[2]:
+    top_handle = handle1
+else:
+    top_handle = handle2
+ret_val = top_handle
+
+objects = ['vase', 'napkin box', 'mask']
+# Query: table.
+table = detect('table')
+ret_val = table
+
+objects = ['brown line', 'red block', 'monitor']
+# Query: brown line.
+brown_line = detect('brown line')
+ret_val = brown_line
+
+objects = ['green block', 'cup holder', 'black block']
+# Query: any block.
+block = detect('green block')
+ret_val = block
+
+objects = ['mouse', 'yellow bowl', 'brown bowl', 'sticker']
+# Query: bowl closest to the sticker.
+yellow_bowl = detect('yellow bowl')
+brown_bowl = detect('brown bowl')
+sticker = detect('sticker')
+if np.linalg.norm(yellow_bowl.position - sticker.position) < np.linalg.norm(brown_bowl.position - sticker.position):
+    closest_bowl = yellow_bowl
+else:
+    closest_bowl = brown_bowl
+ret_val = closest_bowl
+
+objects = ['grape', 'wood tray', 'strawberry', 'white tray', 'blue tray', 'bread']
+# Query: tray that contains the bread.
+wood_tray = detect('wood tray')
+white_tray = detect('white tray')
+bread = detect('bread')
+if np.linalg.norm(wood_tray.position - bread.position) < np.linalg.norm(white_tray.position - bread.position):
+    tray_with_bread = wood_tray
+else:
+    tray_with_bread = white_tray
+ret_val = tray_with_bread
+
+objects = ['glass', 'vase', 'plastic bottle', 'block', 'phone case']
+# Query: anything fragile.
+fragile_items = []
+for obj in ['glass', 'vase']:
+    item = detect(obj)
+    fragile_items.append(item)
+ret_val = fragile_items
+
+objects = ['blue block', 'red block']
+# Query: green block.
+ret_val = None

prompts/VoxPoser/planner_prompt.txt

@@ -0,0 +1,115 @@
+import numpy as np
+from env_utils import execute
+from perception_utils import parse_query_obj
+import action_utils import composer
+
+objects = ['blue block', 'yellow block', 'mug']
+# Query: place the blue block on the yellow block, and avoid the mug at all time.
+composer("grasp the blue block while keeping at least 15cm away from the mug")
+composer("back to default pose")
+composer("move to 5cm on top of the yellow block while keeping at least 15cm away from the mug")
+composer("open gripper")
+# done
+
+objects = ['airpods', 'drawer']
+# Query: Open the drawer slowly.
+composer("grasp the drawer handle, at 0.5x speed")
+composer("move away from the drawer handle by 25cm, at 0.5x speed")
+composer("open gripper, at 0.5x speed")
+# done
+
+objects = ['tissue box', 'tissue', 'bowl']
+# Query: Can you pass me a tissue and place it next to the bowl?
+composer("grasp the tissue")
+composer("back to default pose")
+composer("move to 10cm to the right of the bowl")
+composer("open gripper")
+composer("back to default pose")
+# done
+
+objects = ['charger', 'outlet']
+# Query: unplug the charger from the wall.
+composer("grasp the charger")
+composer("back to default pose")
+# done
+
+objects = ['grape', 'lemon', 'drill', 'router', 'bread', 'tray']
+# Query: put the sweeter fruit in the tray that contains the bread.
+composer("grasp the grape")
+composer("back to default pose")
+composer("move to the top of the tray that contains the bread")
+composer("open gripper")
+# done
+
+objects = ['marbles', 'tray', 'broom']
+# Query: Can you sweep the marbles into the tray?
+composer("grasp the broom")
+composer("back to default pose")
+composer("push the marbles into the tray")
+# done
+
+objects = ['orange', 'QR code', 'lemon', 'drawer']
+# Query: put the sour fruit into the top drawer.
+composer("grasp the top drawer handle")
+composer("move away from the top drawer handle by 25cm")
+composer("open gripper")
+composer("back to default pose")
+composer("grasp the lemon")
+composer("move to 10cm on top of the top drawer")
+composer("open gripper")
+# done
+
+objects = ['fridge', 'hot soup']
+# Query: Open the fridge door and be careful around the hot soup.
+composer("grasp the fridge handle and keep at least 15cm away from the hot soup")
+composer("move away from the fridge handle by 25cm and keep at least 15cm away from the hot soup")
+composer("open gripper")
+# done
+
+objects = ['cyan bowl', 'yellow bowl', 'box', 'ice cream']
+# Query: move to the top of the cyan bowl.
+composer("move to the top of the cyan bowl")
+# done
+
+objects = ['drawer', 'umbrella']
+# Query: close the drawer.
+composer("push close the drawer handle by 25cm")
+# done
+
+objects = ['iPhone', 'airpods']
+# Query: slide the iPhone towards the airpods.
+composer("push the iPhone towards the airpods")
+# done
+
+objects = ['plate', 'steak', 'fork', 'knife', 'spoon']
+# Query: Could you please set up the fork for the steak for me?
+composer("grasp the fork")
+composer("back to default pose")
+composer("move to 10cm to the right of the plate")
+composer("open gripper")
+composer("back to default pose")
+# done
+
+objects = ['lamp', 'switch']
+# Query: Turn off the lamp.
+composer("close the gripper")
+composer("move to the center of the switch")
+composer("back to default pose")
+# done
+
+objects = ['beer']
+# Query: turn close the beer.
+composer("grasp the beer cap")
+composer("turn clockwise by 180 degrees")
+composer("back to default pose")
+# done
+
+objects = ['steak', 'grill', 'plate']
+# Query: Take the steak out of the grill and put it flat on the plate.
+composer("grasp the steak")
+composer("back to default pose")
+composer("rotate the gripper to be 45 degrees slanted relative to the plate")
+composer("move to 10cm on top of the plate")
+composer("open gripper")
+composer("back to default pose")
+# done

prompts/swarm/composer_prompt.txt

@@ -0,0 +1,128 @@
+# Query: Move the soil from place A to B.
+{
+  "tasks": [
+    {
+      "task": "Move excavator and dump truck to point A."
+    },
+    {
+      "task": "Begin cycle of digging, loading, transporting, and unloading until all soil is moved from point A to point B."
+    },
+    {
+      "task": "Wait for excavator to reach point A."
+    },
+    {
+      "task": "Start digging with excavator at point A."
+    },
+    {
+      "task": "Wait for excavator to finish digging and for dump truck to reach point A."
+    },
+    {
+      "task": "Load dug soil into dump truck at point A."
+    },
+    {
+      "task": "Wait for soil loading to complete at point A."
+    },
+    {
+      "task": "Transport soil with dump truck from point A to point B."
+    },
+    {
+      "task": "Unload soil at point B."
+    },
+    {
+      "task": "Return dump truck to point A."
+    },
+    {
+      "task": "Repeat the cycle as necessary until completion."
+    },
+    {
+      "task": "After all soil is moved, return both excavator and dump truck to their starting positions."
+    }
+  ]
+}
+# done
+
+# Query: Move gravel from Site C to Site D.
+{
+  "tasks": [
+    {
+      "task": "Move excavator and dump truck to Site C."
+    },
+    {
+      "task": "Begin cycle of digging, loading, transporting, and unloading until all gravel is moved from Site C to Site D."
+    },
+    {
+      "task": "Wait for excavator to reach Site C."
+    },
+    {
+      "task": "Start digging with excavator at Site C."
+    },
+    {
+      "task": "Wait for excavator to finish digging and for dump truck to reach Site C."
+    },
+    {
+      "task": "Load dug gravel into dump truck at Site C."
+    },
+    {
+      "task": "Wait for gravel loading to complete at Site C."
+    },
+    {
+      "task": "Transport gravel with dump truck from Site C to Site D."
+    },
+    {
+      "task": "Unload gravel at Site D."
+    },
+    {
+      "task": "Return dump truck to Site C."
+    },
+    {
+      "task": "Repeat the cycle as necessary until completion."
+    },
+    {
+      "task": "After all gravel is moved, return both excavator and dump truck to their starting positions."
+    }
+  ]
+}
+# done
+
+# Query: Clean up debris at a construction site.
+{
+  "tasks": [
+    {
+      "task": "Move excavator and hauler to the construction site."
+    },
+    {
+      "task": "Begin cycle of cleaning, loading, transporting, and unloading until the site is clean."
+    },
+    {
+      "task": "Wait for excavator and hauler to reach the construction site."
+    },
+    {
+      "task": "Start cleaning debris with the excavator at the construction site."
+    },
+    {
+      "task": "Wait for excavator to finish cleaning and for hauler to arrive."
+    },
+    {
+      "task": "Load cleaned debris into the hauler."
+    },
+    {
+      "task": "Wait for loading to complete."
+    },
+    {
+      "task": "Transport debris to a designated waste facility."
+    },
+    {
+      "task": "Unload debris at the waste facility."
+    },
+    {
+      "task": "Return the hauler to the construction site."
+    },
+    {
+      "task": "Repeat the cycle until the site is clean."
+    },
+    {
+      "task": "After the site is clean, return all equipment to their starting positions."
+    }
+  ]
+}
+# done

prompts/swarm/dart.txt

@@ -0,0 +1,169 @@
+# Query: Ensure all robots avoid the puddle.
+{
+  "tasks": [
+    {
+      "task": "avoid_areas_for_all_robots_1",
+      "instruction_function": {
+        "name": "avoid_areas_for_all_robots",
+        "robot_type": ["dump_truck", "excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 1 and 2 avoid the puddle.
+{
+  "tasks": [
+    {
+      "task": "avoid_areas_for_specific_robots_1",
+      "instruction_function": {
+        "name": "avoid_areas_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 1 and 2 go to the puddle area.
+{
+  "tasks": [
+    {
+      "task": "target_area_for_specific_robots_1",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: All robots return to start.
+{
+  "tasks": [
+    {
+      "task": "return_to_start_for_all_robots_1",
+      "instruction_function": {
+        "name": "return_to_start_for_all_robots",
+        "robot_type": ["dump_truck", "excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavator 1 performs excavation.
+{
+  "tasks": [
+    {
+      "task": "Excavation_1",
+      "instruction_function": {
+        "name": "Excavation",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: All dump truck perform loading.
+{
+  "tasks": [
+    {
+      "task": "DumpLoading_1",
+      "instruction_function": {
+        "name": "DumpLoading",
+        "robot_type": ["dump_truck"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Truck 1 go to obstacle.
+{
+  "tasks": [
+    {
+      "task": "target_area_for_specific_robots_1",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": [],
+        "start_flag": "",
+        "stop_flag": "end_move_equipment",
+        "parallel_flag": false,
+        "condition": "",
+      },
+      "object_keywords": ["obstacle1"]
+    }
+  ]
+}
+# done
+
+# Query: Two trucks go to the puddle, then load, and then return to their initial position.
+{
+  "tasks": [
+    {
+      "task": "target_area_for_specific_robots_1",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_type": ["dump_truck"],
+        "robot_count": 2,
+        "dependencies": [],
+        "start_flag": "",
+        "stop_flag": "end_move_equipment",
+        "parallel_flag": false,
+        "condition": "",
+        "object_keywords": []
+      },
+      "object_keywords": ["puddle1"]
+    },
+    {
+      "task": "DumpLoading_1",
+      "instruction_function": {
+        "name": "DumpLoading",
+        "robot_type": ["dump_truck"],
+        "robot_count": 2,
+        "dependencies": ["target_area_for_specific_robots_1"],
+        "start_flag": "start_loading_soil",
+        "stop_flag": "end_loading_soil",
+        "parallel_flag": true,
+        "condition": "",
+        "object_keywords": []
+      }
+    },
+    {
+      "task": "return_to_start_for_specific_robots_1",
+      "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "robot_type": ["dump_truck"],
+        "robot_count": 2,
+        "dependencies": ["DumpLoading_1"],
+        "start_flag": "start_return_truck",
+        "stop_flag": "end_return_truck",
+        "parallel_flag": true,
+        "condition": "",
+        "object_keywords": []
+      },
+      "object_keywords": ["puddle1"]
+    }
+  ]
+}
+# done

prompts/swarm/instruction_translator_prompt.txt

@@ -0,0 +1,125 @@
+# Query: move_equipment_to_location
+{
+    "instruction_function": {
+        "name": "move_equipment_to_location",
+        "parameters": {
+            "equipment_ids": ["excavator1", "truck1"],
+            "location": "soil_pile"
+        },
+        "start_flag": "start_move_equipment",
+        "stop_flag": "end_move_equipment",
+        "dependencies": [],
+        "parallel_flag": false
+    }
+}
+# done
+
+# Query: start_digging_at_location
+{
+    "instruction_function": {
+        "name": "Excavation",
+        "parameters": {
+            "location": "soil_pile"
+        },
+        "start_flag": "start_digging",
+        "stop_flag": "end_digging",
+        "dependencies": ["end_move_equipment"],
+        "parallel_flag": true
+    }
+}
+# done
+
+# Query: load_soil_into_truck
+{
+    "instruction_function": {
+        "name": "DumpLoading",
+        "parameters": {
+            "truck_id": "truck1",
+            "location": "soil_pile"
+        },
+        "start_flag": "start_loading_soil",
+        "stop_flag": "end_loading_soil",
+        "dependencies": ["start_digging"],
+        "parallel_flag": true
+    }
+}
+# done
+
+# Query: transport_soil
+{
+    "instruction_function": {
+        "name": "transport_soil",
+        "parameters": {
+            "truck_id": "truck1",
+            "from_location": "soil_pile",
+            "to_location": "pit"
+        },
+        "start_flag": "start_transporting_soil",
+        "stop_flag": "end_transporting_soil",
+        "dependencies": ["start_loading_soil"],
+        "parallel_flag": true
+    }
+}
+# done
+
+# Query: unload_soil_at_location
+{
+    "instruction_function": {
+        "name": "DumpUnloading",
+        "parameters": {
+            "truck_id": "truck1",
+            "location": "pit"
+        },
+        "start_flag": "start_unloading_soil",
+        "stop_flag": "end_unloading_soil",
+        "dependencies": ["end_transporting_soil"],
+        "parallel_flag": false
+    }
+}
+# done
+
+# Query: return_truck_to_location
+{
+    "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "parameters": {
+            "robot_ids": ["truck1"],
+            "location": "soil_pile"
+        },
+        "start_flag": "start_return_truck",
+        "stop_flag": "end_return_truck",
+        "dependencies": ["end_unloading_soil"],
+        "parallel_flag": true
+    }
+}
+# done
+
+# Query: check_if_half_soil_moved
+{
+    "instruction_function": {
+        "name": "check_if_half_soil_moved",
+        "parameters": {
+            "condition": "Soil moved is less than 50%"
+        },
+        "start_flag": "start_check_half_moved",
+        "stop_flag": "end_check_half_moved",
+        "dependencies": ["end_return_truck"],
+        "parallel_flag": false
+    }
+}
+# done
+
+# Query: return_equipment_to_starting_positions
+{
+    "instruction_function": {
+        "name": "return_to_start_for_all_robots",
+        "parameters": {
+            "equipment_ids": ["excavator1", "truck1"]
+        },
+        "start_flag": "start_return_equipment",
+        "stop_flag": "end_return_equipment",
+        "dependencies": ["end_check_half_moved"],
+        "parallel_flag": false
+    }
+}
+# done

prompts/swarm/planner_prompt.txt

@@ -0,0 +1,109 @@
+# Query: Ensure all robots avoid the puddle.
+{
+    "instruction_function": {
+        "name": "avoid_areas_for_all_robots"
+    },
+    "clip_keywords": ["The puddle"]
+}
+# done
+
+# Query: Robots 2 and 3 avoid the puddle.
+{
+    "instruction_function": {
+        "name": "avoid_areas_for_specific_robots",
+        "robot_ids": [2, 3]
+    },
+    "clip_keywords": ["The puddle"]
+}
+# done
+
+# Query: All robots go to the puddle area.
+{
+    "instruction_function": {
+        "name": "target_area_for_all_robots"
+    },
+    "clip_keywords": ["The puddle"]
+}
+# done
+
+# Query: Robots 1 and 4 go to the puddle area.
+{
+    "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": [1, 4]
+    },
+    "clip_keywords": ["The puddle"]
+}
+# done
+
+# Query: All robots resume operations at the puddle.
+{
+    "instruction_function": {
+        "name": "allow_areas_for_all_robots"
+    },
+    "clip_keywords": ["The puddle"]
+}
+# done
+
+# Query: Robots 5 and 6 operate in the puddle area.
+{
+    "instruction_function": {
+        "name": "allow_areas_for_specific_robots",
+        "robot_ids": [5, 6]
+    },
+    "clip_keywords": ["The puddle"]
+}
+# done
+
+# Query: All robots return to start.
+{
+    "instruction_function": {
+        "name": "return_to_start_for_all_robots"
+    }
+}
+# done
+
+# Query: Robots 3 and 4 return to charging stations.
+{
+    "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "robot_ids": [3, 4]
+    }
+}
+# done
+
+# Query: Excavator performs excavation.
+{
+    "instruction_function": {
+        "name": "Excavation",
+        "robot_type": "excavator"
+    }
+}
+# done
+
+# Query: Excavator performs unloading.
+{
+    "instruction_function": {
+        "name": "ExcavatorUnloading",
+        "robot_type": "excavator"
+    }
+}
+# done
+
+# Query: Dump truck performs loading.
+{
+    "instruction_function": {
+        "name": "DumpLoading",
+        "robot_type": "dump"
+    }
+}
+# done
+
+# Query: Dump truck performs unloading.
+{
+    "instruction_function": {
+        "name": "DumpUnloading",
+        "robot_type": "dump"
+    }
+}
+# done

prompts/swarm/task_2_commannd_prompt.txt

@@ -0,0 +1,564 @@
+# Query: Ensure all robots avoid the puddle.
+{
+  "tasks": [
+    {
+      "task": "ensure_all_robots_avoid_puddle",
+      "instruction_function": {
+        "name": "avoid_areas_for_all_robots",
+        "robot_type": ["dump_truck", "excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Trucks 2 and 3 avoid the puddle.
+{
+  "tasks": [
+    {
+      "task": "trucks_2_and_3_avoid_puddle",
+      "instruction_function": {
+        "name": "avoid_areas_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_02", "robot_dump_truck_03"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 1 and 2 avoid the puddle.
+{
+  "tasks": [
+    {
+      "task": "excavators_1_and_2_avoid_puddle",
+      "instruction_function": {
+        "name": "avoid_areas_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: All robots go to the puddle area.
+{
+  "tasks": [
+    {
+      "task": "all_robots_go_to_puddle_area",
+      "instruction_function": {
+        "name": "target_area_for_all_robots",
+        "robot_type": ["dump_truck", "excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 1 and 2 go to the puddle area.
+{
+  "tasks": [
+    {
+      "task": "excavators_1_and_2_go_to_puddle_area",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Truck 1 and 4 go to the puddle area.
+{
+  "tasks": [
+    {
+      "task": "truck_1_and_4_go_to_puddle_area",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_01", "robot_dump_truck_04"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Truck 1 and 4 go to the obstacle area.
+{
+  "tasks": [
+    {
+      "task": "truck_1_and_4_go_to_obstacle_area",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_01", "robot_dump_truck_04"],
+        "dependencies": [],
+        "object_keywords": ["obstacle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 1 and 2 go to the puddle area.
+{
+  "tasks": [
+    {
+      "task": "excavators_1_and_2_go_to_puddle_area",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: All robots resume operations at the puddle.
+{
+  "tasks": [
+    {
+      "task": "all_robots_resume_operations_at_puddle",
+      "instruction_function": {
+        "name": "allow_areas_for_all_robots",
+        "robot_type": ["dump_truck", "excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Trucks 5 and 6 operate in the puddle area.
+{
+  "tasks": [
+    {
+      "task": "trucks_5_and_6_operate_in_puddle_area",
+      "instruction_function": {
+        "name": "allow_areas_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_05", "robot_dump_truck_06"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 1 and 2 operate in the puddle area.
+{
+  "tasks": [
+    {
+      "task": "excavators_1_and_2_operate_in_puddle_area",
+      "instruction_function": {
+        "name": "allow_areas_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": ["puddle1"]
+      }
+    }
+  ]
+}
+# done
+
+# Query: All robots return to start.
+{
+  "tasks": [
+    {
+      "task": "all_robots_return_to_start",
+      "instruction_function": {
+        "name": "return_to_start_for_all_robots",
+        "robot_type": ["dump_truck", "excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Trucks 3 and 4 return to charging stations.
+{
+  "tasks": [
+    {
+      "task": "trucks_3_and_4_return_to_charging_stations",
+      "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_03", "robot_dump_truck_04"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavators 3 and 4 return to charging stations.
+{
+  "tasks": [
+    {
+      "task": "excavators_1_and_2_return_to_charging_stations",
+      "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "robot_ids": ["robot_excavator_01", "robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: All excavator perform excavation.
+{
+  "tasks": [
+    {
+      "task": "all_excavator_perform_excavation",
+      "instruction_function": {
+        "name": "Excavation",
+        "robot_type": ["excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavator 1 performs excavation.
+{
+  "tasks": [
+    {
+      "task": "excavator_1_performs_excavation",
+      "instruction_function": {
+        "name": "Excavation",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavator 2 performs excavation.
+{
+  "tasks": [
+    {
+      "task": "excavator_2_performs_excavation",
+      "instruction_function": {
+        "name": "Excavation",
+        "robot_ids": ["robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: All excavator perform unloading.
+{
+  "tasks": [
+    {
+      "task": "all_excavator_perform_unloading",
+      "instruction_function": {
+        "name": "ExcavatorUnloading",
+        "robot_type": ["excavator"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavator 1 performs unloading.
+{
+  "tasks": [
+    {
+      "task": "excavator_1_performs_unloading",
+      "instruction_function": {
+        "name": "ExcavatorUnloading",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Excavator 2 performs unloading.
+{
+  "tasks": [
+    {
+      "task": "excavator_2_performs_unloading",
+      "instruction_function": {
+        "name": "ExcavatorUnloading",
+        "robot_ids": ["robot_excavator_02"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: All dump truck perform loading.
+{
+  "tasks": [
+    {
+      "task": "all_dump_truck_perform_loading",
+      "instruction_function": {
+        "name": "DumpLoading",
+        "robot_type": ["dump_truck"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Dump truck 3 performs loading.
+{
+  "tasks": [
+    {
+      "task": "dump_truck_3_performs_loading",
+      "instruction_function": {
+        "name": "DumpLoading",
+        "robot_ids": ["robot_dump_truck_03"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Dump truck 1 performs loading.
+{
+  "tasks": [
+    {
+      "task": "dump_truck_1_performs_loading",
+      "instruction_function": {
+        "name": "DumpLoading",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+
+
+# Query: All dump truck perform unloading.
+{
+  "tasks": [
+    {
+      "task": "all_dump_truck_perform_unloading",
+      "instruction_function": {
+        "name": "DumpUnloading",
+        "robot_type": ["dump_truck"],
+        "robot_count": "all",
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Dump truck 5 performs unloading.
+{
+  "tasks": [
+    {
+      "task": "dump_truck_5_performs_unloading",
+      "instruction_function": {
+        "name": "DumpUnloading",
+        "robot_ids": ["robot_dump_truck_05"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Dump truck 2 performs unloading.
+{
+  "tasks": [
+    {
+      "task": "dump_truck_2_performs_unloading",
+      "instruction_function": {
+        "name": "DumpUnloading",
+        "robot_ids": ["robot_dump_truck_02"],
+        "dependencies": [],
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Truck1 go to puddle.
+{
+  "tasks": [
+    {
+      "task": "move_equipment_to_location",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": [],
+        "start_flag": "",
+        "stop_flag": "end_move_equipment",
+        "parallel_flag": false,
+        "condition": "",
+      },
+      "object_keywords": ["puddle1"]
+    }
+  ]
+}
+# done
+
+# Query: After excavator 1 finishes excavation, excavator 2 performs excavation.
+{
+  "tasks": [
+    {
+      "task": "excavator_1_excavation",
+      "instruction_function": {
+        "name": "Excavation",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "start_flag": "",
+        "stop_flag": "end_excavator_1",
+        "parallel_flag": false,
+        "condition": "",
+        "object_keywords": []
+      }
+    },
+    {
+      "task": "excavator_2_excavation",
+      "instruction_function": {
+        "name": "Excavation",
+        "robot_ids": ["robot_excavator_02"],
+        "dependencies": ["excavator_1_excavation"],
+        "start_flag": "excavator_1_excavation",
+        "stop_flag": "end_excavator_2",
+        "parallel_flag": false,
+        "condition": "",
+        "object_keywords": []
+      }
+    }
+  ]
+}
+# done
+
+# Query: Truck 2 goes to the puddle, then Truck 3 goes to the puddle.
+{
+  "tasks": [
+    {
+      "task": "move_truck2_to_location",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_02"],
+        "dependencies": [],
+        "start_flag": "",
+        "stop_flag": "",
+        "parallel_flag": false,
+        "condition": "",
+        "object_keywords": []
+      },
+      "object_keywords": ["puddle1"]
+    },
+    {
+      "task": "move_truck3_to_location",
+      "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "robot_ids": ["robot_dump_truck_03"],
+        "dependencies": ["move_truck2_to_location"],
+        "start_flag": "move_truck2_to_location",
+        "stop_flag": "",
+        "parallel_flag": false,
+        "condition": "",
+        "object_keywords": []
+      },
+      "object_keywords": ["puddle1"]
+    }
+  ]
+}
+# done
+
+# Query: Two trucks go to the puddle, then load, and then return to their initial position.
+{
+  "tasks": [
+    {
+      "task": "move_two_trucks_to_location",
+      "instruction_function": {
+        "name": "target_area_for_specific_robots",
+        "robot_type": ["dump_truck"],
+        "robot_count": 2,
+        "dependencies": [],
+        "start_flag": "",
+        "stop_flag": "end_move_equipment",
+        "parallel_flag": false,
+        "condition": "",
+        "object_keywords": []
+      },
+      "object_keywords": ["puddle1"]
+    },
+    {
+      "task": "two_trucks_perform_loading_soil",
+      "instruction_function": {
+        "name": "DumpLoading",
+        "robot_type": ["dump_truck"],
+        "robot_count": 2,
+        "dependencies": ["move_two_trucks_to_location"],
+        "start_flag": "start_loading_soil",
+        "stop_flag": "end_loading_soil",
+        "parallel_flag": true,
+        "condition": "",
+        "object_keywords": []
+      }
+    },
+    {
+      "task": "return_truck_to_starting_position",
+      "instruction_function": {
+        "name": "return_to_start_for_specific_robots",
+        "robot_type": ["dump_truck"],
+        "robot_count": 2,
+        "dependencies": ["two_trucks_perform_loading_soil"],
+        "start_flag": "start_return_truck",
+        "stop_flag": "end_return_truck",
+        "parallel_flag": true,
+        "condition": "",
+        "object_keywords": []
+      },
+      "object_keywords": ["puddle1"]
+    }
+  ]
+}
+# done

prompts/swarm/task_decomposer_prompt.txt

@@ -0,0 +1,99 @@
+# Query: Move half of the soil from the soil pile to the puddle.
+{
+  "tasks": [
+    {
+      "task": "move_equipment_to_location",
+      "parameters": {
+        "equipment_ids": ["excavator1", "truck1"],
+        "location": "soil_pile"
+      },
+      "start_flag": "",
+      "stop_flag": "end_move_equipment",
+      "dependencies": [],
+      "parallel_flag": false,
+      "condition": "Soil moved is less than 50%"
+    },
+    {
+      "task": "start_digging_at_location",
+      "parameters": {
+        "location": "soil_pile"
+      },
+      "start_flag": "start_digging",
+      "stop_flag": "end_digging",
+      "dependencies": ["end_move_equipment"],
+      "parallel_flag": true,
+      "condition": "Soil moved is less than 50%"
+    },
+    {
+      "task": "load_soil_into_truck",
+      "parameters": {
+        "truck_id": "truck1",
+        "location": "soil_pile"
+      },
+      "start_flag": "start_loading_soil",
+      "stop_flag": "end_loading_soil",
+      "dependencies": ["start_digging"],
+      "parallel_flag": true,
+      "condition": "Soil moved is less than 50%"
+    },
+    {
+      "task": "transport_soil",
+      "parameters": {
+        "truck_id": "truck1",
+        "from_location": "soil_pile",
+        "to_location": "pit"
+      },
+      "start_flag": "start_transporting_soil",
+      "stop_flag": "end_transporting_soil",
+      "dependencies": ["start_loading_soil"],
+      "parallel_flag": true,
+      "condition": "Soil moved is less than 50%"
+    },
+    {
+      "task": "unload_soil_at_location",
+      "parameters": {
+        "truck_id": "truck1",
+        "location": "pit"
+      },
+      "start_flag": "start_unloading_soil",
+      "stop_flag": "end_unloading_soil",
+      "dependencies": ["end_transporting_soil"],
+      "parallel_flag": false,
+      "condition": "Soil moved is less than 50%"
+    },
+    {
+      "task": "return_truck_to_location",
+      "parameters": {
+        "truck_id": "truck1",
+        "location": "soil_pile"
+      },
+      "start_flag": "start_return_truck",
+      "stop_flag": "end_return_truck",
+      "dependencies": ["end_unloading_soil"],
+      "parallel_flag": true,
+      "condition": "Soil moved is less than 50%"
+    },
+    {
+      "task": "check_if_half_soil_moved",
+      "parameters": {
+        "condition": "Soil moved is less than 50%"
+      },
+      "start_flag": "start_check_half_moved",
+      "stop_flag": "end_check_half_moved",
+      "dependencies": ["end_return_truck"],
+      "parallel_flag": false
+    },
+    {
+      "task": "return_equipment_to_starting_positions",
+      "parameters": {
+        "equipment_ids": ["excavator1", "truck1"]
+      },
+      "start_flag": "start_return_equipment",
+      "stop_flag": "end_return_equipment",
+      "dependencies": ["end_check_half_moved"],
+      "parallel_flag": false,
+      "condition": "Half soil moved"
+    }
+  ]
+}
+# done

readme.md

@@ -0,0 +1,88 @@
+---
+title: DART-LLM_Task_Decomposer
+app_file: gradio_llm_interface.py
+sdk: gradio
+sdk_version: 5.38.0
+---
+# QA_LLM_Module
+
+QA_LLM_Module is a core component of the DART-LLM (Dependency-Aware Multi-Robot Task Decomposition and Execution) system. It provides an intelligent interface for parsing natural language instructions into structured, dependency-aware task sequences for multi-robot systems. The module supports multiple LLM providers including OpenAI GPT, Anthropic Claude, and LLaMA models, enabling sophisticated task decomposition with explicit handling of dependencies between subtasks.
+
+## Features
+
+- **Multiple LLM Support**: Compatible with OpenAI GPT-4, GPT-3.5-turbo, Anthropic Claude, and LLaMA models
+- **Dependency-Aware Task Decomposition**: Breaks down complex tasks into subtasks with explicit dependencies
+- **Structured Output Format**: Generates standardized JSON output for consistent task parsing
+- **Real-time Processing**: Supports real-time task decomposition and execution
+
+## Installation
+
+
+
+## Installation
+
+You can choose one of the following ways to set up the module:
+
+### Option 1: Install Dependencies Directly (Recommended for Linux/Mac)
+
+1. Clone the repository and install dependencies:
+   ```bash
+   pip install -r requirements.txt
+   ```
+
+2. Configure API keys in environment variables:
+   ```bash
+   export OPENAI_API_KEY="your_openai_key"
+   export ANTHROPIC_API_KEY="your_anthropic_key" 
+   export GROQ_API_KEY="your_groq_key"
+   ```
+
+### Option 2: Use Docker (Recommended for Windows)
+
+For Windows users, it is recommended to use the pre-configured Docker environment to avoid compatibility issues.
+
+1. Clone the Docker repository:
+   ```bash
+   git clone https://github.com/wyd0817/DART_LLM_Docker.git
+   ```
+
+2. Follow the instructions in the [DART_LLM_Docker](https://github.com/wyd0817/DART_LLM_Docker) repository to build and run the container.
+
+3. Configure API keys in environment variables:
+   ```bash
+   export OPENAI_API_KEY="your_openai_key"
+   export ANTHROPIC_API_KEY="your_anthropic_key" 
+   export GROQ_API_KEY="your_groq_key"
+   ```
+
+## Usage
+
+### Example Usage:
+
+Run the main interface with Gradio:
+   ```bash
+   gradio main.py
+   ```
+
+### Output Format:
+
+The module processes natural language instructions into structured JSON output following this format:
+```json
+{
+    "instruction_function": {
+        "name": "<breakdown function 1>",
+        "dependencies": ["<dep 1>", "<dep 2>", "...", "<dep n>"]
+    },
+    "object_keywords": ["<key 1>", "<key 2>", "...", "<key n>"],
+    ...
+}
+
+```
+
+## Configuration
+
+The module can be configured through:
+
+- `config.py`: Model settings and API configurations
+- `prompts/`: Directory containing prompt templates
+- `llm_request_handler.py`: Core logic for handling LLM requests

requirements.txt

@@ -0,0 +1,47 @@
+wandb==0.19.8
+datasets==3.5.0
+tqdm==4.67.1
+tiktoken==0.9.0
+transformers==4.50.3
+deepspeed==0.16.5
+openai==1.69.0
+fastapi==0.115.12
+gradio==5.23.1
+httpx==0.28.1
+loguru==0.7.3
+pydantic==2.11.1
+python-dotenv==1.1.0
+PyYAML==6.0.2
+requests==2.32.3
+six==1.17.0
+uvicorn==0.34.0
+anthropic==0.20.0
+groq==0.20.0
+peft==0.15.1
+protobuf==5.29.4
+scikit-learn==1.6.1
+scipy==1.8.0
+sentencepiece==0.2.0
+fire==0.7.0
+bs4==0.0.1
+zenhan==0.5.2
+mecab-python3==1.0.10
+pyknp==0.6.1
+langchain==0.3.21
+langchain-anthropic==0.3.10
+langchain-community==0.3.20
+langchain-core==0.3.51
+langchain-deepseek==0.1.3
+langchain-groq==0.3.1
+langchain-ollama==0.3.1
+langchain-openai==0.3.11
+langchain-text-splitters==0.3.7
+sentence_transformers==4.0.1
+faiss-gpu==1.7.2
+ipykernel==6.29.5
+jupyter==1.1.1
+jupyterlab==4.3.6
+numpy==1.26.4
+transforms3d==0.4.2
+matplotlib==3.9.4
+networkx==3.4

ros_message_server.py

@@ -0,0 +1,76 @@
+import rclpy
+from std_msgs.msg import String
+from loguru import logger
+import asyncio
+from llm_request_handler import LLMRequestHandler
+from ros_node_publisher import RosNodePublisher
+from json_processor import JsonProcessor
+from config import MODE_CONFIG, ROS_MESSAGE_MODE
+import json
+
+class RosMessageServer(RosNodePublisher):
+    def __init__(self):
+        super().__init__(ROS_MESSAGE_MODE)
+        self.mode_config = MODE_CONFIG[ROS_MESSAGE_MODE]
+        self._subscriptions = []
+        self.received_tasks = []
+        self.llm_handler = LLMRequestHandler(
+            model_version=self.mode_config["model_version"],
+            max_tokens=self.mode_config["max_tokens"],
+            temperature=self.mode_config["temperature"],
+            frequency_penalty=self.mode_config["frequency_penalty"],
+            list_navigation_once=True
+        )
+        self.initial_messages = self.llm_handler.build_initial_messages(self.mode_config["prompt_file"], ROS_MESSAGE_MODE)
+        self.json_processor = JsonProcessor()
+
+        # Initialize subscriptions based on input topics
+        for input_topic in self.mode_config["input_topics"]:
+            subscription = self.create_subscription(
+                String,
+                input_topic,
+                self.listener_callback,
+                10)
+            self._subscriptions.append(subscription)
+
+        logger.info(f"{self.mode_config['display_name']} node initialized.")
+    
+    def listener_callback(self, msg):
+        task_data = msg.data
+        logger.debug(f"Received raw task data: {task_data}")
+        try:
+            # Parse the task data from the message
+            task_data_clean = task_data.strip('"')
+            self.received_tasks.append({"task": task_data_clean})
+            asyncio.run(self.process_tasks())
+        except Exception as e:
+            logger.error(f"Unexpected error: {e}")
+
+    async def process_tasks(self):
+        while self.received_tasks:
+            task = self.received_tasks.pop(0)
+            logger.debug(f"Processing task: {task}")
+            await self.send_to_gpt(task)
+
+    async def send_to_gpt(self, task):
+        prompt = f"# Task: {task}"
+        messages = self.initial_messages + [{"role": "user", "content": prompt}]
+        response = await self.llm_handler.make_completion(messages)
+        if response:
+            self.publish_response(response)
+        else:
+            self.publish_response("Error: Unable to get response.")
+
+    def publish_response(self, response):
+        response_json = self.json_processor.process_response(response)
+        super().publish_response(response_json)
+
+def main(args=None):
+    rclpy.init(args=args)
+    ros_message_server = RosMessageServer()
+    rclpy.spin(ros_message_server)
+    ros_message_server.destroy_node()
+    rclpy.shutdown()
+
+if __name__ == "__main__":
+    main()

ros_node_publisher.py

@@ -0,0 +1,71 @@
+import rclpy
+from rclpy.node import Node
+from std_msgs.msg import String
+from loguru import logger
+import json
+from collections import OrderedDict
+from config import MODE_CONFIG
+
+class RosNodePublisher(Node):
+    def __init__(self, node_type=None):
+        if not rclpy.ok():
+            rclpy.init()
+        super().__init__(f'robovla_{node_type}_node')
+        self.node_type = node_type
+        self.llm_publishers = {}
+        self.mode_config = MODE_CONFIG[node_type] if node_type else None
+        if node_type:
+            self.initialize_node(node_type)
+
+    def initialize_node(self, node_type):
+        self.node_type = node_type
+        self.mode_config = MODE_CONFIG[node_type]
+        for output_topic in self.mode_config['output_topics']:
+            self.llm_publishers[output_topic] = self.create_publisher(String, output_topic, 10)
+        logger.info(f"robovla {self.node_type} node initialized.")
+
+    def destroy_node(self):
+        super().destroy_node()
+        rclpy.shutdown()
+        self.node_type = None
+        logger.info("ROS node destroyed.")
+
+    def get_node_type(self):
+        return self.node_type
+
+    def is_initialized(self):
+        return self.node_type is not None and rclpy.ok()
+
+    def publish_response(self, response_json):
+        try:
+            for json_key, topic in self.mode_config['json_keys'].items():
+                # Handle top-level key
+                if json_key in response_json:
+                    msg = String()
+                    msg.data = json.dumps(response_json[json_key])
+                    self.llm_publishers[topic].publish(msg)
+                    logger.info(f"Published {json_key} to {topic}: {msg.data}")
+                # Handle nested structure logic
+                else:
+                    # Iterate through all tasks to check for nested keys
+                    for task in response_json['tasks']:
+                        if json_key in task:
+                            if json_key == 'instruction_function':
+                                # Merge task content into instruction_function and rename key to task_name
+                                task_copy = OrderedDict()
+                                task_copy['task_name'] = task['task']
+                                for key, value in task[json_key].items():
+                                    task_copy[key] = value
+                                msg = String()
+                                msg.data = json.dumps(task_copy)
+                                logger.info(f"Merged task_name into instruction_function: {msg.data}")
+                            else:
+                                msg = String()
+                                msg.data = json.dumps(task[json_key])
+                            self.llm_publishers[topic].publish(msg)
+                            logger.info(f"Published {json_key} to {topic}: {msg.data}")
+        except Exception as e:
+            logger.error(f"Unexpected error: {e}")
+
+
+

test_dag_visualization.py

@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""
+Test script for DAG visualization functionality
+"""
+
+import sys
+import os
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from dag_visualizer import DAGVisualizer
+import json
+
+def test_single_task():
+    """Test with a single task (no dependencies)"""
+    print("Testing single task visualization...")
+    
+    task_data = {
+        "tasks": [
+            {
+                "task": "target_area_for_specific_robots_1",
+                "instruction_function": {
+                    "name": "target_area_for_specific_robots",
+                    "robot_ids": ["robot_dump_truck_01"],
+                    "dependencies": [],
+                    "object_keywords": ["puddle1"]
+                }
+            }
+        ]
+    }
+    
+    visualizer = DAGVisualizer()
+    image_path = visualizer.create_dag_visualization(task_data)
+    
+    if image_path and os.path.exists(image_path):
+        print(f"✓ Single task visualization created: {image_path}")
+        return True
+    else:
+        print("✗ Failed to create single task visualization")
+        return False
+
+def test_multiple_tasks_with_dependencies():
+    """Test with multiple tasks with dependencies"""
+    print("Testing multiple tasks with dependencies...")
+    
+    task_data = {
+        "tasks": [
+            {
+                "task": "target_area_for_specific_robots_1",
+                "instruction_function": {
+                    "name": "target_area_for_specific_robots",
+                    "robot_ids": ["robot_dump_truck_01"],
+                    "dependencies": [],
+                    "object_keywords": ["puddle1"]
+                }
+            },
+            {
+                "task": "avoid_areas_for_all_robots_1",
+                "instruction_function": {
+                    "name": "avoid_areas_for_all_robots",
+                    "robot_ids": ["robot_dump_truck_01", "robot_excavator_01"],
+                    "dependencies": ["target_area_for_specific_robots_1"],
+                    "object_keywords": ["obstacle1", "obstacle2"]
+                }
+            },
+            {
+                "task": "move_to_position_1",
+                "instruction_function": {
+                    "name": "move_to_position",
+                    "robot_ids": ["robot_excavator_01"],
+                    "dependencies": ["avoid_areas_for_all_robots_1"],
+                    "object_keywords": ["soil_pile"]
+                }
+            }
+        ]
+    }
+    
+    visualizer = DAGVisualizer()
+    image_path = visualizer.create_dag_visualization(task_data)
+    
+    if image_path and os.path.exists(image_path):
+        print(f"✓ Multiple tasks visualization created: {image_path}")
+        return True
+    else:
+        print("✗ Failed to create multiple tasks visualization")
+        return False
+
+def test_simplified_visualization():
+    """Test simplified visualization"""
+    print("Testing simplified visualization...")
+    
+    task_data = {
+        "tasks": [
+            {
+                "task": "excavate_soil_from_pile",
+                "instruction_function": {
+                    "name": "excavate_soil_from_pile",
+                    "robot_ids": ["robot_excavator_01"],
+                    "dependencies": [],
+                    "object_keywords": ["soil_pile"]
+                }
+            },
+            {
+                "task": "transport_soil_to_pit",
+                "instruction_function": {
+                    "name": "transport_soil_to_pit",
+                    "robot_ids": ["robot_dump_truck_01"],
+                    "dependencies": ["excavate_soil_from_pile"],
+                    "object_keywords": ["pit"]
+                }
+            }
+        ]
+    }
+    
+    visualizer = DAGVisualizer()
+    image_path = visualizer.create_simplified_dag_visualization(task_data)
+    
+    if image_path and os.path.exists(image_path):
+        print(f"✓ Simplified visualization created: {image_path}")
+        return True
+    else:
+        print("✗ Failed to create simplified visualization")
+        return False
+
+def test_invalid_data():
+    """Test with invalid data"""
+    print("Testing invalid data handling...")
+    
+    # Test with empty data
+    visualizer = DAGVisualizer()
+    image_path = visualizer.create_dag_visualization({})
+    
+    if image_path is None:
+        print("✓ Invalid data handled correctly (returned None)")
+        return True
+    else:
+        print("✗ Invalid data not handled correctly")
+        return False
+
+def main():
+    """Run all tests"""
+    print("Starting DAG Visualization Tests...")
+    print("=" * 50)
+    
+    tests = [
+        test_single_task,
+        test_multiple_tasks_with_dependencies,
+        test_simplified_visualization,
+        test_invalid_data
+    ]
+    
+    passed = 0
+    total = len(tests)
+    
+    for test in tests:
+        try:
+            if test():
+                passed += 1
+            print()
+        except Exception as e:
+            print(f"✗ Test failed with exception: {e}")
+            print()
+    
+    print("=" * 50)
+    print(f"Tests passed: {passed}/{total}")
+    
+    if passed == total:
+        print("🎉 All tests passed!")
+        return True
+    else:
+        print("❌ Some tests failed!")
+        return False
+
+if __name__ == "__main__":
+    success = main()
+    sys.exit(0 if success else 1)

test_editing_workflow.py

@@ -0,0 +1,235 @@
+#!/usr/bin/env python3
+"""
+Test script for the enhanced editing workflow
+Tests: Edit Task Plan → Update DAG Visualization → Validate & Deploy
+"""
+
+import sys
+import os
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from gradio_llm_interface import GradioLlmInterface
+import json
+
+def test_task_plan_editor():
+    """Test task plan editor functionality"""
+    print("Testing Task Plan Editor...")
+    print("=" * 40)
+    
+    # Create test task data
+    test_task_data = {
+        "tasks": [
+            {
+                "task": "excavate_soil_from_pile",
+                "instruction_function": {
+                    "name": "excavate_soil_from_pile",
+                    "robot_ids": ["robot_excavator_01"],
+                    "dependencies": [],
+                    "object_keywords": ["soil_pile"]
+                }
+            }
+        ]
+    }
+    
+    interface = GradioLlmInterface()
+    
+    # Test with existing task plan
+    state_with_plan = {'pending_task_plan': test_task_data}
+    result = interface.show_task_plan_editor(state_with_plan)
+    
+    if result and len(result) == 4:
+        editor_update, dag_btn_update, validate_btn_update, status_msg = result
+        
+        print("✓ Editor opened with existing task plan")
+        print(f"  Status: {status_msg[:50]}...")
+        print(f"  Editor visible: {'✓' if editor_update.get('visible') else '✗'}")
+        print(f"  JSON populated: {'✓' if editor_update.get('value') else '✗'}")
+        
+        return True
+    else:
+        print("✗ Failed to open task plan editor")
+        return False
+
+def test_dag_update_from_editor():
+    """Test DAG update from edited JSON"""
+    print("\nTesting DAG Update from Editor...")
+    print("=" * 40)
+    
+    # Sample edited JSON
+    edited_json = """{
+  "tasks": [
+    {
+      "task": "move_to_position_1",
+      "instruction_function": {
+        "name": "move_to_position",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": [],
+        "object_keywords": ["loading_zone"]
+      }
+    },
+    {
+      "task": "load_material_1",
+      "instruction_function": {
+        "name": "load_material",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": ["move_to_position_1"],
+        "object_keywords": ["soil", "material"]
+      }
+    }
+  ]
+}"""
+    
+    interface = GradioLlmInterface()
+    state = {}
+    
+    try:
+        result = interface.update_dag_from_editor(edited_json, state)
+        
+        if result and len(result) == 6:
+            dag_image, validate_btn, editor_vis, dag_btn_vis, status_msg, updated_state = result
+            
+            print("✓ DAG updated from edited JSON")
+            print(f"  Image generated: {'✓' if dag_image else '✗'}")
+            print(f"  Validate button shown: {'✓' if validate_btn.get('visible') else '✗'}")
+            print(f"  Task plan stored: {'✓' if updated_state.get('pending_task_plan') else '✗'}")
+            
+            return True
+        else:
+            print("✗ Failed to update DAG from editor")
+            return False
+            
+    except Exception as e:
+        print(f"✗ Error updating DAG: {e}")
+        return False
+
+def test_invalid_json_handling():
+    """Test handling of invalid JSON"""
+    print("\nTesting Invalid JSON Handling...")
+    print("=" * 40)
+    
+    invalid_json = """{
+  "tasks": [
+    {
+      "task": "invalid_task"
+      "missing_comma": true
+    }
+  ]
+}"""
+    
+    interface = GradioLlmInterface()
+    state = {}
+    
+    try:
+        result = interface.update_dag_from_editor(invalid_json, state)
+        
+        if result and len(result) == 6:
+            dag_image, validate_btn, editor_vis, dag_btn_vis, status_msg, updated_state = result
+            
+            if "JSON Parsing Error" in status_msg:
+                print("✓ Invalid JSON handled correctly")
+                print(f"  Error message displayed: {'✓' if 'JSON Parsing Error' in status_msg else '✗'}")
+                print(f"  Editor kept visible: {'✓' if editor_vis.get('visible') else '✗'}")
+                return True
+            else:
+                print("✗ Invalid JSON not handled correctly")
+                return False
+        else:
+            print("✗ Unexpected result from invalid JSON")
+            return False
+            
+    except Exception as e:
+        print(f"✗ Unexpected exception: {e}")
+        return False
+
+def test_full_workflow():
+    """Test the complete editing workflow"""
+    print("\nTesting Complete Editing Workflow...")
+    print("=" * 40)
+    
+    interface = GradioLlmInterface()
+    
+    # Step 1: Open editor
+    state = {}
+    print("Step 1: Opening task plan editor...")
+    editor_result = interface.show_task_plan_editor(state)
+    
+    if not editor_result:
+        print("✗ Failed at step 1")
+        return False
+    
+    # Step 2: Update DAG with valid JSON
+    valid_json = """{
+  "tasks": [
+    {
+      "task": "complete_workflow_test",
+      "instruction_function": {
+        "name": "test_function",
+        "robot_ids": ["robot_excavator_01"],
+        "dependencies": [],
+        "object_keywords": ["test_object"]
+      }
+    }
+  ]
+}"""
+    
+    print("Step 2: Updating DAG from editor...")
+    update_result = interface.update_dag_from_editor(valid_json, state)
+    
+    if not update_result or len(update_result) != 6:
+        print("✗ Failed at step 2")
+        return False
+    
+    # Step 3: Validate and deploy
+    print("Step 3: Validating and deploying...")
+    deploy_result = interface.validate_and_deploy_task_plan(state)
+    
+    if deploy_result:
+        print("✓ Complete workflow test passed")
+        return True
+    else:
+        print("✗ Failed at step 3")
+        return False
+
+def main():
+    """Run all editing workflow tests"""
+    print("🛠️ Enhanced Editing Workflow Tests")
+    print("=" * 50)
+    
+    tests = [
+        test_task_plan_editor,
+        test_dag_update_from_editor,
+        test_invalid_json_handling,
+        test_full_workflow
+    ]
+    
+    passed = 0
+    total = len(tests)
+    
+    for test in tests:
+        try:
+            if test():
+                passed += 1
+        except Exception as e:
+            print(f"✗ Test failed with exception: {e}")
+    
+    print("\n" + "=" * 50)
+    print(f"Editing Workflow Tests passed: {passed}/{total}")
+    
+    if passed == total:
+        print("🎉 All editing workflow tests passed!")
+        print("\n🔧 Enhanced Workflow Features:")
+        print("  ✓ Manual JSON editing capability")
+        print("  ✓ Real-time DAG visualization updates")
+        print("  ✓ JSON validation and error handling")
+        print("  ✓ Three-step safety workflow:")
+        print("    1. 📝 Edit Task Plan")
+        print("    2. 🔄 Update DAG Visualization") 
+        print("    3. 🔒 Validate & Deploy Task Plan")
+        return True
+    else:
+        print("❌ Some editing workflow tests failed!")
+        return False
+
+if __name__ == "__main__":
+    success = main()
+    sys.exit(0 if success else 1)

test_persistent_editing.py

@@ -0,0 +1,192 @@
+#!/usr/bin/env python3
+"""
+Test script for persistent editing functionality
+Tests multiple edit cycles to ensure task plans persist correctly
+"""
+
+import sys
+import os
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from gradio_llm_interface import GradioLlmInterface
+import json
+
+def test_edit_cycle():
+    """Test complete edit cycle: generate → edit → update → deploy → edit again"""
+    print("Testing Complete Edit Cycle...")
+    print("=" * 50)
+    
+    interface = GradioLlmInterface()
+    
+    # Step 1: Initial task plan (simulating LLM generation)
+    initial_plan = {
+        "tasks": [
+            {
+                "task": "move_soil_1",
+                "instruction_function": {
+                    "name": "move_soil",
+                    "robot_ids": ["robot_excavator_01"],
+                    "dependencies": [],
+                    "object_keywords": ["soil_pile"]
+                }
+            }
+        ]
+    }
+    
+    state = {'pending_task_plan': initial_plan}
+    print("✓ Step 1: Initial task plan created")
+    
+    # Step 2: Open editor with initial plan
+    editor_result = interface.show_task_plan_editor(state)
+    if not editor_result or len(editor_result) != 4:
+        print("✗ Step 2: Failed to open editor")
+        return False
+    
+    editor_update, dag_btn, validate_btn, status = editor_result
+    if "move_soil" in editor_update.get('value', ''):
+        print("✓ Step 2: Editor opened with correct initial plan")
+    else:
+        print("✗ Step 2: Editor does not contain initial plan")
+        return False
+    
+    # Step 3: Edit the plan
+    edited_json = """{
+  "tasks": [
+    {
+      "task": "move_soil_1_edited",
+      "instruction_function": {
+        "name": "move_soil_edited",
+        "robot_ids": ["robot_excavator_01", "robot_dump_truck_01"],
+        "dependencies": [],
+        "object_keywords": ["soil_pile", "edited_keyword"]
+      }
+    },
+    {
+      "task": "transport_soil_1",
+      "instruction_function": {
+        "name": "transport_soil",
+        "robot_ids": ["robot_dump_truck_01"],
+        "dependencies": ["move_soil_1_edited"],
+        "object_keywords": ["destination"]
+      }
+    }
+  ]
+}"""
+    
+    update_result = interface.update_dag_from_editor(edited_json, state)
+    if not update_result or len(update_result) != 6:
+        print("✗ Step 3: Failed to update DAG from editor")
+        return False
+    
+    print("✓ Step 3: DAG updated with edited plan")
+    
+    # Step 4: Deploy the plan
+    deploy_result = interface.validate_and_deploy_task_plan(state)
+    if not deploy_result:
+        print("✗ Step 4: Failed to deploy plan")
+        return False
+    
+    print("✓ Step 4: Plan deployed successfully")
+    
+    # Step 5: Try to edit again (this should show the deployed plan)
+    second_editor_result = interface.show_task_plan_editor(state)
+    if not second_editor_result or len(second_editor_result) != 4:
+        print("✗ Step 5: Failed to open editor second time")
+        return False
+    
+    second_editor_update, _, _, second_status = second_editor_result
+    if "move_soil_1_edited" in second_editor_update.get('value', ''):
+        print("✓ Step 5: Editor opened with deployed plan (persistent editing working)")
+        return True
+    else:
+        print("✗ Step 5: Editor lost the deployed plan content")
+        print(f"  Editor content: {second_editor_update.get('value', 'No content')[:100]}...")
+        return False
+
+def test_empty_state_handling():
+    """Test editor behavior with completely empty state"""
+    print("\nTesting Empty State Handling...")
+    print("=" * 40)
+    
+    interface = GradioLlmInterface()
+    empty_state = {}
+    
+    result = interface.show_task_plan_editor(empty_state)
+    if result and len(result) == 4:
+        editor_update, _, _, status = result
+        if "example_task_1" in editor_update.get('value', ''):
+            print("✓ Empty state shows example template")
+            return True
+        else:
+            print("✗ Empty state does not show proper template")
+            return False
+    else:
+        print("✗ Failed to handle empty state")
+        return False
+
+def test_malformed_state_handling():
+    """Test editor behavior with malformed state data"""
+    print("\nTesting Malformed State Handling...")
+    print("=" * 40)
+    
+    interface = GradioLlmInterface()
+    
+    # Test with empty tasks array
+    malformed_state = {
+        'pending_task_plan': {
+            'tasks': []
+        }
+    }
+    
+    result = interface.show_task_plan_editor(malformed_state)
+    if result and len(result) == 4:
+        editor_update, _, _, status = result
+        if "example_task_1" in editor_update.get('value', ''):
+            print("✓ Malformed state (empty tasks) handled correctly")
+            return True
+        else:
+            print("✗ Malformed state not handled properly")
+            return False
+    else:
+        print("✗ Failed to handle malformed state")
+        return False
+
+def main():
+    """Run all persistent editing tests"""
+    print("🔄 Persistent Editing Tests")
+    print("=" * 50)
+    
+    tests = [
+        test_edit_cycle,
+        test_empty_state_handling,
+        test_malformed_state_handling
+    ]
+    
+    passed = 0
+    total = len(tests)
+    
+    for test in tests:
+        try:
+            if test():
+                passed += 1
+        except Exception as e:
+            print(f"✗ Test failed with exception: {e}")
+    
+    print("\n" + "=" * 50)
+    print(f"Persistent Editing Tests passed: {passed}/{total}")
+    
+    if passed == total:
+        print("🎉 All persistent editing tests passed!")
+        print("\n🔄 Persistent Editing Features:")
+        print("  ✓ Task plans persist through edit cycles")
+        print("  ✓ Deployed plans can be re-edited")
+        print("  ✓ State management handles edge cases")
+        print("  ✓ Proper fallback to templates when needed")
+        return True
+    else:
+        print("❌ Some persistent editing tests failed!")
+        return False
+
+if __name__ == "__main__":
+    success = main()
+    sys.exit(0 if success else 1)

test_safety_workflow.py

@@ -0,0 +1,141 @@
+#!/usr/bin/env python3
+"""
+Test script for safety confirmation workflow
+"""
+
+import sys
+import os
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
+
+from gradio_llm_interface import GradioLlmInterface
+import json
+
+def test_safety_workflow():
+    """Test the safety confirmation workflow"""
+    print("Testing Safety Confirmation Workflow...")
+    print("=" * 50)
+    
+    # Create mock task data
+    mock_task_data = {
+        "tasks": [
+            {
+                "task": "excavate_soil_from_pile",
+                "instruction_function": {
+                    "name": "excavate_soil_from_pile",
+                    "robot_ids": ["robot_excavator_01"],
+                    "dependencies": [],
+                    "object_keywords": ["soil_pile"]
+                }
+            },
+            {
+                "task": "transport_soil_to_pit",
+                "instruction_function": {
+                    "name": "transport_soil_to_pit", 
+                    "robot_ids": ["robot_dump_truck_01"],
+                    "dependencies": ["excavate_soil_from_pile"],
+                    "object_keywords": ["pit"]
+                }
+            }
+        ]
+    }
+    
+    # Initialize interface
+    interface = GradioLlmInterface()
+    
+    # Create mock state with pending task plan
+    mock_state = {
+        'pending_task_plan': mock_task_data,
+        'history': []
+    }
+    
+    print("✓ Mock state created with pending task plan")
+    print(f"  Tasks: {len(mock_task_data['tasks'])}")
+    
+    # Test validation and deployment
+    try:
+        result = interface.validate_and_deploy_task_plan(mock_state)
+        
+        if result:
+            confirmation_msg, approved_image_path, button_update, updated_state = result
+            
+            print("\n📋 Validation Results:")
+            print(f"  Confirmation Message: {confirmation_msg[:100]}...")
+            print(f"  Image Generated: {'✓' if approved_image_path else '✗'}")
+            print(f"  Button Hidden: {'✓' if not button_update.get('visible', True) else '✗'}")
+            print(f"  State Updated: {'✓' if updated_state.get('pending_task_plan') is None else '✗'}")
+            
+            return True
+        else:
+            print("✗ No result returned from validation function")
+            return False
+            
+    except Exception as e:
+        print(f"✗ Error during validation: {e}")
+        return False
+
+def test_empty_state():
+    """Test with empty state (no pending task plan)"""
+    print("\nTesting Empty State Handling...")
+    print("=" * 30)
+    
+    interface = GradioLlmInterface()
+    empty_state = {'history': []}
+    
+    try:
+        result = interface.validate_and_deploy_task_plan(empty_state)
+        
+        if result:
+            warning_msg, image_path, button_update, state = result
+            
+            if "No Task Plan to Deploy" in warning_msg:
+                print("✓ Empty state handled correctly")
+                return True
+            else:
+                print("✗ Unexpected warning message")
+                return False
+        else:
+            print("✗ No result returned for empty state")
+            return False
+            
+    except Exception as e:
+        print(f"✗ Error handling empty state: {e}")
+        return False
+
+def main():
+    """Run safety workflow tests"""
+    print("🔒 Safety Confirmation Workflow Tests")
+    print("=" * 50)
+    
+    tests = [
+        test_safety_workflow,
+        test_empty_state
+    ]
+    
+    passed = 0
+    total = len(tests)
+    
+    for test in tests:
+        try:
+            if test():
+                passed += 1
+        except Exception as e:
+            print(f"✗ Test failed with exception: {e}")
+    
+    print("\n" + "=" * 50)
+    print(f"Safety Tests passed: {passed}/{total}")
+    
+    if passed == total:
+        print("🎉 All safety workflow tests passed!")
+        print("\n🔐 Safety Features:")
+        print("  ✓ Task plan approval workflow")
+        print("  ✓ Visual confirmation before deployment") 
+        print("  ✓ Error handling and validation")
+        print("  ✓ State management for pending plans")
+        return True
+    else:
+        print("❌ Some safety tests failed!")
+        return False
+
+if __name__ == "__main__":
+    success = main()
+    sys.exit(0 if success else 1)