Automated model update: 2025-07-15 20:00:00

scripts/README.md

@@ -0,0 +1,259 @@
+# UFC Prediction Pipeline Automation Scripts
+
+This directory contains automation scripts that run the UFC prediction pipeline and automatically push logs and model updates to your git repository.
+
+## Scripts Overview
+
+### 1. `update_models.ps1` (Windows PowerShell)
+Production-ready automation script for Windows Task Scheduler with git integration.
+
+**Features:**
+- Runs model updates with resource limits and timeouts
+- Commits and pushes logs to git repository automatically
+- Handles missing models and new data detection
+- Comprehensive error handling and logging
+- Lock file mechanism to prevent concurrent runs
+
+### 2. `update_models.sh` (Linux/Unix Bash)
+Production-ready automation script for cron jobs with git integration.
+
+**Features:**
+- Same functionality as PowerShell version but for Unix systems
+- Memory and CPU resource limits
+- Git integration for automatic log syncing
+- Process locking and error recovery
+
+### 3. `startup_check.ps1` (Windows Startup)
+Lightweight script that runs when you start your laptop.
+
+**Features:**
+- Quick model update check on system startup
+- Desktop notification if models are updated
+- Automatic git sync of startup logs
+- Minimal resource usage
+
+## Setup Instructions
+
+### Prerequisites
+
+1. **Git Repository Setup**
+   ```bash
+   # Make sure your UFC project is a git repository
+   cd /path/to/ufc
+   git init  # if not already a git repo
+   git remote add origin https://github.com/yourusername/ufc.git
+   
+   # Configure git credentials (for automation)
+   git config user.name "Your Name"
+   git config user.email "[email protected]"
+   
+   # Optional: Set up credential caching to avoid password prompts
+   git config credential.helper store
+   ```
+
+2. **Create logs directory**
+   ```bash
+   mkdir -p logs
+   ```
+
+### Windows Setup (PowerShell Scripts)
+
+#### 1. Configure Script Paths
+Edit the script files and update the `PROJECT_DIR` variable:
+```powershell
+$PROJECT_DIR = "C:\Users\Alvaro\Desktop\ufc"  # Update this path
+```
+
+#### 2. Set Execution Policy
+```powershell
+# Run as Administrator
+Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope LocalMachine
+```
+
+#### 3. Schedule with Task Scheduler
+
+**For Regular Updates (`update_models.ps1`):**
+1. Open Task Scheduler (`taskschd.msc`)
+2. Create Basic Task
+   - Name: "UFC Model Update"
+   - Trigger: Daily (or your preferred frequency)
+   - Action: Start a program
+   - Program: `powershell.exe`
+   - Arguments: `-File "C:\Users\Alvaro\Desktop\ufc\scripts\update_models.ps1"`
+
+**For Startup Check (`startup_check.ps1`):**
+1. Create Basic Task
+   - Name: "UFC Startup Check"
+   - Trigger: At startup
+   - Action: Start a program
+   - Program: `powershell.exe`
+   - Arguments: `-File "C:\Users\Alvaro\Desktop\ufc\scripts\startup_check.ps1"`
+
+### Linux/Unix Setup (Bash Scripts)
+
+#### 1. Configure Script Paths
+Edit `update_models.sh` and update the path:
+```bash
+PROJECT_DIR="/path/to/your/ufc"  # Update this path
+```
+
+#### 2. Make Scripts Executable
+```bash
+chmod +x scripts/update_models.sh
+```
+
+#### 3. Setup Cron Job
+```bash
+# Edit crontab
+crontab -e
+
+# Add line for daily updates at 2 AM
+0 2 * * * /path/to/ufc/scripts/update_models.sh
+
+# Or for updates every 6 hours
+0 */6 * * * /path/to/ufc/scripts/update_models.sh
+```
+
+## Git Integration Features
+
+### Automatic Commits
+The scripts automatically commit and push:
+- **Log files** (`logs/model_update.log`, `logs/startup_update.log`)
+- **Model files** (if retrained)
+- **Output files** (predictions, accuracy reports, etc.)
+- **Data files** (if updated with new fights)
+
+### Commit Messages
+Automated commits use descriptive messages:
+- `"Automated model update: 2025-01-15 14:30:22"`
+- `"Startup model check: 2025-01-15 08:15:33"`
+
+### Git Error Handling
+- Scripts verify git repository existence
+- Check for changes before committing
+- Handle network errors gracefully
+- Fall back to local commits if push fails
+- Continue operation even if git fails
+
+## Monitoring and Logs
+
+### Log Files
+- `logs/model_update.log` - Main automation logs
+- `logs/startup_update.log` - Startup check logs
+- `output/model_results.json` - Latest prediction results
+
+### What Gets Logged
+- Pipeline execution start/end times
+- Model retraining decisions
+- Git operations (add, commit, push)
+- Error messages and stack traces
+- Resource usage and timeouts
+
+### Sample Log Output
+```
+[2025-01-15 14:30:15] Starting automated model update and git sync...
+[2025-01-15 14:30:16] Running model update pipeline...
+[2025-01-15 14:32:45] Model update pipeline completed successfully.
+[2025-01-15 14:32:46] Syncing changes to git repository...
+[2025-01-15 14:32:46] Git: Adding all changes
+[2025-01-15 14:32:47] Git: Adding all changes completed successfully
+[2025-01-15 14:32:47] Git: Committing changes
+[2025-01-15 14:32:48] Git: Committing changes completed successfully
+[2025-01-15 14:32:48] Git: Pushing to remote repository
+[2025-01-15 14:32:52] Git: Pushing to remote repository completed successfully
+[2025-01-15 14:32:52] Successfully pushed all changes to git repository.
+[2025-01-15 14:32:52] Automated update completed successfully with git sync.
+```
+
+## Security Considerations
+
+### Git Credentials
+- Use SSH keys instead of passwords for git authentication
+- Consider using git credential helpers for HTTPS
+- Never hardcode credentials in scripts
+
+### File Permissions
+```bash
+# Secure script files
+chmod 750 scripts/*.sh
+chmod 750 scripts/*.ps1
+
+# Secure log directory
+chmod 755 logs/
+```
+
+## Troubleshooting
+
+### Common Issues
+
+1. **Git Authentication Errors**
+   ```bash
+   # Setup SSH key authentication
+   ssh-keygen -t ed25519 -C "[email protected]"
+   # Add public key to GitHub/GitLab
+   ```
+
+2. **PowerShell Execution Policy**
+   ```powershell
+   # Check current policy
+   Get-ExecutionPolicy
+   
+   # Set to allow scripts
+   Set-ExecutionPolicy RemoteSigned -Scope CurrentUser
+   ```
+
+3. **Path Issues**
+   - Verify PROJECT_DIR paths in all scripts
+   - Use absolute paths to avoid confusion
+   - Check that Python is in system PATH
+
+4. **Resource Limits**
+   - Adjust memory limits in scripts if needed
+   - Monitor system resources during execution
+   - Consider running during off-peak hours
+
+### Debug Mode
+Add debug logging by modifying scripts:
+```powershell
+# PowerShell debug
+$DebugPreference = "Continue"
+```
+
+```bash
+# Bash debug
+set -x  # Add to top of script
+```
+
+## Customization
+
+### Frequency
+- Modify cron schedule or Task Scheduler triggers
+- Consider UFC event calendar for optimal timing
+- Balance between freshness and resource usage
+
+### Resource Limits
+```powershell
+# PowerShell - adjust timeout
+$MAX_TIMEOUT_MINUTES = 180  # 3 hours
+```
+
+```bash
+# Bash - adjust memory and timeout
+MAX_MEMORY="8G"  # Increase memory limit
+NICE_LEVEL="5"   # Higher priority
+```
+
+### Notification Settings
+Modify `startup_check.ps1` to customize notifications:
+```powershell
+# Custom notification
+[System.Windows.Forms.MessageBox]::Show("Custom message", "Title", "OK", "Information")
+```
+
+## Support
+
+For issues or questions:
+1. Check log files for error details
+2. Verify git repository status manually
+3. Test scripts manually before scheduling
+4. Review this documentation for configuration details 

scripts/startup_check.ps1

@@ -11,22 +11,77 @@ function Write-Log {
     Write-Host "[$timestamp] $Message"
 }
 
+# Function to safely execute git commands
+function Invoke-GitCommand {
+    param($Command, $Description)
+    try {
+        Write-Log "Git: $Description"
+        $result = Invoke-Expression "git $Command" 2>&1
+        if ($LASTEXITCODE -eq 0) {
+            Write-Log "Git: $Description completed successfully"
+            return $true
+        } else {
+            Write-Log "Git: $Description failed with exit code $LASTEXITCODE"
+            return $false
+        }
+    } catch {
+        Write-Log "Git: Error executing $Description - $($_.Exception.Message)"
+        return $false
+    }
+}
+
+# Function to commit and push changes
+function Push-ChangesToGit {
+    if (!(Test-Path ".git")) {
+        Write-Log "Not in a git repository. Skipping git operations."
+        return
+    }
+    
+    # Check git status
+    $gitStatus = git status --porcelain 2>&1
+    if ($LASTEXITCODE -ne 0 -or [string]::IsNullOrWhiteSpace($gitStatus)) {
+        Write-Log "No changes to commit or git error."
+        return
+    }
+    
+    # Add and commit changes
+    if ((Invoke-GitCommand "add ." "Adding startup changes") -and 
+        (Invoke-GitCommand "commit -m `"Startup model check: $(Get-Date -Format 'yyyy-MM-dd HH:mm:ss')`"" "Committing startup changes")) {
+        Invoke-GitCommand "push" "Pushing startup changes to remote"
+    }
+}
+
 try {
     Write-Log "Startup model check initiated..."
     Set-Location $PROJECT_DIR
     
     # Run a quick check (won't retrain unless needed)
+    Write-Log "Running model update pipeline..."
     $result = python -m src.main --pipeline update 2>&1
-    Write-Log "Update result: $result"
+    Write-Log "Pipeline result: $result"
     
-    if ($result -like "*retraining*") {
+    # Push logs and any changes to git
+    Write-Log "Syncing changes to git repository..."
+    Push-ChangesToGit
+    
+    if ($result -like "*retraining*" -or $result -like "*new data*") {
         # Show notification if models were updated
         Add-Type -AssemblyName System.Windows.Forms
-        [System.Windows.Forms.MessageBox]::Show("UFC models were updated with new data!", "Model Update", "OK", "Information")
+        [System.Windows.Forms.MessageBox]::Show("UFC models were updated with new data and synced to git!", "Model Update", "OK", "Information")
+        Write-Log "Models were updated and synced to git."
+    } else {
+        Write-Log "No model updates needed, logs synced to git."
     }
     
 } catch {
     Write-Log "Error during startup check: $($_.Exception.Message)"
+    
+    # Try to push error logs to git
+    try {
+        Push-ChangesToGit
+    } catch {
+        Write-Log "Failed to push error logs to git: $($_.Exception.Message)"
+    }
 }
 
 Write-Log "Startup check completed." 

scripts/update_models.ps1

@@ -1,5 +1,5 @@
 # UFC Model Update Script for Windows Task Scheduler
-# This script safely updates models with resource limits
+# This script safely updates models with resource limits and pushes logs to git
 
 # Configuration
 $PROJECT_DIR = "C:\Users\Alvaro\Desktop\ufc"  # Change this to your actual path
@@ -21,6 +21,72 @@ function Write-Log {
     Write-Host "[$timestamp] $Message"
 }
 
+# Function to safely execute git commands
+function Invoke-GitCommand {
+    param($Command, $Description)
+    try {
+        Write-Log "Git: $Description"
+        $result = Invoke-Expression "git $Command" 2>&1
+        if ($LASTEXITCODE -eq 0) {
+            Write-Log "Git: $Description completed successfully"
+            if ($result) { Write-Log "Git output: $result" }
+            return $true
+        } else {
+            Write-Log "Git: $Description failed with exit code $LASTEXITCODE"
+            Write-Log "Git error: $result"
+            return $false
+        }
+    } catch {
+        Write-Log "Git: Error executing $Description - $($_.Exception.Message)"
+        return $false
+    }
+}
+
+# Function to commit and push changes
+function Push-ChangesToGit {
+    Write-Log "Starting git operations..."
+    
+    # Check if we're in a git repository
+    if (!(Test-Path ".git")) {
+        Write-Log "Not in a git repository. Skipping git operations."
+        return
+    }
+    
+    # Check git status
+    $gitStatus = git status --porcelain 2>&1
+    if ($LASTEXITCODE -ne 0) {
+        Write-Log "Failed to check git status. Skipping git operations."
+        return
+    }
+    
+    if ([string]::IsNullOrWhiteSpace($gitStatus)) {
+        Write-Log "No changes to commit."
+        return
+    }
+    
+    # Add all changes
+    if (!(Invoke-GitCommand "add ." "Adding all changes")) {
+        return
+    }
+    
+    # Create commit message with timestamp
+    $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"
+    $commitMessage = "Automated model update: $timestamp"
+    
+    # Commit changes
+    if (!(Invoke-GitCommand "commit -m `"$commitMessage`"" "Committing changes")) {
+        return
+    }
+    
+    # Push to remote
+    if (!(Invoke-GitCommand "push" "Pushing to remote repository")) {
+        Write-Log "Failed to push changes. Changes are committed locally."
+        return
+    }
+    
+    Write-Log "Successfully pushed all changes to git repository."
+}
+
 # Check if another update is already running
 if (Test-Path $LOCK_FILE) {
     Write-Log "Another update process is already running. Exiting."
@@ -31,28 +97,38 @@ if (Test-Path $LOCK_FILE) {
 $PID | Out-File -FilePath $LOCK_FILE
 
 try {
-    Write-Log "Starting model update check..."
+    Write-Log "Starting automated model update and git sync..."
     
     # Change to project directory
     Set-Location $PROJECT_DIR
     
+    # Verify we're in the right directory
+    if (!(Test-Path "src\main.py")) {
+        Write-Log "Error: Not in UFC project directory. Expected to find src\main.py"
+        exit 1
+    }
+    
     # Create a job to run the update with timeout
+    Write-Log "Running model update pipeline..."
     $job = Start-Job -ScriptBlock {
         param($projectDir)
         Set-Location $projectDir
-        python -m src.main --pipeline update
+        python -m src.main --pipeline update 2>&1
     } -ArgumentList $PROJECT_DIR
     
     # Wait for job completion with timeout
+    $updateSuccess = $false
     if (Wait-Job $job -Timeout ($MAX_TIMEOUT_MINUTES * 60)) {
         $result = Receive-Job $job
         $exitCode = $job.State
         
         if ($exitCode -eq "Completed") {
-            Write-Log "Model update completed successfully."
-            Write-Log "Output: $result"
+            Write-Log "Model update pipeline completed successfully."
+            Write-Log "Pipeline output: $result"
+            $updateSuccess = $true
         } else {
-            Write-Log "Model update failed. State: $exitCode"
+            Write-Log "Model update pipeline failed. State: $exitCode"
+            Write-Log "Pipeline output: $result"
         }
     } else {
         Write-Log "Model update timed out after $MAX_TIMEOUT_MINUTES minutes. Stopping job."
@@ -61,12 +137,30 @@ try {
     
     Remove-Job $job -Force
     
+    # Always attempt to push logs and any changes to git
+    Write-Log "Syncing changes to git repository..."
+    Push-ChangesToGit
+    
+    if ($updateSuccess) {
+        Write-Log "Automated update completed successfully with git sync."
+    } else {
+        Write-Log "Automated update completed with errors, but logs have been synced to git."
+    }
+    
 } catch {
-    Write-Log "Error during model update: $($_.Exception.Message)"
+    Write-Log "Critical error during automated update: $($_.Exception.Message)"
+    
+    # Try to push error logs to git
+    try {
+        Set-Location $PROJECT_DIR
+        Push-ChangesToGit
+    } catch {
+        Write-Log "Failed to push error logs to git: $($_.Exception.Message)"
+    }
 } finally {
     # Cleanup lock file
     if (Test-Path $LOCK_FILE) {
         Remove-Item $LOCK_FILE -Force
     }
-    Write-Log "Update check completed."
+    Write-Log "Automated update process finished."
 } 

scripts/update_models.sh

@@ -1,7 +1,7 @@
 #!/bin/bash
 
 # UFC Model Update Script for Cron
-# This script safely updates models with resource limits
+# This script safely updates models with resource limits and pushes logs to git
 
 # Configuration
 PROJECT_DIR="/path/to/your/ufc"  # Change this to your actual path
@@ -16,6 +16,70 @@ mkdir -p "$PROJECT_DIR/logs"
 # Function to log with timestamp
 log() {
     echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
+    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1"
+}
+
+# Function to safely execute git commands
+git_command() {
+    local command="$1"
+    local description="$2"
+    
+    log "Git: $description"
+    if output=$(git $command 2>&1); then
+        log "Git: $description completed successfully"
+        if [ -n "$output" ]; then
+            log "Git output: $output"
+        fi
+        return 0
+    else
+        log "Git: $description failed with exit code $?"
+        log "Git error: $output"
+        return 1
+    fi
+}
+
+# Function to commit and push changes
+push_changes_to_git() {
+    log "Starting git operations..."
+    
+    # Check if we're in a git repository
+    if [ ! -d ".git" ]; then
+        log "Not in a git repository. Skipping git operations."
+        return
+    fi
+    
+    # Check git status
+    if ! git_status=$(git status --porcelain 2>&1); then
+        log "Failed to check git status. Skipping git operations."
+        return
+    fi
+    
+    if [ -z "$git_status" ]; then
+        log "No changes to commit."
+        return
+    fi
+    
+    # Add all changes
+    if ! git_command "add ." "Adding all changes"; then
+        return
+    fi
+    
+    # Create commit message with timestamp
+    local timestamp=$(date '+%Y-%m-%d %H:%M:%S')
+    local commit_message="Automated model update: $timestamp"
+    
+    # Commit changes
+    if ! git_command "commit -m \"$commit_message\"" "Committing changes"; then
+        return
+    fi
+    
+    # Push to remote
+    if ! git_command "push" "Pushing to remote repository"; then
+        log "Failed to push changes. Changes are committed locally."
+        return
+    fi
+    
+    log "Successfully pushed all changes to git repository."
 }
 
 # Check if another update is already running
@@ -30,21 +94,30 @@ echo $$ > "$LOCK_FILE"
 # Cleanup function
 cleanup() {
     rm -f "$LOCK_FILE"
-    log "Update process finished."
+    log "Automated update process finished."
 }
 
 # Set trap to cleanup on exit
 trap cleanup EXIT
 
-log "Starting model update check..."
+log "Starting automated model update and git sync..."
 
 # Change to project directory
 cd "$PROJECT_DIR"
 
+# Verify we're in the right directory
+if [ ! -f "src/main.py" ]; then
+    log "Error: Not in UFC project directory. Expected to find src/main.py"
+    exit 1
+fi
+
+log "Running model update pipeline..."
+
 # Run the update with resource limits
 # nice: lower CPU priority
 # timeout: kill if it takes longer than 2 hours
 # ulimit: limit memory usage
+UPDATE_SUCCESS=false
 nice -n "$NICE_LEVEL" timeout 7200 bash -c "
     ulimit -v $((4 * 1024 * 1024))  # 4GB virtual memory limit
     python -m src.main --pipeline update 2>&1
@@ -53,13 +126,22 @@ nice -n "$NICE_LEVEL" timeout 7200 bash -c "
 # Check exit code
 EXIT_CODE=$?
 if [ $EXIT_CODE -eq 0 ]; then
-    log "Model update completed successfully."
+    log "Model update pipeline completed successfully."
+    UPDATE_SUCCESS=true
 elif [ $EXIT_CODE -eq 124 ]; then
     log "Model update timed out after 2 hours."
 elif [ $EXIT_CODE -eq 137 ]; then
     log "Model update killed due to memory limit."
 else
-    log "Model update failed with exit code: $EXIT_CODE"
+    log "Model update pipeline failed with exit code: $EXIT_CODE"
 fi
 
-log "Update check completed." 
+# Always attempt to push logs and any changes to git
+log "Syncing changes to git repository..."
+push_changes_to_git
+
+if [ "$UPDATE_SUCCESS" = true ]; then
+    log "Automated update completed successfully with git sync."
+else
+    log "Automated update completed with errors, but logs have been synced to git."
+fi