# Tools Directory Structure Documentation

## Overview
The Tools directory contains a comprehensive PHP Migration Tracking System with database-driven file management, dependency scanning, and GUI interface.

## Core Files

### Main Application
- **`run_me.py`** - Main GUI application (2306 lines)
  - `MigrationGUI` class - Primary interface with tkinter
  - File list management with enhanced highlighting system
  - Dependencies tab with live status display
  - Quick Mode for rapid updates
  - OpenAI integration for compatibility checking
  - Quick search functionality for finding files by content
  - Deep search for references to active file across codebase
  - Bulk operations and advanced filtering

### Startup Scripts
- **`run_with_openai.sh`** - Bash startup script for PHP Migration GUI with OpenAI support
  - **Functionality**:
    - Changes directory to script location for relative path execution
    - Checks for existing Python virtual environment (`venv` directory)
    - If venv doesn't exist: creates it using `python3 -m venv venv`, activates it, installs `openai` package via pip
    - If venv exists: directly activates the existing environment
    - Launches the main GUI application with `python3 run_me.py`
  - **Dependencies**: Requires `python3`, `pip`, and internet access for initial OpenAI package installation
  - **Purpose**: Ensures isolated Python environment with required dependencies before running the application

### Database Layer
- **`db_utils.py`** - Database abstraction layer
  - `MigrationDB` class for SQLite operations
  - File CRUD operations
  - Links/dependencies management
  - Statistics and reporting queries

### Core Functionality Scripts
- **`inventory.py`** - File system scanner (212 lines)
  - `scan_directory()` - Recursively scans vmserver10/intranet
  - `update_database()` - Adds/updates file records
  - `verify_files()` - Checks file existence
  - Preserves existing file status and flags

- **`scan_links.py`** - Dependency scanner (473 lines)
  - `LinkScanner` class for PHP dependency analysis
  - Regex patterns for include/require detection
  - Path normalization for various include formats
  - Auto-discovery of new files from dependencies
  - Batch scanning capabilities

### Data Management
- **`migrate_data.py`** - Legacy data import
  - Imports from existing progress.md and TO_DO.md files
  - Merges with database while preserving existing data

- **`backup.py`** - System backup functionality
  - Creates timestamped backups
  - Includes database and configuration files

### Utility Scripts
- **`verify_files.py`** - File existence verification
  - Generates missing files lists for server collection
  - Updates file_exists flags in database

- **`cleanup_duplicates.py`** - Database maintenance
  - Removes duplicate file entries
  - Handles server path duplicates

- **`report.py`** - Report generation
  - Summary reports in Markdown format
  - CSV exports for spreadsheet analysis
  - Dependency relationship reports

### Link and File Correction Scripts
- **`check_malformed_links.py`** - Database link validation
  - Scans database for malformed link patterns ($baseurl, <?php, etc.)
  - Reports links containing problematic path components
  - Identifies exact malformed paths for correction

- **`fix_links.py`** - PHP file link correction
  - `LinkFixer` class for automated link fixes
  - Replaces malformed include paths in PHP source files
  - Supports dry-run mode for previewing changes
  - Processes all PHP files recursively in target directory

- **`fix_malformed_files.py`** - Database file path correction
  - Corrects malformed file entries with $baseurl or PHP print statements
  - Parses and normalizes corrupted primary_folder/sub_folder paths
  - Removes truncated malformed entries
  - Updates database with corrected file paths

- **`update_malformed_links.py`** - Database link path updates
  - Fixes existing link entries with malformed target paths
  - Applies pattern-based corrections for common issues
  - Updates links containing $baseurl or PHP print statements
  - Maintains referential integrity during corrections

### Progress and Scanning Scripts
- **`generate_progress_file.py`** - Progress tracking file generation
  - Scans filesystem for PHP files and updates progress.md
  - Adds new files to existing progress table
  - Preserves existing status information
  - Sorts entries by primary folder, subfolder, and filename

- **`Scanfile.py`** - Single file dependency scanner
  - Scans individual PHP files for include/require references
  - Converts relative paths to intranet format
  - Adds discovered dependencies to TO_DO.md
  - Prevents duplicate entries in todo lists

- **`scanforlinks.py`** - Progress file link status checker
  - Reads progress.md to identify unscanned files
  - Outputs files with 'No' in links column
  - Used for batch processing workflows

### Configuration
- **`openai_config.json`** - OpenAI API key storage (optional)
- **`migration.db`** - SQLite database file
- **`OPENAI_SETUP.md`** - OpenAI configuration guide (optional feature)
- **`QUICK_START.md`** - User documentation

## Database Schema

### Tables
1. **files** - Main file inventory
   - id, primary_folder, sub_folder, file_name
   - status (new/keep/archive/remove)
   - php8_rewritten, tested, links_scanned flags
   - file_exists, checked flags
   - notes, created_at, updated_at

2. **links** - File dependencies
   - id, source_file_id, target_path
   - created_at

## GUI Structure (run_me.py)

### Main Components
- **File List (Treeview)** - Main file browser with columns:
  - Checked, Path, Status, PHP8, Tested, Links, Exists
  - Enhanced highlighting system (NEW FEATURE)
  - Sortable columns with visual indicators

- **Details Panel (Notebook)**
  - File Info tab - Status, flags, notes editing
  - Dependencies tab - Live dependency status display

- **Filter System** - Multiple filter controls:
  - Status, PHP8, Tested, Links, Exists filters
  - Search functionality with live updates

### Enhanced Highlighting System (Latest Addition)
- **Checked Files**: Pastel yellow background (`#FFFACD`)
- **Selected Unscanned**: Light red background (`#FFB6C1`) for files with unscanned dependencies
- **Complete Dependency Files**: Light green background (`#98FB98`) for dependency files with all flags complete (✅ PHP8, ✅ Tested, ✅ Links)
- **Incomplete Dependency Files**: Light yellow background (`#FFFFE0`) for dependency files with any incomplete flags (❌ in PHP8, Tested, or Links columns)

#### Key Methods:
- `on_file_select()` - Enhanced with conditional highlighting
- `apply_selection_highlighting()` - Applies red/green highlighting based on links status
- `highlight_dependency_files()` - Highlights all dependency files in green
- `clear_dependency_highlighting()` - Removes dependency highlighting
- `refresh_file_list_display()` - Maintains highlighting during refreshes
- `perform_quick_search()` - Searches file contents for specific terms
- `perform_deep_search()` - Finds all references to active file across codebase
- `toggle_checkbox_column()` - Bulk selection/deselection of files

### Menu Structure
- **File Menu**: Open, Check Compatibility, Exit
- **System Menu**: Rescan, Verify, Generate Lists, Scan Links, Import, Backup, Statistics
- **Reports Menu**: Summary, CSV, Dependencies, Custom
- **Tools Menu**: Bulk Update, Cleanup, OpenAI Config
- **Help Menu**: Quick Start, About

### Quick Mode Features
- Double-click status cycling: new → keep → archive → remove
- Double-click flag toggling for PHP8, Tested, Links columns
- Instant updates without save button

## Workflow Integration

### File Discovery Process
1. `inventory.py` scans filesystem
2. `scan_links.py` discovers dependencies
3. Auto-adds missing dependency files to database
4. `verify_files.py` checks file existence

### Migration Tracking
1. Files start with 'new' status
2. Progress tracked via PHP8, Tested, Links flags
3. Dependencies analyzed and displayed
4. Reports generated for progress monitoring

## Recent Enhancements

### Enhanced File Highlighting (2025-01-19)
- **Purpose**: Visual indication of dependency scan status
- **Implementation**:
  - Red highlighting for files with unscanned dependencies (❌ in Links column)
  - Green highlighting for all dependency files when a scanned file is selected (✅ in Links column)
- **Files Modified**: `run_me.py`
- **Methods Added**:
  - `apply_selection_highlighting()`
  - `highlight_dependency_files()`
  - `clear_dependency_highlighting()`
- **Tag Configurations**:
  - `selected_unscanned` - Light red (`#FFB6C1`)
  - `dependency_file` - Light green (`#98FB98`)

### Advanced Search and Correction Tools (2025-10-05)
- **New Scripts Added**:
  - Link validation and correction suite (`check_malformed_links.py`, `fix_links.py`, `update_malformed_links.py`)
  - File path correction utilities (`fix_malformed_files.py`)
  - Progress file management (`generate_progress_file.py`)
  - Individual file scanning (`Scanfile.py`, `scanforlinks.py`)
- **GUI Enhancements**:
  - Quick search functionality for content-based file discovery
  - Deep search for cross-references to active files
  - Bulk checkbox operations for efficient file management
- **Database Maintenance**:
  - Automated correction of malformed paths and links
  - Pattern-based fixes for common PHP migration issues
  - Integrity checking and cleanup utilities

### In-Memory Logging System (2025-11-05)
- **Purpose**: Real-time debugging and activity monitoring during development
- **Implementation**:
  - Added `add_log()` function for in-memory logging (no disk writes)
  - Integrated logging throughout GUI application functions
  - Log window with scrollbar under File Details section
  - Full-width log display with automatic scrolling to latest entries
- **Files Modified**: `migration_tools_gui.py`
- **Features**:
  - Logs all major application events (button clicks, file operations, errors)
  - Enhanced click handlers with proper event methods (`on_dependency_click`, `on_dependency_enter`, `on_dependency_leave`)
  - Real-time activity monitoring for debugging click events and user interactions
  - Memory-based logging - logs persist only during application session
- **UI Components**:
  - Log text widget with vertical scrollbar
  - Courier font for consistent log formatting
  - Auto-scroll to bottom on new log entries
  - 8-line height with word wrapping

## Development Notes

### Code Organization
- Object-oriented design with clear separation of concerns
- Database abstraction layer for easy maintenance
- Comprehensive error handling and user feedback
- Extensive documentation and help system

### Extension Points
- Plugin system for additional scanners
- Custom report generators
- Additional file status types
- Enhanced dependency analysis

### Maintenance Requirements
- Regular database backups
- Periodic file existence verification
- Link scanning for new files
- Progress report generation

## Future Enhancement Areas
- Batch operations on selected files
- Advanced search and filtering
- Export/import functionality
- Integration with version control systems
- Automated testing framework

---

**Note**: This structure document should be updated whenever significant changes are made to the Tools directory structure, database schema, or core functionality.

**Last Updated**: 2025-11-05 - Added documentation for in-memory logging system and enhanced click handlers