1️⃣ Goals

Consolidate all migration work (PHP8 upgrade, dead-code audit, dependencies) in one system.

Track every PHP file (and other assets) in a single database with clear status fields.

Provide tools for:

Inventory & classification (keep/archive/remove)

Migration steps (rewrite, test, scan)

Backup of all source + DB

Reporting (markdown / CSV)

2️⃣ Folder Layout (inside MountPoints/Intrastats.thevault.me.uk)
vmserver10/
   intranet/        <-- original + migrated code
Progress/           <-- generated reports (optional)
Tools/              <-- all migration utilities
   migration.db
   inventory.py
   scan_links.py
   report.py
   migration_gui.py
   backup_all.py


Everything new lives in Tools.

3️⃣ Database Design (migration.db)

Table: files

column	type	purpose
id	INTEGER PK	internal id
primary_folder	TEXT	first path segment
sub_folder	TEXT	optional second segment
file_name	TEXT	filename
status	TEXT	new | keep | archive | remove
php8_rewritten	INTEGER(0/1)	
tested	INTEGER(0/1)	
links_scanned	INTEGER(0/1)	
notes	TEXT	
last_updated	DATETIME	

Table: links

column	type	purpose
id	INTEGER PK	
source_file_id	INTEGER → files.id	
target_path	TEXT	
created_on	DATETIME	

All statuses are in one place: no progress.md vs TO_DO.md.

4️⃣ Core Scripts (in Tools/)
Script	Purpose
inventory.py	Walk ../vmserver10/intranet, insert/update files table (set status='new' if unseen).
scan_links.py	For a file (or all), parse includes/requires → links table.
report.py	Generate progress reports as markdown / CSV.
backup_all.py	Zip everything (Tools, vmserver10/intranet, DB).
migration_gui.py	Tkinter GUI: filter, mark flags, open file, trigger scans/backups.
5️⃣ Backup Script (backup_all.py)
#!/usr/bin/env python3
import os, zipfile
from datetime import datetime

ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), '..'))
BACKUP_DIR = os.path.join(ROOT, 'Backups')

def create_backup():
    if not os.path.exists(BACKUP_DIR):
        os.makedirs(BACKUP_DIR)

    ts = datetime.now().strftime('%Y%m%d_%H%M%S')
    zip_path = os.path.join(BACKUP_DIR, f'intrastats_migration_{ts}.zip')

    with zipfile.ZipFile(zip_path, 'w', zipfile.ZIP_DEFLATED) as z:
        for base in ['vmserver10', 'Tools']:
            bpath = os.path.join(ROOT, base)
            for root, dirs, files in os.walk(bpath):
                # exclude Backups folder itself
                dirs[:] = [d for d in dirs if d.lower() != 'backups']
                for f in files:
                    fp = os.path.join(root, f)
                    arc = os.path.relpath(fp, ROOT)
                    z.write(fp, arc)

    print(f"✅ Backup created: {zip_path}")

if __name__ == '__main__':
    create_backup()

6️⃣ GUI (high level)

Filter bar (status = keep / new / archive)

Show one file at a time with:

Path

Status selector (keep/archive/remove)

Checkboxes: PHP8 done / Tested / Scanned

Notes box

Buttons:

Save → update DB

Scan Links

Open File (launch editor)

Backup

Next

7️⃣ Migration Workflow

Initial import

cd Tools
python3 inventory.py


→ Seeds migration.db.

Classify

Use GUI to set status for new files.

Upgrade pass

For keep:

Rewrite for PHP8 → tick php8_rewritten.

Test on new server → tick tested.

Run scan_links.py or GUI “Scan Links” → tick links_scanned.

Reports

python3 report.py --csv or --md.

Backups

Run backup_all.py anytime or via GUI.

8️⃣ Optional Extensions

Add history table to record every flag change.

Web dashboard instead of Tkinter if you prefer.

Support other file types (JS, CSS) by adding a file_type column.

9️⃣ Next Steps

Create migration.db and schema.

Write inventory.py (scans PHP tree, fills DB).

Replace old GUI with migration_gui.py (query/update DB instead of parsing markdown).

Port any notes from progress.md/TO_DO.md into DB’s notes column.

Retire text files once DB has all data.
