Clean up release documentation

Add production UI screenshots
2026-08-19 15:24:11 +05:30 · 2026-08-19 15:18:32 +05:30 · 2026-08-19 15:09:33 +05:30 · 2026-08-19 13:36:35 +05:30
18 changed files with 660 additions and 519 deletions
@@ -1,519 +0,0 @@
 # NX9 WireGuard (`nx9-wg`) — Full Technical Architecture & Stack Report
 ![Rust](https://img.shields.io/badge/Rust-Stable-orange)
 ![SQLite](https://img.shields.io/badge/SQLite-Embedded-blue)
 ![Platform](https://img.shields.io/badge/Platform-Linux-lightgrey)
 ![License](https://img.shields.io/badge/License-MIT%20OR%20Apache--2.0-green)
 ![Version](https://img.shields.io/badge/Version-v1.0.0-purple)
 ![Ecosystem](https://img.shields.io/badge/NX9-Ecosystem-6d5df6)
 > **"Software people can own, understand, and control."**
 > — [NX9 Systems (https://nx9.in)](https://nx9.in)
 ---
 ## 📑 Table of Contents
 1. [Executive Summary](#1-executive-summary)
 2. [The NX9 Philosophy in Implementation](#2-the-nx9-philosophy-in-implementation)
 3. [The Architectural Paradigm Shift](#3-the-architectural-paradigm-shift)
 4. [Six-Crate Workspace Architecture](#4-six-crate-workspace-architecture)
 5. [Complete Capability Inventory](#5-complete-capability-inventory)
 6. [Native Linux Execution Planes](#6-native-linux-execution-planes)
 7. [Closed-Loop Reconciliation & Convergence](#7-closed-loop-reconciliation--convergence)
 8. [Embedded Single Page Application (SPA) & WebSockets](#8-embedded-single-page-application-spa--websockets)
 9. [REST API & WebSocket Protocol Reference](#9-rest-api--websocket-protocol-reference)
 10. [Native CLI Command System](#10-native-cli-command-system)
 11. [Security Model & Capability Isolation](#11-security-model--capability-isolation)
 12. [Disaster Recovery, Backups & Upgrades](#12-disaster-recovery-backups--upgrades)
 13. [Release Engineering & Deployment Lifecycle](#13-release-engineering--deployment-lifecycle)
 14. [Quality Assurance & Verification Evidence](#14-quality-assurance--verification-evidence)
 15. [Ecosystem & License Summary](#15-ecosystem--license-summary)
 ---
 ## 1. Executive Summary
 `nx9-wg` is a sovereign, self-hosted, Linux-native VPN and network control plane built directly around the Linux kernel's in-tree WireGuard implementation (`wireguard.ko`).
 Rather than functioning as a fragile user interface wrapper that shells out to external command-line utilities (`wg`, `ip`, `nft`, `sysctl`), `nx9-wg` establishes an integrated, single-binary architecture. It combines **authoritative SQLite desired-state persistence**, **direct kernel Netlink execution** (RTNETLINK and WireGuard Generic Netlink), **in-process Netfilter firewall/NAT compilation** (`libnftables.so.1`), **continuous closed-loop reconciliation**, a **multi-format native CLI**, and an **embedded zero-dependency Single Page Application (SPA) Web UI**.
 ---
 ## 2. The NX9 Philosophy in Implementation
 Every design and architectural choice in `nx9-wg` directly reflects the core philosophy of the **NX9 Ecosystem** ([https://nx9.in](https://nx9.in)):
 | NX9 Principle | Core Intent | `nx9-wg` Implementation |
 | :--- | :--- | :--- |
 | 🔑 **Operator Ownership** | Full control over binaries, configurations, databases, keys, and backups with zero dependency on cloud-hosted control planes. | 100% local operation. Private keys, configuration data, and encryption parameters never leave the operator's host. |
 | 🖥️ **Self-Hosting First** | Built to be deployed and operated effortlessly by a single administrator without requiring Kubernetes or external infrastructure. | Self-contained single executable. Bootstrapped with a single command (`nx9-wg init`), running seamlessly under standard systemd. |
 | ⚡ **Simplicity Over Complexity** | One binary, one configuration file, one SQLite database, one administrator. | No multi-daemon orchestration, no external message queues, no Node.js runtime, no Python scripts, and zero shell wrappers. |
 | 🛡️ **Privacy by Default** | Zero analytics, zero telemetry collection, zero third-party tracking, and zero assumptions of external cloud connectivity. | No phone-home telemetry. Completely air-gapped capable with all static assets, fonts, and scripts embedded in the binary. |
 | 🔒 **Security by Design** | Modern cryptography, strict invariants, and append-only audit logging embedded from day one. | Argon2id password hashing, SHA-256 token digests, X25519 key generation, single-admin database constraint (`CHECK (id=1)`), and strict secret redaction. |
 | 🔧 **Operational Excellence** | Diagnostics, backup/restore, migration tools, CLI management, and comprehensive documentation built-in. | Multi-subsystem automated health inspections, atomic online SQLite `VACUUM INTO` snapshots with SHA-256 manifests, and 100% CLI parity. |
 | ⏳ **Long-Term Stability** | Avoid dependency churn. Build software that remains understandable and maintainable years into the future. | Built in stable native Rust (Edition 2024), standard SQLite 3 storage, standard TOML configuration, and POSIX-compliant systemd integration. |
 | 🌐 **Open Source First** | 100% free and open-source software under permissive licensing. | Dual-licensed under `MIT OR Apache-2.0`. Complete freedom to inspect, audit, build, and extend. |
 | ♾️ **Zero Vendor Lock-In** | Standard data formats, open protocols, and zero proprietary cloud lock-in. | Standard SQLite database, standard WireGuard `.conf` files, standard RFC-compliant JSON/YAML/CSV output formats, and open Netlink sockets. |
 ---
 ## 3. The Architectural Paradigm Shift
 ### Typical WireGuard Management Wrappers
 ```
 ┌──────────┐     ┌──────────────────────┐     ┌────────────────────────────┐     ┌──────────────┐
 │  Web UI  │ ──► │  Text Config Files   │ ──► │  wg / ip / nft / sysctl    │ ──► │ Linux Kernel │
 │ (Node/Py)│     │(/etc/wireguard/*.conf│     │ (Subprocess Spawning)      │     │ (wireguard.ko│
 └──────────┘     └──────────────────────┘     └────────────────────────────┘     └──────────────┘
 ```
 *Disadvantages: Fragile process spawning, race conditions, lack of atomic state, broken host routing, vulnerability to configuration drift, and heavy runtime dependency footprints.*
 ### Whereas `nx9-wg` is a Native Control & Execution Plane:
 ```
                                ┌───────────────────────────────────┐
                                │             nx9-wg                │
                                └─────────────────┬─────────────────┘
                                                  │
                 ┌────────────────────────────────┴────────────────────────────────┐
                 │                                                                 │
       ┌─────────▼─────────┐                                             ┌─────────▼─────────┐
       │   Desired State   │                                             │    Live State     │
       │  (Authoritative)  │                                             │  (Kernel Cache)   │
       └─────────┬─────────┘                                             └─────────▲─────────┘
                 │                                                                 │
       ┌─────────▼─────────┐                                             ┌─────────┴─────────┐
       │  SQLite 3 (WAL)   │                                             │   Linux Kernel    │
       └─────────┬─────────┘                                             └─────────▲─────────┘
                 │                                                                 │
                 │                                                       Live Netlink Telemetry
                 ▼                                                                 │
       ┌───────────────────┐                                                       │
       │   Reconciliation  │ ◄─────────────────────────────────────────────────────┘
       │      Engine       │
       └─────────┬─────────┘
                 │
                 ├── 🔐 WireGuard Generic Netlink (family "wireguard")
                 ├── 🌐 RTNETLINK (Links, IPv4/IPv6 Addresses, Routes)
                 ├── 🔥 Netfilter / libnftables FFI (table inet nx9_wg)
                 └── ↔️ Direct Procfs IP Forwarding (/proc/sys/net)
                 │
                 ▼
       ┌───────────────────┐
       │  Linux Networking │
       └───────────────────┘
 ```
 ---
 ## 4. Six-Crate Workspace Architecture
 `nx9-wg` is architected as a modular six-crate Cargo workspace ensuring clean separation of concerns, zero circular dependencies, and isolated testability:
 ```
 nx9-wg (Root Executable & Unified Entrypoint)
 ├── 📦 crates/nx9-wg-core     — Domain models, RFC validation, cryptography, configuration
 ├── 📦 crates/nx9-wg-db       — SQLite storage engine, migration framework, repositories
 ├── 📦 crates/nx9-wireguard   — WireGuard Generic Netlink, RTNETLINK link engine, config & QR
 ├── 📦 crates/nx9-wg-network  — RTNETLINK routes/addresses, libnftables Netfilter, procfs
 ├── 📦 crates/nx9-wg-api      — Axum REST router, WebSockets, auth, reconciliation, IP allocator
 └── 📦 crates/nx9-wg-ui       — Design tokens, CSS compiler, view models, SPA integration
 ```
 ---
 ## 5. Complete Capability Inventory
 `nx9-wg` delivers a comprehensive suite of 20 core infrastructure capabilities:
 | Icon | Capability | Architectural Description |
 | :---: | :--- | :--- |
 | 🔐 | **WireGuard Interface Lifecycle** | In-process RTNETLINK link creation (`RTM_NEWLINK`), state toggling (`IFF_UP`/`IFF_DOWN`), and WireGuard Generic Netlink cryptokey configuration (`WG_CMD_SET_DEVICE`). |
 | 👥 | **Cryptographic Peer Enrollment** | Dynamic Curve25519 public key management, preshared keys, CIDR allowed IPs, persistent keepalive intervals, and atomic `ReplacePeers` synchronization. |
 | 🌐 | **IPv4 & IPv6 Address Management** | Native Netlink address assignment (`RTM_NEWADDR` / `RTM_DELADDR`) across WireGuard interfaces without invoking `ip addr`. |
 | 🛣️ | **Kernel Route Management** | Routing table synchronization (`RTM_NEWROUTE` / `RTM_DELROUTE`) with strict default gateway protection and multi-tenant non-interference invariants. |
 | 🔥 | **nftables Netfilter Firewall** | In-process rule compilation and transactional application via `libnftables.so.1` confined strictly to `table inet nx9_wg`. |
 | 🛡️ | **Scoped NAT & Masquerading** | Outbound NAT masquerade dynamically calculated and applied strictly to managed WireGuard client subnets, preventing host network disruption. |
 | ↔️ | **Direct Procfs IP Forwarding** | Direct atomic mutation of `/proc/sys/net/ipv4/ip_forward` and `/proc/sys/net/ipv6/conf/all/forwarding` without invoking `sysctl`. |
 | 📡 | **Live Kernel Telemetry** | Real-time extraction of handshake timestamps, rx/tx byte counters, and roaming remote socket endpoints directly from kernel sockets. |
 | 🔄 | **Desired-State Reconciliation** | Continuous closed-loop control cycle bringing the Linux kernel execution plane into alignment with authoritative SQLite storage. |
 | 🧭 | **5-Subsystem Drift Detection** | Deterministic, read-only calculation of state divergence across interfaces, peers, routes, firewall rules, and IP forwarding. |
 | ♻️ | **Cold-Boot Restart Recovery** | Autonomous reconstruction of kernel network topology, routes, and firewall rules upon daemon startup or host reboot. |
 | 🧪 | **Cross-Platform Simulation Engine** | In-memory simulated execution planes enabling full UI and CLI development on macOS and Windows without requiring Linux Netlink. |
 | 🖥️ | **Multi-Format Native CLI** | 100% native CLI coverage across all 17 subcommands supporting `table`, `json`, `yaml`, and `csv` output with script-friendly exit codes. |
 | 🌐 | **Axum REST API & WebSockets** | High-performance asynchronous HTTP server with session/bearer authentication and a real-time WebSocket event broadcaster (`/api/v1/ws`). |
 | 💻 | **Embedded Zero-Dependency SPA** | Modern HTML5/CSS3/Vanilla ES6+ Single Page Application compiled into the binary with Light/Dark theme support and 15 interactive views. |
 | 📊 | **Automated Health Diagnostics** | Built-in inspection across 9 subsystems with structured status reporting, root-cause diagnostics, and actionable remediation hints. |
 | 💾 | **Atomic Disaster Recovery Backups** | Online non-blocking SQLite `VACUUM INTO` snapshots with SHA-256 manifest verification and automatic pre-restore safety checkpoints. |
 | 🔑 | **Single-Administrator Security** | Database-enforced single identity (`CHECK (id=1)`), Argon2id password hashing, SHA-256 API token storage, and brute-force login rate limiting. |
 | 📱 | **Client Profiles & QR Engine** | Provider/device MTU profiles (1280 vs 1360 vs 1420), collision-resistant IP allocation, and pure Rust vector SVG, PNG, and ASCII QR generation. |
 | 📦 | **Production Release Packaging** | Standalone distribution archive generator (`scripts/package-release.sh`), automated installer/uninstaller, and hardened systemd service unit. |
 ---
 ## 6. Native Linux Execution Planes
 `nx9-wg` enforces a **Zero Subprocess Guarantee** across the entire production codebase:
 ```
 ┌────────────────────────────────────────────────────────────────────────────────────────┐
 │                                   Rust Production Binary                               │
 ├────────────────────────────┬────────────────────────────┬──────────────────────────────┤
 │ NativeLinuxWireGuardEngine │  NativeLinuxNetworkEngine  │   NativeLinuxNftablesEngine  │
 ├────────────────────────────┼────────────────────────────┼──────────────────────────────┤
 │ • AF_NETLINK               │ • NETLINK_ROUTE            │ • In-process libnftables FFI │
 │ • NETLINK_GENERIC (wg)     │ • RTM_NEWLINK / DELLINK    │ • nft_ctx_new()              │
 │ • WG_CMD_SET_DEVICE        │ • RTM_NEWADDR / DELADDR    │ • Atomic Netfilter batch     │
 │ • WG_CMD_GET_DEVICE        │ • RTM_NEWROUTE / DELROUTE  │ • Scoped: table inet nx9_wg  │
 │ • WGDEVICE_F_REPLACE_PEERS │ • Direct /proc/sys writes  │ • Zero host table flushes    │
 └─────────────┬──────────────┴─────────────┬──────────────┴──────────────┬───────────────┘
              ▼                            ▼                             ▼
 ┌────────────────────────────────────────────────────────────────────────────────────────┐
 │                                     Linux Kernel                                       │
 │                (wireguard.ko • RTNETLINK • Netfilter • /proc/sys/net)                  │
 └────────────────────────────────────────────────────────────────────────────────────────┘
 ```
 ---
 ## 7. Closed-Loop Reconciliation & Convergence
 Reconciliation is the foundational control loop that bridges authoritative SQLite state with the Linux kernel:
 ```
   ┌──────────────────────────────────────────────────────────────────┐
   │ 1. SQLite Desired State (Authoritative Persistent Source of Truth│
   └────────────────────────────────┬─────────────────────────────────┘
                                    │
                                    ▼
   ┌──────────────────────────────────────────────────────────────────┐
   │ 2. Live Kernel Query (WireGuard Genl, RTNL routes, table nx9_wg) │
   └────────────────────────────────┬─────────────────────────────────┘
                                    │
                                    ▼
   ┌──────────────────────────────────────────────────────────────────┐
   │ 3. Drift Detection (Read-Only Deterministic Multi-Subsystem Diff)│
   └────────────────────────────────┬─────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    │ has_drift == false?           │
                    ├───────────────────────┬───────┤
                    │ YES                   │ NO    │
                    ▼                       ▼       │
             ┌─────────────┐         ┌──────────────▼────────────────┐
             │  Converged  │         │ 4. Acquire Async Mutex Lock   │
             │  (In Sync)  │         └──────────────┬────────────────┘
             └─────────────┘                        │
                                                    ▼
                                     ┌───────────────────────────────┐
                                     │ 5. Execute Native Mutations   │
                                     │    (Genl SET_DEVICE, RTNL)    │
                                     └──────────────┬────────────────┘
                                                    │
                                                    ▼
                                     ┌───────────────────────────────┐
                                     │ 6. Post-Apply Verification    │
                                     └──────────────┬────────────────┘
                                                    │
                                    ┌───────────────┴───────────────┐
                                    ▼                               ▼
                             ┌─────────────┐                 ┌─────────────┐
                             │  Converged  │                 │   Partial   │
                             │  (100% Sync)│                 │   Failure   │
                             └─────────────┘                 └─────────────┘
 ```
 ### The Six Reconciliation Lifecycle States
 1. **`Plan`**: Read-only calculation of drift between SQLite and kernel.
 2. **`Applying`**: In-progress dispatch of native mutations across execution planes.
 3. **`Verifying`**: Querying live kernel state to confirm applied changes took effect.
 4. **`Converged`**: 100% synchronization achieved with zero remaining drift.
 5. **`PartialFailure`**: One or more execution planes failed during apply (e.g. permission error).
 6. **`DriftRemains`**: Apply completed without fatal error, but post-verification detected unapplied state.
 ---
 ## 8. Embedded Single Page Application (SPA) & WebSockets
 The `nx9-wg` frontend is a zero-dependency HTML5/CSS/JavaScript SPA embedded directly into the Rust binary:
 - **Zero External Toolchains**: No Node.js, npm, Webpack, Vite, React, or external CDN dependencies.
 - **Embedded In-Memory Delivery**: Bundled at compile-time via `include_str!()` and served from memory.
 - **Design Tokens**: Custom CSS token system ([`crates/nx9-wg-ui/src/css.rs`](file:///home/sunil/Programs/nx9-wg/crates/nx9-wg-ui/src/css.rs)) supporting Light and Dark modes.
 - **Real-Time WebSocket Stream**: Subscribes to `ws://<host>/api/v1/ws` for live handshakes and drift alerts without polling.
 - **15 Interactive Views**:
  1. `#dashboard` — System overview, uptime, interface/peer counts, health cards.
  2. `#interfaces` — WireGuard interface CRUD, listen port, MTU, state toggles.
  3. `#peers` — Enrolled peer table, real-time handshakes, profile resolution, `.conf` export, SVG QR modal.
  4. `#networks` — Subnet network ranges, CIDR masks, available IP inspector.
  5. `#routes` — Kernel route definitions, gateway assignments, interface scoping.
  6. `#firewall` — nftables packet filtering rules in `table inet nx9_wg`, priority sorting.
  7. `#nat` — Managed subnet NAT masquerade status and instant toggle.
  8. `#forwarding` — Kernel IPv4/IPv6 packet forwarding status and toggle.
  9. `#reconciliation` — Real-time kernel drift overview, action plan table, interactive Apply button.
  10. `#diagnostics` — Automated multi-subsystem health checks with remediation hints.
  11. `#live-state` — Raw Linux Netlink telemetry, active kernel interfaces, live routing table.
  12. `#settings` — Key-value appliance parameters and danger zone reset controls.
  13. `#backups` — Online SQLite backup snapshots list, instant backup creation, `.db` download.
  14. `#audit` — Append-only security and administrative audit trail.
  15. `#administrator` — Admin account verification, password rotation, and one-time API token generation.
 ---
 ## 9. REST API & WebSocket Protocol Reference
 ### Authentication Mechanisms
 - **Session Cookie**: `nx9_session=<UUID>` returned via `POST /api/v1/auth/login`.
 - **Bearer Token**: `Authorization: Bearer nx9_<UUID>_<SECRET>` passed in HTTP headers.
 ### Complete REST Route Inventory
 ```http
 POST   /api/v1/auth/login                    - Authenticate administrator & create session
 POST   /api/v1/auth/logout                   - Invalidate active session
 GET    /api/v1/auth/session                  - Query authenticated session info
 POST   /api/v1/auth/password                 - Rotate admin password (invalidates all sessions)
 GET    /api/v1/auth/tokens                   - List active API token metadata
 POST   /api/v1/auth/tokens                   - Generate new API token (one-time raw secret return)
 DELETE /api/v1/auth/tokens/{id}              - Revoke an API token
 GET    /api/v1/system                        - System operational overview and object counts
 GET    /api/v1/system/health                 - Public health check endpoint
 GET    /api/v1/system/version                - Version, build edition, and architecture
 GET    /api/v1/system/settings               - List all appliance key-value settings
 PUT    /api/v1/system/settings               - Upsert appliance setting
 GET    /api/v1/interfaces                    - List all WireGuard interfaces
 POST   /api/v1/interfaces                    - Create WireGuard interface
 GET    /api/v1/interfaces/{id}               - Get interface details
 PUT    /api/v1/interfaces/{id}               - Update interface configuration
 DELETE /api/v1/interfaces/{id}               - Delete interface (cascades to peers)
 POST   /api/v1/interfaces/{id}/enable        - Set interface IFF_UP
 POST   /api/v1/interfaces/{id}/disable       - Set interface IFF_DOWN
 GET    /api/v1/interfaces/{id}/status        - Query live kernel netlink telemetry
 GET    /api/v1/interfaces/{id}/peers         - List peers attached to interface
 POST   /api/v1/interfaces/{id}/peers         - Enroll new peer on interface
 GET    /api/v1/peers/{id}                    - Get peer details
 PUT    /api/v1/peers/{id}                    - Update peer parameters
 DELETE /api/v1/peers/{id}                    - Delete peer
 POST   /api/v1/peers/{id}/enable             - Enable peer
 POST   /api/v1/peers/{id}/disable            - Disable peer
 GET    /api/v1/peers/{id}/config             - Download client .conf file
 GET    /api/v1/peers/{id}/qr                 - Render QR code (SVG / PNG / ASCII)
 GET    /api/v1/networks                      - List subnet networks
 POST   /api/v1/networks                      - Create subnet network
 GET    /api/v1/networks/{id}                 - Get network details
 DELETE /api/v1/networks/{id}                 - Delete network
 GET    /api/v1/networks/{id}/available       - List available unallocated IP addresses
 GET    /api/v1/routes                        - List kernel routing entries
 POST   /api/v1/routes                        - Create routing entry
 DELETE /api/v1/routes/{id}                   - Delete routing entry
 GET    /api/v1/firewall/rules                - List nftables firewall rules
 POST   /api/v1/firewall/rules                - Create firewall rule
 DELETE /api/v1/firewall/rules/{id}           - Delete firewall rule
 POST   /api/v1/firewall/rules/{id}/enable    - Enable firewall rule
 POST   /api/v1/firewall/rules/{id}/disable   - Disable firewall rule
 GET    /api/v1/reconcile/plan                - Read-only drift calculation plan
 POST   /api/v1/reconcile/apply               - Serialized kernel apply and convergence check
 GET    /api/v1/diagnostics/all               - Run full diagnostics across all 9 subsystems
 GET    /api/v1/diagnostics/{subsystem}       - Run diagnostics for single subsystem
 GET    /api/v1/backups                       - List backup records
 POST   /api/v1/backups/create                - Trigger atomic online VACUUM INTO snapshot
 GET    /api/v1/backups/{id}/download         - Download raw SQLite database snapshot
 POST   /api/v1/backups/{id}/restore          - Restore database with automatic safety backup
 DELETE /api/v1/backups/{id}                  - Delete backup snapshot file and metadata
 GET    /api/v1/audit                         - Query append-only audit trail
 ```
 ---
 ## 10. Native CLI Command System
 `nx9-wg` provides 100% native CLI coverage across all 17 subcommands:
 ```bash
 # Global output formats: --format table | json | yaml | csv
 nx9-wg version
 nx9-wg serve --bind 127.0.0.1:8080
 nx9-wg init --generate-password --write-password-file /var/lib/nx9-wg/admin-password
 # Administration & Authentication
 nx9-wg admin info
 nx9-wg admin password
 nx9-wg admin token create "ci-pipeline" --expires-in-days 90 --write-token-file /tmp/token
 nx9-wg admin token list
 nx9-wg admin token revoke <TOKEN_UUID>
 # Interface & Peer Management
 nx9-wg interface list
 nx9-wg interface create wg0 --address-v4 10.100.0.1/24 --port 51820 --mtu 1420
 nx9-wg peer create --interface wg0 --name alice --profile full_tunnel --mtu 1280
 nx9-wg peer qr <PEER_UUID>
 nx9-wg peer config <PEER_UUID>
 # Networking, Firewall & NAT
 nx9-wg route add --destination 192.168.50.0/24 --gateway 10.100.0.2 --interface-name wg0
 nx9-wg firewall add --name "allow-dns" --protocol udp --port 53 --action accept --priority 10
 nx9-wg nat enable
 nx9-wg forwarding enable
 # State Reconciliation & Diagnostics
 nx9-wg reconcile plan
 nx9-wg reconcile apply
 nx9-wg diagnostics all
 # Disaster Recovery
 nx9-wg backup create --description "Pre-upgrade snapshot"
 nx9-wg backup verify /var/lib/nx9-wg/backups/nx9-backup-...db
 nx9-wg backup restore <BACKUP_UUID>
 ```
 ---
 ## 11. Security Model & Capability Isolation
 ### A. Single Administrator Identity
 - Database-level integrity constraint: `CHECK (id = 1)` in `admins` table.
 - Eliminates multi-tenant privilege escalation and role-confusion attack surfaces.
 ### B. Credential Protection
 - **Passwords**: Hashed with Argon2id using unique cryptographic salts.
 - **API Tokens**: Stored exclusively as SHA-256 digests in SQLite; raw tokens are displayed once upon creation.
 - **Secret Redaction**: Private keys, preshared keys, and password hashes implement custom `std::fmt::Debug` implementations returning `[REDACTED]`.
 ### C. Hardened systemd Sandbox (`nx9-wg.service`)
 ```ini
 [Unit]
 Description=NX9 WireGuard Native VPN Platform
 After=network.target network-online.target
 [Service]
 Type=simple
 ExecStart=/usr/local/bin/nx9-wg serve
 Restart=always
 RestartSec=5s
 # Minimal Linux Capabilities
 CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
 AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
 # Sandboxing Directives
 ProtectSystem=strict
 ProtectHome=true
 PrivateTmp=true
 ProtectControlGroups=true
 RestrictSUIDSGID=true
 LockPersonality=true
 NoNewPrivileges=true
 # Allowed Network Address Families
 RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
 # Explicit Read-Write Paths (for state and direct procfs forwarding)
 ReadWritePaths=/var/lib/nx9-wg /etc/nx9-wg /var/log/nx9-wg /proc/sys/net
 ProtectKernelTunables=false
 # Systemd Directory Management
 StateDirectory=nx9-wg
 ConfigurationDirectory=nx9-wg
 LogsDirectory=nx9-wg
 ```
 ---
 ## 12. Disaster Recovery, Backups & Upgrades
 ### Atomic Online Backups
 - Uses SQLite `VACUUM INTO` to produce non-blocking, consistent binary database snapshots while the daemon is actively serving traffic.
 - Generates SHA-256 manifest records for each snapshot.
 ### Safe Rollback Workflow
 ```
 ┌────────────────────────────────────────────────────────┐
 │ 1. Operator Initiates Restore (nx9-wg backup restore)   │
 └───────────────────────────┬────────────────────────────┘
                             │
 ┌───────────────────────────▼────────────────────────────┐
 │ 2. Verify Backup File Size, Magic Header & SHA-256     │
 └───────────────────────────┬────────────────────────────┘
                             │
 ┌───────────────────────────▼────────────────────────────┐
 │ 3. Create Pre-Restore Safety Snapshot (Automatic Fallback│
 └───────────────────────────┬────────────────────────────┘
                             │
 ┌───────────────────────────▼────────────────────────────┐
 │ 4. Close Connection Pools, Replace .db, Clean WAL/SHM  │
 └───────────────────────────┬────────────────────────────┘
                             │
 ┌───────────────────────────▼────────────────────────────┐
 │ 5. Reopen Database & Execute Reconciliation Engine     │
 │    (Kernel state converged to restored desired state)   │
 └────────────────────────────────────────────────────────┘
 ```
 ---
 ## 13. Release Engineering & Deployment Lifecycle
 ### Automated Packaging Pipeline ([`scripts/package-release.sh`](file:///home/sunil/Programs/nx9-wg/scripts/package-release.sh))
 Generates self-contained, reproducible distribution archives in `target/dist/`:
 - `nx9-wg-v1.0.0-linux-x86_64.tar.gz` (7.8 MB)
 - `nx9-wg-v1.0.0-linux-x86_64.tar.xz` (5.0 MB)
 - `nx9-wg-v1.0.0-linux-x86_64.sha256` (Cryptographic checksum manifest)
 ### Production Filesystem Layout & Permissions
 ```
 /usr/local/bin/nx9-wg               0755  root:root  - Native Executable Binary
 /etc/nx9-wg/                        0750  root:root  - Configuration Directory
 └── config.toml                    0640  root:root  - Production Configuration
 /var/lib/nx9-wg/                    0700  root:root  - State & SQLite Directory
 ├── nx9-wg.db                      0600  root:root  - Authoritative SQLite Database
 ├── nx9-wg.db-wal                  0600  root:root  - WAL Journal
 ├── admin-password                 0600  root:root  - Initial Bootstrap Password
 └── backups/                       0700  root:root  - Backup Snapshots Directory
 /var/log/nx9-wg/                    0750  root:root  - Operational Logs
 /etc/systemd/system/nx9-wg.service  0644  root:root  - Hardened Service Unit
 ```
 ---
 ## 14. Quality Assurance & Verification Evidence
 All quality gates have been executed and verified clean:
 | Quality Gate | Verification Command | Result |
 | :--- | :--- | :---: |
 | **Code Formatting** | `cargo fmt --all -- --check` | **PASS** (Zero diffs) |
 | **Workspace Compilation** | `cargo check --workspace` | **PASS** (Zero errors) |
 | **Workspace Unit Tests** | `cargo test --workspace` | **PASS** (**162 / 162 passed**, 100%) |
 | **Clippy Linter Check** | `cargo clippy --workspace --all-targets --all-features -- -D warnings` | **PASS** (Zero warnings) |
 | **Comprehensive CLI Suite** | `LIVE=0 bash scripts/test-cli-comprehensive.sh` | **PASS** (**203 passed** / 7 skipped) |
 | **Native Integration Suite** | `LIVE=0 bash scripts/test-native-integration.sh` | **PASS** (**19 passed** / 1 skipped) |
 | **Dedicated Live Kernel Suite** | `LIVE=0 bash scripts/test-live-kernel.sh` | **PASS** (**23 passed** / 1 skipped) |
 | **Subprocess Safety Audit** | Automated source scan for `Command::new` | **PASS** (Zero subprocesses) |
 | **Secret Leakage Audit** | Automated audit for plaintext credentials | **PASS** (Zero secrets leaked) |
 | **Standalone Package Verification**| Fresh directory extraction & independent run | **PASS** (Standalone execution) |
 | **Git Diff Whitespace Audit** | `git diff --check` | **PASS** (Zero whitespace issues) |
 ---
 ## 15. Ecosystem & License Summary
 `nx9-wg` is part of the **NX9 Ecosystem** ([https://nx9.in](https://nx9.in)) created by **Sunil Thakare**.
 Dual-licensed under either:
 - **MIT License** ([`LICENSE-MIT`](file:///home/sunil/Programs/nx9-wg/LICENSE-MIT))
 - **Apache License, Version 2.0** ([`LICENSE-APACHE`](file:///home/sunil/Programs/nx9-wg/LICENSE-APACHE))
 at your option.
 ---
 > **NX9 WireGuard** — Sovereign, Self-Hosted, Linux-Native Network Infrastructure.
@@ -392,3 +392,68 @@ For detailed subsystem documentation, see the [**Documentation Index**](docs/REA
 - [Testing Specification](docs/TESTING.md)
 - [Development Guide](docs/development.md)
 - [Docker Deployment](docs/docker.md)
 ## 17. Web UI Screenshots
 The following screenshots provide visual evidence of the production Web UI and its native
 WireGuard/network administration workflow. Sensitive endpoint and cryptographic values in
 the captured evidence have been redacted where applicable.
 ### Dashboard
 ![NX9-WG Dashboard](screenshots/dashboard.png)
 ### Interfaces
 ![NX9-WG Interfaces](screenshots/interfaces.png)
 ### Peer Management
 ![NX9-WG Peers](screenshots/peers.png)
 ### New Peer Enrollment
 ![NX9-WG New Peer Enrollment](screenshots/new_peer_enrollment.png)
 ### Client Configuration Export
 ![NX9-WG Client Configuration](screenshots/peer_config.png)
 ### QR Code Export
 ![NX9-WG QR Export](screenshots/qr_export.png)
 ### Networks
 ![NX9-WG Networks](screenshots/networks.png)
 ### IP Forwarding
 ![NX9-WG IP Forwarding](screenshots/ip_forwarding.png)
 ### NAT & Masquerade
 ![NX9-WG NAT & Masquerade](screenshots/nat_masquerate.png)
 ### Reconciliation
 ![NX9-WG Reconciliation](screenshots/reconciliation.png)
 ### Diagnostics — System, Network & WAN
 ![NX9-WG Diagnostics](screenshots/diagnostics1.png)
 ### Diagnostics — Firewall, NAT, MTU & Reconciliation
 ![NX9-WG Diagnostics Details](screenshots/diagnostics2.png)
 ### Settings
 ![NX9-WG Settings](screenshots/settings.png)
 ### Backups
 ![NX9-WG Backups](screenshots/backup.png)
 > **Additional evidence:** `screenshots/admin-instance.png` contains the captured administrative
 > instance documentation and is retained in the repository alongside the UI screenshots.
@@ -0,0 +1,595 @@
 # NX9-WG Stress Test --- Remote LAN Multi-Path Integration
 **Project:** NX9-WG\
 **Test Type:** Integration / Stress Test\
 **Status:** PASS\
 **Date:** 2026-08-19\
 **Purpose:** Validate robust remote access to a protected LAN through
 NX9-WG across substantially different underlying network paths.
 ------------------------------------------------------------------------
 ## 1. Executive Summary
 This test validates that `nx9-wg` can provide authenticated, routed
 access to a remote `192.168.1.0/24` LAN while the WireGuard client uses
 different and independent Internet/access-network paths.
 The test deliberately exercised NX9-WG through:
 1.  Cellular 5G → mobile WAN AP → laptop.
 2.  Airtel Wi-Fi → Pixel Wi-Fi Station + Access Point concurrency →
    laptop.
 3.  A separate Wi-Fi network → laptop.
 In all tested paths, the same NX9-WG peer successfully established the
 VPN and reached hosts and services on the protected LAN.
 The strongest validation was successful interactive SSH access to
 `192.168.1.200`, in addition to ICMP, HTTP, network filesystem access,
 and LAN host discovery.
 **Overall result: PASS.**
 ------------------------------------------------------------------------
 ## 2. Test Objective
 Validate that NX9-WG:
 -   establishes a WireGuard tunnel independently of the client's
    underlying access network;
 -   correctly routes traffic from a remote peer to the protected LAN;
 -   handles different upstream NAT/network topologies;
 -   provides access to multiple LAN hosts rather than only a single
    endpoint;
 -   supports real application traffic over the tunnel;
 -   preserves connectivity when the client changes access-network
    topology;
 -   does not require changes to the protected LAN or ISP router
    configuration.
 This test is intended as an **integration and robustness validation**,
 not as a throughput benchmark.
 ------------------------------------------------------------------------
 ## 3. Network Topology
 ### Protected LAN
 ``` text
 192.168.1.0/24
 ```
 Representative hosts:
 ``` text
 192.168.1.1      Airtel AirFiber / Nokia AAP321NK
 192.168.1.200    Debian server / Thakares IoT Hub
 ```
 ### NX9-WG peer
 ``` text
 WireGuard interface: Office-Laptop
 WireGuard address:   10.100.0.6/32
 MTU:                 1280
 ```
 The important architectural property is that the client-side underlay
 network can change while the WireGuard overlay identity and protected
 LAN remain unchanged.
 ------------------------------------------------------------------------
 # 4. Test Scenario A --- Cellular 5G
 ## Path
 ``` text
 5G Internet
    │
    ▼
 Mobile phone
    │
    │ WAN AP / hotspot
    ▼
 Laptop
    │
    │ NX9-WG
    ▼
 NX9-WG server
    │
    │ routed LAN access
    ▼
 192.168.1.0/24
 ```
 ## Procedure
 1.  Mobile phone connected to cellular 5G.
 2.  Mobile phone provided WAN/AP connectivity.
 3.  Laptop connected to the mobile AP.
 4.  NX9-WG VPN was enabled.
 5.  Laptop received an Internet address on the mobile network.
 6.  Remote LAN routes became reachable through NX9-WG.
 ## Observed client addressing
 ``` text
 wlan0: 10.137.106.48/24
 WireGuard: 10.100.0.6/32
 ```
 ## Validation
 ### Internet
 ``` text
 ping google.com
 ```
 Result:
 ``` text
 0% packet loss
 ```
 ### LAN host
 ``` text
 ping 192.168.1.200
 ```
 Result:
 ``` text
 0% packet loss
 ```
 ### LAN discovery
 An initial ordinary `nmap -sP 192.168.1.0/24` did not discover hosts
 while the LAN was only reachable through the routed WireGuard path.
 Individual routed hosts were nevertheless reachable.
 This was subsequently validated more comprehensively in Scenario C using
 `nmap` with the active routed path.
 ------------------------------------------------------------------------
 # 5. Test Scenario B --- Wi-Fi STA + AP Concurrency
 This was the more interesting underlay test.
 The Pixel device supports simultaneous Wi-Fi client (STA) and Access
 Point operation.
 ## Path
 ``` text
                 Airtel Wi-Fi
                      │
                      ▼
              Pixel Wi-Fi STA
                      │
              Pixel Wi-Fi AP
                      │
                      ▼
                   Laptop
                      │
                   NX9-WG
                      │
                      ▼
               NX9-WG server
                      │
                      ▼
               192.168.1.0/24
 ```
 ## Procedure
 1.  Pixel connected to Airtel Wi-Fi.
 2.  Pixel simultaneously enabled its Access Point.
 3.  Laptop connected to the Pixel AP.
 4.  Laptop enabled NX9-WG.
 5.  No cellular Internet was used.
 6.  Remote LAN access worked immediately.
 ## Result
 **PASS**
 This demonstrates that NX9-WG remains functional when the client reaches
 the Internet through a Wi-Fi STA/AP-concurrent intermediate device.
 No cellular fallback was required.
 ------------------------------------------------------------------------
 # 6. Test Scenario C --- Remote LAN Application and Host Validation
 A further test was performed from an external Wi-Fi network.
 ## Client addressing
 ``` text
 wlan0:
 10.24.3.48/24
 WireGuard:
 10.100.0.6/32
 ```
 The underlying Wi-Fi network therefore differed from the protected LAN.
 ## Internet validation
 ``` bash
 ping nx9.in
 ```
 Observed:
 ``` text
 5 packets transmitted
 5 received
 0% packet loss
 min/avg/max/mdev:
 82.653 / 93.637 / 101.691 / 7.049 ms
 ```
 ## Public Internet validation
 ``` bash
 ping google.com
 ```
 Observed:
 ``` text
 5 packets transmitted
 5 received
 0% packet loss
 min/avg/max/mdev:
 87.633 / 118.502 / 165.108 / 30.034 ms
 ```
 ## Airtel gateway validation
 ``` bash
 ping 192.168.1.1
 ```
 Observed:
 ``` text
 2 packets transmitted
 2 received
 0% packet loss
 min/avg/max/mdev:
 98.975 / 99.509 / 100.043 / 0.534 ms
 ```
 ## LAN server validation
 ``` bash
 ping 192.168.1.200
 ```
 Observed:
 ``` text
 3 packets transmitted
 3 received
 0% packet loss
 min/avg/max/mdev:
 88.959 / 95.690 / 105.145 / 6.882 ms
 ```
 ------------------------------------------------------------------------
 # 7. LAN Host Discovery
 The routed LAN was scanned using:
 ``` bash
 nmap -sP 192.168.1.0/24
 ```
 The following 17 hosts were discovered:
 ``` text
 192.168.1.1
 192.168.1.3
 192.168.1.4
 192.168.1.5
 192.168.1.6
 192.168.1.7
 192.168.1.8
 192.168.1.9
 192.168.1.10
 192.168.1.13
 192.168.1.18
 192.168.1.20
 192.168.1.60
 192.168.1.64
 192.168.1.100
 192.168.1.152
 192.168.1.200
 ```
 Result:
 ``` text
 17 hosts up
 ```
 This is significant because it validates routed access to the LAN as a
 network segment rather than connectivity to only one predefined host.
 ------------------------------------------------------------------------
 # 8. Application-Level Validation
 ## HTTP
 The NX9-WG client successfully accessed:
 ``` text
 http://192.168.1.200:8008
 ```
 The Thakares IoT Hub web application loaded successfully.
 ## Network filesystem
 The Linux desktop successfully accessed the remote Debian server:
 ``` text
 192.168.1.200
 /home/sunil
 ```
 ## SSH
 The strongest application-level validation was:
 ``` bash
 ssh 192.168.1.200
 ```
 which produced a normal interactive login:
 ``` text
 Linux thakares 6.12.101+deb13-rt-amd64
 Debian GNU/Linux
 x86_64
 ```
 The remote shell was successfully entered and exited normally.
 This confirms:
 -   TCP connectivity;
 -   routing;
 -   return-path routing;
 -   firewall/forwarding compatibility;
 -   SSH service accessibility;
 -   stable encrypted transport;
 -   real bidirectional application traffic.
 ------------------------------------------------------------------------
 # 9. Test Results
  Validation                           Result
  ------------------------------------ --------
  WireGuard tunnel establishment       PASS
  External Internet through tunnel     PASS
  Cellular 5G underlay                 PASS
  Wi-Fi underlay                       PASS
  Wi-Fi STA + AP concurrent underlay   PASS
  Protected LAN reachability           PASS
  Airtel gateway `192.168.1.1`         PASS
  LAN server `192.168.1.200`           PASS
  ICMP                                 PASS
  LAN host discovery                   PASS
  HTTP application                     PASS
  Network filesystem access            PASS
  SSH                                  PASS
  Interactive remote shell             PASS
  Multiple LAN hosts                   PASS
  No cellular dependency               PASS
 ------------------------------------------------------------------------
 # 10. Key Finding
 The same NX9-WG peer:
 ``` text
 10.100.0.6/32
 ```
 successfully accessed:
 ``` text
 192.168.1.0/24
 ```
 while its underlying client network changed.
 Examples included:
 ``` text
 10.137.106.0/24
 10.24.3.0/24
 ```
 This demonstrates a clean separation between:
 -   **underlay:** whatever network currently provides Internet
    connectivity;
 -   **overlay:** the authenticated NX9-WG tunnel;
 -   **protected network:** the routed `192.168.1.0/24` LAN.
 The protected LAN required no modification for these tests.
 ------------------------------------------------------------------------
 # 11. Why This Is a Meaningful Stress Test
 This test does not attempt to measure maximum WireGuard throughput.
 Instead, it stresses the **network topology and routing assumptions** of
 NX9-WG.
 The tested paths introduce:
 -   different client networks;
 -   different NAT environments;
 -   mobile AP routing;
 -   Wi-Fi STA/AP concurrency;
 -   additional network hops;
 -   remote access to an entire private subnet;
 -   multiple simultaneous LAN destinations;
 -   multiple application protocols.
 The successful results indicate that NX9-WG is not dependent on a
 particular client access network.
 ------------------------------------------------------------------------
 # 12. Airtel LAN Collision Use Case
 The test originated from a practical problem involving an Airtel
 AirFiber Nokia AAP321NK using:
 ``` text
 192.168.1.1
 ```
 which overlaps with the existing home LAN addressing.
 Instead of modifying the ISP-controlled router configuration, NX9-WG
 provided an independent routed management path:
 ``` text
 External network
      │
      ▼
   NX9-WG
      │
      ▼
 192.168.1.0/24
      │
      ├── 192.168.1.1
      ├── 192.168.1.200
      └── other LAN hosts
 ```
 This demonstrates a practical operational benefit of the NX9-WG
 architecture: remote authenticated access to infrastructure can remain
 available without requiring ISP router reconfiguration.
 ------------------------------------------------------------------------
 # 13. What This Test Does Not Establish
 This test should **not** be interpreted as a performance benchmark.
 The following remain outside the scope of this test:
 -   maximum throughput;
 -   sustained high-volume transfer;
 -   CPU utilisation;
 -   simultaneous high-volume traffic from many peers;
 -   large-scale concurrent peer testing;
 -   packet-loss recovery under severe loss;
 -   roaming during an active session;
 -   MTU/fragmentation testing under load;
 -   IPv6 routed-LAN testing;
 -   server restart/recovery testing;
 -   client sleep/resume testing;
 -   long-duration soak testing.
 These should be covered by separate tests if required.
 ------------------------------------------------------------------------
 # 14. Recommended Future Stress Tests
 Future NX9-WG validation can extend this matrix with:
 ### Performance
 ``` text
 iperf3
 ```
 -   TCP throughput;
 -   UDP throughput;
 -   bidirectional traffic;
 -   sustained transfers.
 ### Reliability
 -   1-hour soak test;
 -   24-hour soak test;
 -   repeated tunnel reconnects;
 -   server restart;
 -   client suspend/resume.
 ### Mobility
 -   Wi-Fi → 5G;
 -   5G → Wi-Fi;
 -   AP changes while tunnel is active;
 -   changing NAT environments.
 ### Scale
 -   multiple simultaneous peers;
 -   multiple LAN destinations;
 -   concurrent HTTP/SSH/file traffic.
 ### Network edge cases
 -   MTU stress;
 -   fragmentation;
 -   packet loss;
 -   high latency;
 -   jitter;
 -   IPv6.
 ------------------------------------------------------------------------
 # 15. Final Assessment
 **NX9-WG Remote LAN Multi-Path Integration Test: PASS**
 The implementation successfully provided authenticated routed access
 from externally connected clients to the protected `192.168.1.0/24` LAN
 across multiple substantially different network paths.
 The successful SSH session to `192.168.1.200`, access to the IoT Hub,
 network filesystem access, and discovery of 17 LAN hosts provide strong
 practical evidence that the VPN is functioning as a complete remote-LAN
 connectivity solution rather than merely establishing a WireGuard
 handshake.
 ------------------------------------------------------------------------
 **Test conclusion:**
 > NX9-WG successfully maintained functional remote access to the
 > protected LAN independently of the underlying client access network,
 > including cellular, Wi-Fi, and Wi-Fi STA/AP concurrent paths.
 **Status: PASS**
Author	SHA1	Message	Date
thakares	d25846c58c	Clean up release documentation	2026-08-19 15:24:11 +05:30
thakares	2f06c8faa9	Add production UI screenshots	2026-08-19 15:18:32 +05:30
thakares	d710deb8b0	Add production UI screenshots	2026-08-19 15:09:33 +05:30
thakares	a6208ef330	docs: add remote LAN multi-path stress test validation	2026-08-19 13:36:35 +05:30