Reconciliation Engine & State Convergence Architecture

The ReconciliationEngine is the core architectural subsystem of nx9-wg. It implements a continuous, deterministic control loop ensuring the live Linux kernel state matches the authoritative desired state stored in SQLite.

1. The Closed-Loop Reconciliation Cycle

flowchart TD
    subgraph SOT["1. Authoritative Source of Truth"]
        DB[("SQLite Database\n(Desired State)")]
    end

    subgraph Drift["2. Drift Detection & Planning"]
        Live["Query Live Kernel State\n(WireGuard Genl, RTNL, Netfilter, procfs)"]
        Plan["Reconciliation Engine: plan()\n(Read-Only Deterministic Diff)"]
    end

    subgraph Mutation["3. Serialized Apply & Convergence"]
        Lock["Acquire Async Reconcile Mutex Lock"]
        Apply["Execute Native Mutations\n(WireGuard SET_DEVICE, RTNL routes, nftables)"]
        Verify["Post-Apply Verification Diff"]
    end

    subgraph Outcome["4. Convergence Lifecycle State"]
        Converged["Converged (In Sync)\nhas_drift = false"]
        PartialFail["Partial Failure / Drift Remains\n(Descriptive Error & Safe State)"]
    end

    DB --> Plan
    Live --> Plan
    Plan -->|Drift Detected| Lock
    Lock --> Apply
    Apply --> Verify
    Verify -->|Zero Differences| Converged
    Verify -->|Errors Encountered| PartialFail

2. Six Convergence Lifecycle States

Every reconciliation execution produces a structured ReconciliationReport modeling one of six Phase 6 states:

Lifecycle State	Description	Action Required
`Plan`	Read-only calculation of drift between SQLite and kernel.	None (Dry run)
`Applying`	Native mutations actively dispatching across execution planes.	In progress
`Verifying`	Post-apply live query verifying kernel reflects desired state.	In progress
`Converged`	All desired resources verified present in kernel with zero drift.	None (Healthy)
`PartialFailure`	One or more execution planes failed during apply (e.g. EPERM).	Inspect diagnostic remediation hints
`DriftRemains`	Apply completed without crash, but verification detected remaining drift.	Re-evaluate desired configuration

3. Subsystem Drift Detection Matrix

The plan() method calculates exact drift across 5 independent subsystems:

pub struct ReconciliationPlan {
    pub has_drift: bool,
    pub interface_changes: usize,
    pub peer_changes: usize,
    pub route_changes: usize,
    pub firewall_changes: usize,
    pub forwarding_change: bool,
    pub actions: Vec<PlannedAction>,
}

A. WireGuard Interfaces

Checks if desired interfaces (Interface) exist in kernel links via RTNETLINK.
Detects missing interfaces, wrong MTU, down status, or public key mismatch.
Dynamic Port Drift Tolerance: When desired listen_port is None (standard for Upstream interfaces), the reconciler accepts kernel-selected ephemeral dynamic ports without generating false drift.
Orphan Interface Detection: Scans live kernel WireGuard interfaces; any interface present in kernel but absent from SQLite desired state is scheduled for removal (delete_orphan_interface).

B. Cryptographic Peers

Queries live WireGuard device via WG_CMD_GET_DEVICE.
Detects missing peers, changed public keys, altered allowed IPs, or mismatched persistent keepalive intervals.
Role-Aware Cryptokey Routing: Overlay peers are checked against assigned /32 or /128 tunnel addresses, while Upstream provider peers are checked against configured full-tunnel AllowedIPs (0.0.0.0/0, ::/0).

C. Kernel Routes

Queries active kernel routes via RTM_GETROUTE.
Evaluates exact equality on destination CIDR, gateway IP, interface name, and route metric.
Protects host default gateway (192.168.1.1) and physical WAN interfaces from unwanted modifications.

D. nftables Firewall & NAT

Compares desired rules in SQLite against live rules in table inet nx9_wg.
Detects missing rules, priority shifts, or altered NAT masquerade subnet policies.
Outbound NAT masquerading remains scoped exclusively to Overlay client subnets.

E. IP Forwarding

Inspects /proc/sys/net/ipv4/ip_forward and /proc/sys/net/ipv6/conf/all/forwarding.
Flags drift if forwarding is disabled when VPN routing is configured.

4. Mutex Serialization & Concurrency Safety

Reconciliation mutations are protected by an asynchronous Mutex:

Zero Race Conditions: CLI commands (nx9-wg reconcile apply), Web UI actions (POST /api/v1/reconcile/apply), and background periodic cron jobs cannot execute concurrent kernel mutations.
Read-Only Plan Concurrency: Multiple callers can query reconcile plan simultaneously without blocking, as plan() performs read-only queries.

5. Restart Recovery & Multi-Cycle Idempotency

Clean Cold-Start Recovery: When nx9-wg starts or restarts, the background daemon queries the kernel, detects unapplied state from SQLite, and applies all interfaces, peers, routes, and firewall rules in one unified cycle.
Idempotent Convergence: Running reconcile apply multiple times in succession produces zero mutations (NOOP) once convergence is achieved.
Telemetry Protection: Live kernel telemetry (transfer bytes, handshake timestamps) is ingested into memory/events and NEVER overwrites authoritative desired configuration in SQLite.

6. Orphan Interface Removal & Empty-State Guard

Deterministic Orphan Cleanup: When an interface is deleted or an unmanaged kernel device is detected, apply() removes the orphan interface from the Linux kernel.
Empty-Desired-State Safety Guard: If SQLite returns zero desired interfaces while live kernel interfaces are present, apply() aborts immediately with an error rather than mass-deleting kernel interfaces, protecting against catastrophic link destruction during transient database read errors.

5.9 KiB Raw Permalink Blame History