# Reconciliation Engine & State Convergence Architecture

The `ReconciliationEngine` is the core architectural subsystem of `nx9-wg`. It implements a continuous, deterministic control loop ensuring the live Linux kernel state matches the authoritative desired state stored in SQLite.

---

## 1. The Closed-Loop Reconciliation Cycle

```mermaid
flowchart TD
    subgraph SOT["1. Authoritative Source of Truth"]
        DB[("SQLite Database\n(Desired State)")]
    end

    subgraph Drift["2. Drift Detection & Planning"]
        Live["Query Live Kernel State\n(WireGuard Genl, RTNL, Netfilter, procfs)"]
        Plan["Reconciliation Engine: plan()\n(Read-Only Deterministic Diff)"]
    end

    subgraph Mutation["3. Serialized Apply & Convergence"]
        Lock["Acquire Async Reconcile Mutex Lock"]
        Apply["Execute Native Mutations\n(WireGuard SET_DEVICE, RTNL routes, nftables)"]
        Verify["Post-Apply Verification Diff"]
    end

    subgraph Outcome["4. Convergence Lifecycle State"]
        Converged["Converged (In Sync)\nhas_drift = false"]
        PartialFail["Partial Failure / Drift Remains\n(Descriptive Error & Safe State)"]
    end

    DB --> Plan
    Live --> Plan
    Plan -->|Drift Detected| Lock
    Lock --> Apply
    Apply --> Verify
    Verify -->|Zero Differences| Converged
    Verify -->|Errors Encountered| PartialFail
```

---

## 2. Six Convergence Lifecycle States

Every reconciliation execution produces a structured `ReconciliationReport` modeling one of six Phase 6 states:

| Lifecycle State | Description | Action Required |
| :--- | :--- | :--- |
| **`Plan`** | Read-only calculation of drift between SQLite and kernel. | None (Dry run) |
| **`Applying`** | Native mutations actively dispatching across execution planes. | In progress |
| **`Verifying`** | Post-apply live query verifying kernel reflects desired state. | In progress |
| **`Converged`** | All desired resources verified present in kernel with zero drift. | None (Healthy) |
| **`PartialFailure`** | One or more execution planes failed during apply (e.g. EPERM). | Inspect diagnostic remediation hints |
| **`DriftRemains`** | Apply completed without crash, but verification detected remaining drift. | Re-evaluate desired configuration |

---

## 3. Subsystem Drift Detection Matrix

The `plan()` method calculates exact drift across 5 independent subsystems:

```rust
pub struct ReconciliationPlan {
    pub has_drift: bool,
    pub interface_changes: usize,
    pub peer_changes: usize,
    pub route_changes: usize,
    pub firewall_changes: usize,
    pub forwarding_change: bool,
    pub actions: Vec<PlannedAction>,
}
```

### A. WireGuard Interfaces
- Checks if desired interfaces (`Interface`) exist in kernel links via RTNETLINK.
- Detects missing interfaces, wrong MTU, down status, or public key mismatch.
- **Dynamic Port Drift Tolerance**: When desired `listen_port` is `None` (standard for Upstream interfaces), the reconciler accepts kernel-selected ephemeral dynamic ports without generating false drift.
- **Orphan Interface Detection**: Scans live kernel WireGuard interfaces; any interface present in kernel but absent from SQLite desired state is scheduled for removal (`delete_orphan_interface`).

### B. Cryptographic Peers
- Queries live WireGuard device via `WG_CMD_GET_DEVICE`.
- Detects missing peers, changed public keys, altered allowed IPs, or mismatched persistent keepalive intervals.
- **Role-Aware Cryptokey Routing**: Overlay peers are checked against assigned `/32` or `/128` tunnel addresses, while Upstream provider peers are checked against configured full-tunnel AllowedIPs (`0.0.0.0/0, ::/0`).

### C. Kernel Routes
- Queries active kernel routes via `RTM_GETROUTE`.
- Evaluates exact equality on destination CIDR, gateway IP, interface name, and route metric.
- Protects host default gateway (`192.168.1.1`) and physical WAN interfaces from unwanted modifications.

### D. nftables Firewall & NAT
- Compares desired rules in SQLite against live rules in `table inet nx9_wg`.
- Detects missing rules, priority shifts, or altered NAT masquerade subnet policies.
- Outbound NAT masquerading remains scoped exclusively to Overlay client subnets.

### E. IP Forwarding
- Inspects `/proc/sys/net/ipv4/ip_forward` and `/proc/sys/net/ipv6/conf/all/forwarding`.
- Flags drift if forwarding is disabled when VPN routing is configured.

---

## 4. Mutex Serialization & Concurrency Safety

Reconciliation mutations are protected by an asynchronous Mutex:
- **Zero Race Conditions**: CLI commands (`nx9-wg reconcile apply`), Web UI actions (`POST /api/v1/reconcile/apply`), and background periodic cron jobs cannot execute concurrent kernel mutations.
- **Read-Only Plan Concurrency**: Multiple callers can query `reconcile plan` simultaneously without blocking, as `plan()` performs read-only queries.

---

## 5. Restart Recovery & Multi-Cycle Idempotency

1. **Clean Cold-Start Recovery**: When `nx9-wg` starts or restarts, the background daemon queries the kernel, detects unapplied state from SQLite, and applies all interfaces, peers, routes, and firewall rules in one unified cycle.
2. **Idempotent Convergence**: Running `reconcile apply` multiple times in succession produces zero mutations (NOOP) once convergence is achieved.
3. **Telemetry Protection**: Live kernel telemetry (transfer bytes, handshake timestamps) is ingested into memory/events and NEVER overwrites authoritative desired configuration in SQLite.

---

## 6. Orphan Interface Removal & Empty-State Guard

- **Deterministic Orphan Cleanup**: When an interface is deleted or an unmanaged kernel device is detected, `apply()` removes the orphan interface from the Linux kernel.
- **Empty-Desired-State Safety Guard**: If SQLite returns zero desired interfaces while live kernel interfaces are present, `apply()` aborts immediately with an error rather than mass-deleting kernel interfaces, protecting against catastrophic link destruction during transient database read errors.