120 lines
5.9 KiB
Markdown
120 lines
5.9 KiB
Markdown
# Reconciliation Engine & State Convergence Architecture
|
|
|
|
The `ReconciliationEngine` is the core architectural subsystem of `nx9-wg`. It implements a continuous, deterministic control loop ensuring the live Linux kernel state matches the authoritative desired state stored in SQLite.
|
|
|
|
---
|
|
|
|
## 1. The Closed-Loop Reconciliation Cycle
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
subgraph SOT["1. Authoritative Source of Truth"]
|
|
DB[("SQLite Database\n(Desired State)")]
|
|
end
|
|
|
|
subgraph Drift["2. Drift Detection & Planning"]
|
|
Live["Query Live Kernel State\n(WireGuard Genl, RTNL, Netfilter, procfs)"]
|
|
Plan["Reconciliation Engine: plan()\n(Read-Only Deterministic Diff)"]
|
|
end
|
|
|
|
subgraph Mutation["3. Serialized Apply & Convergence"]
|
|
Lock["Acquire Async Reconcile Mutex Lock"]
|
|
Apply["Execute Native Mutations\n(WireGuard SET_DEVICE, RTNL routes, nftables)"]
|
|
Verify["Post-Apply Verification Diff"]
|
|
end
|
|
|
|
subgraph Outcome["4. Convergence Lifecycle State"]
|
|
Converged["Converged (In Sync)\nhas_drift = false"]
|
|
PartialFail["Partial Failure / Drift Remains\n(Descriptive Error & Safe State)"]
|
|
end
|
|
|
|
DB --> Plan
|
|
Live --> Plan
|
|
Plan -->|Drift Detected| Lock
|
|
Lock --> Apply
|
|
Apply --> Verify
|
|
Verify -->|Zero Differences| Converged
|
|
Verify -->|Errors Encountered| PartialFail
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Six Convergence Lifecycle States
|
|
|
|
Every reconciliation execution produces a structured `ReconciliationReport` modeling one of six Phase 6 states:
|
|
|
|
| Lifecycle State | Description | Action Required |
|
|
| :--- | :--- | :--- |
|
|
| **`Plan`** | Read-only calculation of drift between SQLite and kernel. | None (Dry run) |
|
|
| **`Applying`** | Native mutations actively dispatching across execution planes. | In progress |
|
|
| **`Verifying`** | Post-apply live query verifying kernel reflects desired state. | In progress |
|
|
| **`Converged`** | All desired resources verified present in kernel with zero drift. | None (Healthy) |
|
|
| **`PartialFailure`** | One or more execution planes failed during apply (e.g. EPERM). | Inspect diagnostic remediation hints |
|
|
| **`DriftRemains`** | Apply completed without crash, but verification detected remaining drift. | Re-evaluate desired configuration |
|
|
|
|
---
|
|
|
|
## 3. Subsystem Drift Detection Matrix
|
|
|
|
The `plan()` method calculates exact drift across 5 independent subsystems:
|
|
|
|
```rust
|
|
pub struct ReconciliationPlan {
|
|
pub has_drift: bool,
|
|
pub interface_changes: usize,
|
|
pub peer_changes: usize,
|
|
pub route_changes: usize,
|
|
pub firewall_changes: usize,
|
|
pub forwarding_change: bool,
|
|
pub actions: Vec<PlannedAction>,
|
|
}
|
|
```
|
|
|
|
### A. WireGuard Interfaces
|
|
- Checks if desired interfaces (`Interface`) exist in kernel links via RTNETLINK.
|
|
- Detects missing interfaces, wrong MTU, down status, or public key mismatch.
|
|
- **Dynamic Port Drift Tolerance**: When desired `listen_port` is `None` (standard for Upstream interfaces), the reconciler accepts kernel-selected ephemeral dynamic ports without generating false drift.
|
|
- **Orphan Interface Detection**: Scans live kernel WireGuard interfaces; any interface present in kernel but absent from SQLite desired state is scheduled for removal (`delete_orphan_interface`).
|
|
|
|
### B. Cryptographic Peers
|
|
- Queries live WireGuard device via `WG_CMD_GET_DEVICE`.
|
|
- Detects missing peers, changed public keys, altered allowed IPs, or mismatched persistent keepalive intervals.
|
|
- **Role-Aware Cryptokey Routing**: Overlay peers are checked against assigned `/32` or `/128` tunnel addresses, while Upstream provider peers are checked against configured full-tunnel AllowedIPs (`0.0.0.0/0, ::/0`).
|
|
|
|
### C. Kernel Routes
|
|
- Queries active kernel routes via `RTM_GETROUTE`.
|
|
- Evaluates exact equality on destination CIDR, gateway IP, interface name, and route metric.
|
|
- Protects host default gateway (`192.168.1.1`) and physical WAN interfaces from unwanted modifications.
|
|
|
|
### D. nftables Firewall & NAT
|
|
- Compares desired rules in SQLite against live rules in `table inet nx9_wg`.
|
|
- Detects missing rules, priority shifts, or altered NAT masquerade subnet policies.
|
|
- Outbound NAT masquerading remains scoped exclusively to Overlay client subnets.
|
|
|
|
### E. IP Forwarding
|
|
- Inspects `/proc/sys/net/ipv4/ip_forward` and `/proc/sys/net/ipv6/conf/all/forwarding`.
|
|
- Flags drift if forwarding is disabled when VPN routing is configured.
|
|
|
|
---
|
|
|
|
## 4. Mutex Serialization & Concurrency Safety
|
|
|
|
Reconciliation mutations are protected by an asynchronous Mutex:
|
|
- **Zero Race Conditions**: CLI commands (`nx9-wg reconcile apply`), Web UI actions (`POST /api/v1/reconcile/apply`), and background periodic cron jobs cannot execute concurrent kernel mutations.
|
|
- **Read-Only Plan Concurrency**: Multiple callers can query `reconcile plan` simultaneously without blocking, as `plan()` performs read-only queries.
|
|
|
|
---
|
|
|
|
## 5. Restart Recovery & Multi-Cycle Idempotency
|
|
|
|
1. **Clean Cold-Start Recovery**: When `nx9-wg` starts or restarts, the background daemon queries the kernel, detects unapplied state from SQLite, and applies all interfaces, peers, routes, and firewall rules in one unified cycle.
|
|
2. **Idempotent Convergence**: Running `reconcile apply` multiple times in succession produces zero mutations (NOOP) once convergence is achieved.
|
|
3. **Telemetry Protection**: Live kernel telemetry (transfer bytes, handshake timestamps) is ingested into memory/events and NEVER overwrites authoritative desired configuration in SQLite.
|
|
|
|
---
|
|
|
|
## 6. Orphan Interface Removal & Empty-State Guard
|
|
|
|
- **Deterministic Orphan Cleanup**: When an interface is deleted or an unmanaged kernel device is detected, `apply()` removes the orphan interface from the Linux kernel.
|
|
- **Empty-Desired-State Safety Guard**: If SQLite returns zero desired interfaces while live kernel interfaces are present, `apply()` aborts immediately with an error rather than mass-deleting kernel interfaces, protecting against catastrophic link destruction during transient database read errors.
|