nx9-wg/docs/TESTING.md

# NX9-WG v1.1.0 — Comprehensive Testing Specification

This document is the authoritative testing and release-acceptance specification for NX9-WG.

NX9-WG is a native Linux WireGuard, networking, firewall/NAT, reconciliation, telemetry, and WebUI control plane. Testing therefore covers both the Rust control plane and the Linux kernel data plane.

> **Release principle:** Passing unit and integration tests does not substitute for physical WireGuard client validation. A production VPN release must distinguish simulated/control-plane evidence from real packet-path evidence.

---

## 1. Testing Philosophy

Testing is layered from deterministic Rust unit tests through native Linux kernel integration and real client acceptance. Each layer has a defined scope and must not be represented as evidence for a different layer.

Primary invariants:

- SQLite is authoritative desired state.
- Linux kernel state is live state.
- Reconciliation is deterministic and idempotent.
- Server-side WireGuard peer `AllowedIPs` represent cryptokey routing, not client routing policy.
- Client-side `AllowedIPs` represent the client's routing policy.
- NAT and forwarding must operate on real packets, not merely generated nftables rules.
- Live peer status must be derived from kernel telemetry.
- Secrets must never leak through logs, CLI output, API responses, or test artifacts.

## 2. Release Quality Gates

| Gate | Command / Evidence | Requirement |
|---|---|---|
| Formatting | `cargo fmt --all -- --check` | PASS |
| Compilation | `cargo check --workspace --all-targets` | PASS |
| Lint | `cargo clippy --workspace --all-targets --all-features -- -D warnings` | PASS |
| Workspace tests | `cargo test --workspace` | All tests PASS |
| Release build | `cargo build --release --workspace` | PASS |
| CLI suite | `LIVE=0 bash scripts/test-cli-comprehensive.sh` | PASS |
| Native integration | `LIVE=0 bash scripts/test-native-integration.sh` | PASS |
| Kernel suite | `LIVE=0 bash scripts/test-live-kernel.sh` | PASS in SAFE baseline; LIVE acceptance requires dedicated host |
| Diff hygiene | `git diff --check` | PASS |
| Physical client | Android WireGuard acceptance | Required for production VPN certification |

## 3. Workspace Unit Tests

Run:

```bash
cargo test --workspace
```

The v1.1.0 documentation baseline records **195 passing workspace tests**. The final release count must always be regenerated after code changes; documentation must never assume that the historical count remains unchanged.

Focused crates may be run independently:

```bash
cargo test -p nx9-wg-core
cargo test -p nx9-wireguard
cargo test -p nx9-wg-api
cargo test -p nx9-wg-db
cargo test -p nx9-wg-network
cargo test -p nx9-wg-ui
```

## 4. Formatting, Compilation & Clippy

```bash
cargo fmt --all -- --check
cargo check --workspace --all-targets
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo build --release --workspace
```

No release is accepted with formatting drift, compiler warnings promoted by `-D warnings`, or a non-reproducible release build.

## 5. Core Domain & Validation Tests

Validate:

- Interface identity and CIDR validation.
- Peer tunnel address validation.
- Full-tunnel, split-tunnel, and custom client profiles.
- Server/client `AllowedIPs` semantic separation.
- Non-overlapping server-side cryptokey routes.
- MTU validation.
- Endpoint validation.
- Interface-name uniqueness.
- Key preservation during interface edits.

## 6. WireGuard Engine Tests

The WireGuard engine must validate:

- Interface creation and deletion.
- Private/public key configuration.
- Listen port and MTU.
- Peer creation/update/removal.
- `ReplaceAllowedIps` behavior.
- Server-side peer routes derived from assigned addresses.
- Learned endpoint and handshake telemetry.
- Idempotent synchronization.

For a road-warrior peer such as `10.100.0.9/32`, the Linux kernel peer must receive `10.100.0.9/32`, not the client's `0.0.0.0/0` full-tunnel route.

## 7. Client Configuration & QR Tests

Verify generated client configuration:

```ini
[Interface]
Address = 10.100.0.9/32

[Peer]
AllowedIPs = 0.0.0.0/0
Endpoint = <configured-server-endpoint>:51820
```

For IPv4-only server interfaces, do not silently export `::/0` unless IPv6 service is actually configured and intended.

Verify:

- Explicit endpoint override wins.
- Persistent `server_endpoint` setting is consumed automatically.
- Endpoint includes a valid UDP port.
- QR generation is equivalent to exported configuration.
- Terminal QR rendering works.
- WebUI QR rendering is present and scannable.

## 8. REST API Tests

Cover:

- Interface CRUD.
- Interface editing.
- Peer CRUD.
- Peer telemetry enrichment.
- Server endpoint settings.
- QR/config export.
- Reconciliation endpoints.
- Diagnostics.
- Authentication and authorization.
- Invalid input and conflict responses.

Telemetry responses must expose current kernel-derived endpoint, handshake, RX, and TX values where available.

## 9. WebUI Tests

Verify:

- Interface list displays existing interfaces.
- Interface Edit action exists.
- Edit form preserves public identity and does not regenerate keys.
- Peer list displays tunnel address and learned endpoint.
- QR action is available.
- Endpoint configuration is visible in Settings.
- Refresh Telemetry obtains fresh kernel state.
- Status states are truthful.

Status semantics:

| State | Condition |
|---|---|
| Connected | Active peer with handshake age < 180 seconds |
| Awaiting Handshake | Active peer with no observed handshake |
| Disconnected | Active peer with handshake age >= 180 seconds |
| Disabled | Peer disabled |
| Expired | Peer expired |
| Revoked | Peer revoked |

## 10. Timestamp & Telemetry Tests

Backend timestamps must carry an explicit UTC offset. Browser parsing must not reinterpret UTC database timestamps as local wall-clock timestamps.

Test:

- Never-handshaken peer.
- Fresh handshake.
- Handshake exactly around the 180-second boundary.
- Stale handshake.
- RX/TX counters increasing.
- Learned endpoint changing due to roaming.
- SQLite telemetry cache update.

## 11. Reconciliation Tests

Required lifecycle:

```text
Desired SQLite State
        ↓
Reconciliation Plan
        ↓
Native Linux Engines
        ↓
Kernel State
        ↓
Live Telemetry
        ↓
Zero Drift
```

Test deliberate drift in:

- Interface address.
- Interface MTU.
- Interface listen port.
- Peer server-side `AllowedIPs`.
- Peer keepalive.
- Routes.
- Firewall/NAT state.

For every mutation:

```bash
sudo nx9-wg reconcile plan
sudo nx9-wg reconcile apply
sudo nx9-wg reconcile plan
```

The final plan must report zero drift.

## 12. Network & Routing Tests

Verify:

- `10.100.0.0/24 dev wg0` exists when `10.100.0.1/24` is assigned.
- Peer `/32` routes resolve through `wg0`.
- No unintended default-route replacement occurs.
- Existing LAN routes remain intact.
- Route deletion/recreation converges safely.
- Split-tunnel routes remain distinct from full-tunnel client routing.

## 13. Forwarding Tests

Verify:

```bash
sudo nx9-wg live forwarding --json
```

Expected IPv4 forwarding is enabled for a full-tunnel road-warrior deployment.

Where IPv6 is not configured, IPv6 forwarding must not create an accidental blackhole or misleading client configuration.

## 14. Firewall & NAT Tests

The managed nftables table is:

```text
table inet nx9_wg
```

Verify:

- Atomic ruleset application.
- Forward chain behavior.
- Established/related traffic handling.
- NAT masquerade scoped to `10.100.0.0/24`.
- No unrelated nftables table is flushed.
- Packet counters increase during real client traffic.

A successful ruleset-generation test is not equivalent to packet-level NAT validation.

## 15. SAFE Mode Testing (`LIVE=0`)

SAFE mode is the default for developer workstations:

```bash
LIVE=0 bash scripts/test-cli-comprehensive.sh
LIVE=0 bash scripts/test-native-integration.sh
LIVE=0 bash scripts/test-live-kernel.sh
```

SAFE mode validates control-plane logic, parsers, deterministic builders, simulations, and read-only kernel inspection without intentionally mutating production networking.

## 16. LIVE Kernel Testing (`LIVE=1`)

LIVE testing requires a dedicated disposable Linux host or VM with appropriate privileges.

```bash
sudo -E LIVE=1 bash scripts/test-native-integration.sh
sudo -E LIVE=1 bash scripts/test-live-kernel.sh
```

The test harness must:

- Capture a baseline.
- Use isolated test resources.
- Avoid changing default routes.
- Avoid modifying unrelated nftables tables.
- Restore forwarding state.
- Remove only resources created by the test.
- Compare post-test state against baseline.

## 17. Comprehensive CLI Suite

Run:

```bash
LIVE=0 bash scripts/test-cli-comprehensive.sh
```

The suite covers CLI commands, help, output formats, authentication, profiles, interfaces, peers, routes, firewall, NAT, forwarding, reconciliation, backup, audit, live inspection, diagnostics, environment handling, explicit database paths, and source-level safety checks.

The documented baseline is **203 passed / 7 skipped**; regenerate the count after any test changes.

## 18. Native Integration Suite

Run:

```bash
LIVE=0 bash scripts/test-native-integration.sh
```

The documented baseline is **19 passed / 1 skipped**.

The suite validates the desired-state-to-native-engine-to-kernel architecture, drift injection, convergence, and diagnostic evidence.

## 19. Dedicated Live-Kernel Suite

Run:

```bash
LIVE=0 bash scripts/test-live-kernel.sh
```

The documented SAFE baseline is **23 passed / 1 skipped**. A true production release should additionally capture a `LIVE=1` evidence run on the intended Linux platform.

## 20. Security, Secrets & Script Safety

Audit requirements:

- No plaintext private keys in normal human-readable diagnostics.
- No passwords in logs.
- No preshared keys in normal status output.
- No token secrets in logs or test output.
- No production subprocess execution from the Rust control plane.
- Scripts use strict shell options and bounded cleanup.
- Live tests use isolated temporary resources.
- Installer/uninstaller operations are explicit and privilege-aware.

The repository scripts are:

- `scripts/install.sh`
- `scripts/package-release.sh`
- `scripts/test-cli-comprehensive.sh`
- `scripts/test-cli-live.sh`
- `scripts/test-live-kernel.sh`
- `scripts/test-native-integration.sh`
- `scripts/uninstall.sh`

## 21. Physical Android / Road-Warrior Acceptance

A real WireGuard client is mandatory evidence for production VPN certification.

Minimum test:

1. Generate QR/config for a test peer.
2. Import into the official Android WireGuard client.
3. Connect over LAN first.
4. Verify `ping 10.100.0.1`.
5. Verify Internet IP connectivity such as `ping 1.1.1.1`.
6. Verify DNS resolution.
7. Load an HTTPS page.
8. Confirm server live telemetry reports endpoint, handshake, RX, and TX.
9. Disconnect and verify stale-state transition.
10. Reconnect and verify telemetry refresh.

For WAN road-warrior certification:

1. Disable Android Wi-Fi.
2. Use 4G/5G cellular data.
3. Configure the public server endpoint.
4. Forward UDP 51820 to the NX9-WG host.
5. Verify handshake, tunnel reachability, DNS, and full-tunnel Internet.

## 22. Release Acceptance Matrix

| Acceptance Gate | v1.1.0 Evidence Status |
|---|---|
| Real Android handshake | **PASS — operator verified** |
| Tunnel control connectivity | **PASS — operator verified** |
| Full-tunnel Internet | **PASS — operator verified** |
| NAT/forwarding data plane | **PASS — operator verified through working Internet path; packet-counter evidence should be retained for formal audit** |
| Disconnect/reconnect status | **PASS — implementation verified; retain physical transition evidence for audit** |
| Interface editing | **PASS — operator verified** |
| Persistent server endpoint / QR | **PASS — operator verified** |
| WebUI live peer status | **PASS — operator verified with connected mobile client** |
| Final reconciliation | **PASS — zero drift observed** |
| IPv6 safety | **PASS — code/config validation; live IPv6 remains deployment-specific** |
| External cellular/WAN road-warrior | **NOT VERIFIED unless separately executed and recorded** |
| Post-reboot physical-client persistence | **NOT VERIFIED unless separately executed and recorded** |

**Release rule:** Do not convert an unexecuted physical gate into PASS merely because simulated or unit tests pass.

## 23. Server Reboot Acceptance

On the actual deployment host:

```bash
sudo reboot
```

After boot:

```bash
sudo systemctl status nx9-wg --no-pager
sudo nx9-wg live interface show wg0 --json
sudo nx9-wg live peer list wg0 --json
sudo nx9-wg reconcile plan
```

Verify that desired state reconstructs:

- `wg0`.
- Interface address.
- Listen port and MTU.
- Server-side peer `/32` routes.
- Forwarding.
- nftables/NAT.
- WebUI/API availability.

Then reconnect a physical client and repeat the data-plane acceptance.

## 24. Release Packaging Verification

Run:

```bash
cargo build --release --workspace
bash scripts/package-release.sh
```

Verify:

- Package name contains `v1.1.0`.
- Binary reports `1.1.0`.
- README and CHANGELOG are included.
- `docs/TESTING.md` is included.
- Installation scripts are executable.
- Archive extracts into an isolated directory.
- SHA-256 manifest matches the generated archives.

## 25. Version & Repository Consistency Audit

Run:

```bash
cargo metadata --no-deps --format-version 1
./target/release/nx9-wg --version
grep -RIn --exclude-dir=.git 'v0.8.0\|0.8.0' .
git diff --check
```

Historical backup/runtime artifacts are not release documentation and must not be packaged as source or distribution state.

All current release-facing references must identify v1.1.0.

## 26. Final Release Command Set

The minimum final gate is:

```bash
cargo fmt --all -- --check
cargo check --workspace --all-targets
cargo clippy --workspace --all-targets --all-features -- -D warnings
cargo test --workspace
cargo build --release --workspace
git diff --check
```

Then execute the appropriate SAFE and LIVE suites, followed by physical client acceptance and package verification.

## 27. Evidence Retention

For a formal release record, retain:

- Exact commit ID.
- `cargo test --workspace` output.
- CLI/integration/kernel test summaries.
- `nx9-wg --version` output.
- `systemctl status nx9-wg` output.
- Live WireGuard interface/peer JSON.
- Reconciliation plan output.
- Android handshake and Internet verification evidence.
- WAN/cellular evidence where performed.
- Reboot evidence where performed.
- Release archive SHA-256 checksums.

This evidence separates **software correctness**, **kernel integration correctness**, and **real-world VPN data-plane correctness**.