docs: update TESTING.md for v1.0.1 — 95 tests, proptest section, known gaps
This commit is contained in:
thakares committed
1 parent
3225509713
commit
3f445eead5
1 file changed
+168
-117
+168
-117
@@ -2,14 +2,15 @@
|
||||
|
||||
ChronoSeal maintains a rigorous, security-first test suite focused on cryptographic correctness, deterministic server ↔ WASM parity, mutation engine integrity, replay resistance, tampering detection, behavioral validation, and storage reliability.
|
||||
|
||||
As of **v0.6.1**, the project contains **93 passing tests** across the server, WASM, and shared protocol crates.
|
||||
As of **v1.0.1**, the project contains **95 passing tests** across the server, WASM, and shared protocol crates.
|
||||
|
||||
| Crate | Tests |
|
||||
| ------------------- | -----: |
|
||||
| ---------------------------- | ------ |
|
||||
| `chronoseal-server` | 33 |
|
||||
| `chronoseal-wasm` | 24 |
|
||||
| `shared` | 36 |
|
||||
| **Total** | **93** |
|
||||
| `shared` (unit) | 36 |
|
||||
| `shared` (property-based) | 2 |
|
||||
| **Total** | **95** |
|
||||
|
||||
---
|
||||
|
||||
@@ -17,12 +18,12 @@ As of **v0.6.1**, the project contains **93 passing tests** across the server, W
|
||||
|
||||
ChronoSeal testing prioritizes:
|
||||
|
||||
* **Security invariants** over raw coverage metrics
|
||||
* **Deterministic parity** between server and browser WASM runtimes
|
||||
* **Negative-path testing** (tampering, replay, malformed input, edge cases)
|
||||
* **Fuzz-style and randomized testing** for mutation logic
|
||||
* **Performance regression detection**
|
||||
* **Long-term protocol stability**
|
||||
- **Security invariants** over raw coverage metrics
|
||||
- **Deterministic parity** between server and browser WASM runtimes
|
||||
- **Negative-path testing** (tampering, replay, malformed input, edge cases)
|
||||
- **Property-based and fuzz-style testing** for mutation logic and VM robustness
|
||||
- **Performance regression detection**
|
||||
- **Long-term protocol stability**
|
||||
|
||||
Particular emphasis is placed on ensuring that browser-side WASM execution produces identical results to server-side validation.
|
||||
|
||||
@@ -34,21 +35,21 @@ Particular emphasis is placed on ensuring that browser-side WASM execution produ
|
||||
|
||||
Configuration tests verify:
|
||||
|
||||
* Database backend selection
|
||||
* TOML configuration parsing
|
||||
* Command-line override behavior
|
||||
* Default configuration values
|
||||
* Runtime initialization logic
|
||||
- Database backend selection
|
||||
- TOML configuration parsing
|
||||
- Command-line override behavior
|
||||
- Default configuration values
|
||||
- Runtime initialization logic
|
||||
|
||||
Supported backends include:
|
||||
|
||||
* `sqlite-in-memory`
|
||||
* `sqlite-in-disk`
|
||||
* `valkey`
|
||||
- `sqlite-in-memory`
|
||||
- `sqlite-in-disk`
|
||||
- `valkey` (redis-compatible via r2d2 connection pool)
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_apply_run_args_overrides_db_type
|
||||
test_default_db_type_is_sqlite_in_memory
|
||||
test_toml_parses_db_type_kebab_case
|
||||
@@ -64,17 +65,17 @@ test_valkey_store_operations
|
||||
|
||||
Session tests validate:
|
||||
|
||||
* Session creation
|
||||
* Public key validation
|
||||
* Expiration handling
|
||||
* Replay attack prevention
|
||||
* Mutation step enforcement
|
||||
* Commitment verification
|
||||
* Long-running deterministic parity
|
||||
- Session creation
|
||||
- Public key validation
|
||||
- Expiration handling
|
||||
- Replay attack prevention
|
||||
- Mutation step enforcement
|
||||
- Commitment verification
|
||||
- Long-running deterministic parity
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_create_session_rejects_invalid_public_key_length
|
||||
test_expired_session_is_rejected
|
||||
test_replay_attack_is_rejected
|
||||
@@ -93,17 +94,17 @@ The Synthetic Gene Mutation Engine is one of the most security-critical componen
|
||||
|
||||
Testing focuses on:
|
||||
|
||||
* Deterministic server/client parity
|
||||
* Mutation order execution
|
||||
* Gene state integrity
|
||||
* Preview → Commit → Discard lifecycle
|
||||
* Randomized mutation programs
|
||||
* Edge-case validation
|
||||
* Performance regression detection
|
||||
- Deterministic server/client parity
|
||||
- Mutation order execution
|
||||
- Gene state integrity
|
||||
- Preview → Commit → Discard lifecycle
|
||||
- Randomized mutation programs
|
||||
- Edge-case validation
|
||||
- Performance regression detection
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_server_client_parity_across_random_orders
|
||||
test_generate_order_is_deterministic_for_seeded_rng
|
||||
test_invalid_positions_wrap_deterministically
|
||||
@@ -118,15 +119,15 @@ test_performance_smoke_mutation_execution
|
||||
|
||||
Heartbeat validation tests verify:
|
||||
|
||||
* Successful state advancement
|
||||
* Silent rejection behavior
|
||||
* Commitment validation
|
||||
* Rate limiting
|
||||
* Next-state mutation generation
|
||||
- Successful state advancement
|
||||
- Silent rejection behavior
|
||||
- Commitment validation
|
||||
- Rate limiting
|
||||
- Next-state mutation generation
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_handler_success_returns_next_mutation_fields
|
||||
test_handler_tampered_commitment_is_silent_failure
|
||||
test_handler_rate_limit_returns_no_mutation_data
|
||||
@@ -138,16 +139,16 @@ test_handler_rate_limit_returns_no_mutation_data
|
||||
|
||||
Behavioral validation tests verify:
|
||||
|
||||
* Minimum mouse activity
|
||||
* Minimum movement distance
|
||||
* Pause detection
|
||||
* Speed thresholds
|
||||
* Optional activity requirements
|
||||
* Fingerprint-related validation paths
|
||||
- Minimum mouse activity
|
||||
- Minimum movement distance
|
||||
- Pause detection
|
||||
- Speed thresholds
|
||||
- Optional activity requirements
|
||||
- Fingerprint-related validation paths
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_validate_mouse_success
|
||||
test_validate_mouse_insufficient_events
|
||||
test_validate_mouse_insufficient_distance
|
||||
@@ -162,23 +163,28 @@ test_validate_mouse_require_activity_toggle
|
||||
|
||||
Storage tests verify:
|
||||
|
||||
* SQLite in-memory operation
|
||||
* SQLite disk-backed operation & pool concurrency
|
||||
* Valkey compatibility mode & pool concurrency operations
|
||||
* Session CRUD behavior
|
||||
* Expiration cleanup
|
||||
* Runtime statistics reporting
|
||||
- SQLite in-memory operation
|
||||
- SQLite disk-backed operation and pool concurrency
|
||||
- Valkey compatibility mode and r2d2 pool concurrency
|
||||
- Session CRUD behavior including the `opcodes` field
|
||||
- Expiration cleanup
|
||||
- Runtime statistics reporting
|
||||
|
||||
These tests ensure storage implementations remain interchangeable without affecting protocol behavior.
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_sqlite_pool_concurrency
|
||||
test_valkey_pool_concurrency
|
||||
test_valkey_store_operations
|
||||
```
|
||||
|
||||
> **Note:** The concurrent write collision path in `update_session` (the `old_last_hash`
|
||||
> optimistic concurrency guard) is not yet covered by an automated test. Two goroutines
|
||||
> advancing the same chain simultaneously is a security-relevant race condition.
|
||||
> A dedicated test is planned for v1.1.0 (see Future Improvements).
|
||||
|
||||
---
|
||||
|
||||
## 7. VM Core
|
||||
@@ -187,28 +193,19 @@ The VM core is tested extensively across both WASM and shared crates.
|
||||
|
||||
Coverage includes:
|
||||
|
||||
* ADD
|
||||
* SUB
|
||||
* MUL
|
||||
* XOR
|
||||
* AND
|
||||
* OR
|
||||
* NOT
|
||||
* HASH
|
||||
* ROT
|
||||
* PUSH
|
||||
- ADD, SUB, MUL, XOR, AND, OR, NOT, HASH, ROT, PUSH
|
||||
|
||||
Edge cases include:
|
||||
|
||||
* Stack underflow
|
||||
* Truncated instructions
|
||||
* Unknown opcodes
|
||||
* Wrapping arithmetic
|
||||
* Invalid instruction streams
|
||||
- Stack underflow
|
||||
- Truncated instructions
|
||||
- Unknown opcodes
|
||||
- Wrapping arithmetic
|
||||
- Invalid instruction streams
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_add
|
||||
test_add_wrapping
|
||||
test_sub
|
||||
@@ -224,16 +221,38 @@ test_rejects_truncated_instruction
|
||||
|
||||
---
|
||||
|
||||
## 8. Property-Based Tests
|
||||
|
||||
ChronoSeal uses [`proptest`](https://github.com/proptest-rs/proptest) for property-based testing of core protocol invariants against arbitrary random input.
|
||||
|
||||
Tests live in `shared/tests/proptests.rs` and run as part of `cargo test --workspace`.
|
||||
|
||||
```
|
||||
test_vm_execute_never_panics
|
||||
test_gene_environment_roundtrip_never_panics
|
||||
```
|
||||
|
||||
`test_vm_execute_never_panics` feeds arbitrary `Vec<u8>` byte sequences into the stack machine
|
||||
and asserts that execution never panics and that the instruction pointer never exceeds the
|
||||
program length. This guards against any future VM opcode handler introducing undefined
|
||||
behaviour on malformed input.
|
||||
|
||||
`test_gene_environment_roundtrip_never_panics` feeds arbitrary bytes into
|
||||
`gene::decode_environment` and asserts graceful failure rather than a panic, covering the full
|
||||
space of malformed environment payloads a client could send.
|
||||
|
||||
---
|
||||
|
||||
# Server Test Coverage (`chronoseal-server`)
|
||||
|
||||
The server crate currently contains **33 tests** covering:
|
||||
|
||||
* Configuration
|
||||
* Runtime initialization
|
||||
* Session management
|
||||
* Heartbeat validation
|
||||
* Rate limiting
|
||||
* Trust validation
|
||||
- Configuration
|
||||
- Runtime initialization
|
||||
- Session management
|
||||
- Heartbeat validation
|
||||
- Rate limiting
|
||||
- Trust validation
|
||||
|
||||
The server tests focus heavily on protocol enforcement and security validation.
|
||||
|
||||
@@ -243,16 +262,16 @@ The server tests focus heavily on protocol enforcement and security validation.
|
||||
|
||||
The WASM crate currently contains **24 tests** covering:
|
||||
|
||||
* VM execution
|
||||
* Browser-side mutation lifecycle
|
||||
* Gene initialization
|
||||
* Mutation preview
|
||||
* Mutation commit/discard behavior
|
||||
* Deterministic parity with shared logic
|
||||
- VM execution
|
||||
- Browser-side mutation lifecycle
|
||||
- Gene initialization
|
||||
- Mutation preview
|
||||
- Mutation commit/discard behavior
|
||||
- Deterministic parity with shared logic
|
||||
|
||||
Example tests:
|
||||
|
||||
```text
|
||||
```
|
||||
test_preview_commitment_matches_shared_engine
|
||||
test_commit_applies_preview
|
||||
test_discard_preview_keeps_committed_state
|
||||
@@ -265,13 +284,14 @@ These tests ensure browser-generated commitments remain consistent with server e
|
||||
|
||||
# Shared Crate Coverage (`shared`)
|
||||
|
||||
The shared crate currently contains **36 tests** and represents the core protocol implementation used by both server and browser runtimes.
|
||||
The shared crate currently contains **36 unit tests** and **2 property-based tests**, representing
|
||||
the core protocol implementation used by both server and browser runtimes.
|
||||
|
||||
Coverage includes:
|
||||
|
||||
### Synthetic Gene Engine
|
||||
|
||||
```text
|
||||
```
|
||||
test_new_state_with_default_size
|
||||
test_new_state_rejects_invalid_sizes
|
||||
test_commitment_changes_when_gene_or_environment_changes
|
||||
@@ -281,7 +301,7 @@ test_table_driven_randomized_environment_roundtrip
|
||||
|
||||
### Mutation Engine
|
||||
|
||||
```text
|
||||
```
|
||||
test_opcode_insert
|
||||
test_opcode_delete
|
||||
test_opcode_mutate_point
|
||||
@@ -292,7 +312,7 @@ test_mutation_chain
|
||||
|
||||
### Validation & Hardening
|
||||
|
||||
```text
|
||||
```
|
||||
test_rejects_stack_underflow
|
||||
test_rejects_truncated_instruction
|
||||
test_rejects_unknown_opcode
|
||||
@@ -301,7 +321,7 @@ test_zero_length_gene_is_rejected
|
||||
|
||||
### Deterministic Parity
|
||||
|
||||
```text
|
||||
```
|
||||
test_server_client_parity_across_random_orders
|
||||
test_generate_order_is_deterministic_for_seeded_rng
|
||||
test_invalid_positions_wrap_deterministically
|
||||
@@ -309,25 +329,47 @@ test_invalid_positions_wrap_deterministically
|
||||
|
||||
### Fuzz & Regression Testing
|
||||
|
||||
```text
|
||||
```
|
||||
test_fuzz_style_random_program_bytes_do_not_diverge
|
||||
test_performance_smoke_mutation_execution
|
||||
test_vm_instruction_budget_soft_cap
|
||||
```
|
||||
|
||||
### Property-Based Tests (`shared/tests/proptests.rs`)
|
||||
|
||||
```
|
||||
test_vm_execute_never_panics
|
||||
test_gene_environment_roundtrip_never_panics
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
# Tooling Crates
|
||||
|
||||
## `chronoseal-replay`
|
||||
|
||||
A standalone replay and audit tool for offline verification of recorded ChronoSeal session
|
||||
chains. It is a developer and forensic utility, not a library, and currently carries no
|
||||
automated tests. Integration tests against captured session fixtures are planned.
|
||||
|
||||
## `fuzz/`
|
||||
|
||||
Contains libFuzzer targets for deeper coverage of the VM and gene codec. Run separately
|
||||
via `cargo +nightly fuzz run <target>` — not part of the standard `cargo test` suite.
|
||||
|
||||
---
|
||||
|
||||
# Running the Test Suite
|
||||
|
||||
Run the full workspace:
|
||||
|
||||
```bash
|
||||
```
|
||||
cargo test --workspace
|
||||
```
|
||||
|
||||
Run individual crates:
|
||||
|
||||
```bash
|
||||
```
|
||||
cargo test -p shared
|
||||
cargo test -p chronoseal-wasm
|
||||
cargo test -p chronoseal-server
|
||||
@@ -335,7 +377,7 @@ cargo test -p chronoseal-server
|
||||
|
||||
Display test output:
|
||||
|
||||
```bash
|
||||
```
|
||||
cargo test -- --nocapture
|
||||
```
|
||||
|
||||
@@ -343,18 +385,22 @@ cargo test -- --nocapture
|
||||
|
||||
# Critical Security Tests
|
||||
|
||||
The following tests protect ChronoSeal's core protocol guarantees and should be treated as **release-blocking** if they fail:
|
||||
The following tests protect ChronoSeal's core protocol guarantees and should be treated as
|
||||
**release-blocking** if they fail:
|
||||
|
||||
```text
|
||||
```
|
||||
test_mutation_commitment_tamper_is_rejected
|
||||
test_replay_attack_is_rejected
|
||||
test_handler_tampered_commitment_is_silent_failure
|
||||
test_server_client_parity_across_random_orders
|
||||
test_deterministic_server_client_parity_across_many_heartbeats
|
||||
test_fuzz_style_random_program_bytes_do_not_diverge
|
||||
test_vm_execute_never_panics
|
||||
test_gene_environment_roundtrip_never_panics
|
||||
```
|
||||
|
||||
These tests directly validate resistance to replay attacks, protocol divergence, mutation tampering, and commitment forgery.
|
||||
These tests directly validate resistance to replay attacks, protocol divergence, mutation
|
||||
tampering, commitment forgery, and VM panic on adversarial input.
|
||||
|
||||
---
|
||||
|
||||
@@ -365,40 +411,45 @@ When adding new functionality:
|
||||
1. Prefer placing protocol logic tests in `shared/`
|
||||
2. Ensure server ↔ WASM parity is validated
|
||||
3. Include negative-path test cases
|
||||
4. Add randomized testing where appropriate
|
||||
4. Add randomized or property-based testing where appropriate
|
||||
5. Update this document when introducing major new categories
|
||||
|
||||
---
|
||||
|
||||
# Future Improvements
|
||||
|
||||
Planned enhancements include:
|
||||
|
||||
* Property-based testing using `proptest`
|
||||
* Browser-driven end-to-end integration tests
|
||||
* Valkey concurrency and failover testing
|
||||
* Automated benchmark execution in CI
|
||||
* Expanded mutation-engine fuzzing
|
||||
* CI-enforced performance regression thresholds
|
||||
- **Concurrent chain write collision test** — verify that two simultaneous heartbeats for the
|
||||
same session are handled correctly by the `old_last_hash` optimistic concurrency guard in
|
||||
`update_session` (security-critical, planned for v1.1.0)
|
||||
- **Browser-driven end-to-end integration tests** — full Playwright or wasm-bindgen-test
|
||||
harness exercising the complete init → heartbeat loop in a real browser environment
|
||||
- **Valkey failover testing** — verify graceful degradation and reconnection under r2d2 pool
|
||||
exhaustion and server-side connection drops
|
||||
- **Automated benchmark execution in CI** — enforce performance regression thresholds for
|
||||
mutation engine and hash chain operations
|
||||
- **Expanded mutation-engine fuzzing** — additional libFuzzer targets for `vm_extensions`
|
||||
opcodes introduced in v0.7.0
|
||||
- **CI-enforced performance regression thresholds** — gate releases on measured latency bounds
|
||||
|
||||
---
|
||||
|
||||
# Conclusion
|
||||
|
||||
ChronoSeal's testing strategy is centered on preserving deterministic behavior, cryptographic correctness, and protocol integrity.
|
||||
ChronoSeal's testing strategy is centered on preserving deterministic behavior, cryptographic
|
||||
correctness, and protocol integrity.
|
||||
|
||||
The current suite of **93 tests** provides broad coverage across:
|
||||
The current suite of **95 tests** provides broad coverage across:
|
||||
|
||||
* Session security
|
||||
* Heartbeat validation
|
||||
* Mutation engine correctness
|
||||
* Deterministic server/WASM parity
|
||||
* Trust validation
|
||||
* Storage abstraction
|
||||
* Replay resistance
|
||||
* Protocol hardening
|
||||
- Session security
|
||||
- Heartbeat validation
|
||||
- Mutation engine correctness
|
||||
- Deterministic server/WASM parity
|
||||
- Trust and behavioral validation
|
||||
- Storage abstraction
|
||||
- Replay resistance
|
||||
- Protocol hardening
|
||||
- Property-based VM and gene codec robustness
|
||||
|
||||
Maintaining and expanding this test suite remains a core project priority as ChronoSeal evolves.
|
||||
|
||||
**Last Updated:** May 2026 (v0.6.1)
|
||||
|
||||
**Last Updated:** May 2026 (v1.0.1)
|
||||
Reference in new issue
Block a user