From 4c697e9adf4663b3f3e09090074831a3d6630213 Mon Sep 17 00:00:00 2001 From: Sunil Thakare Date: Thu, 23 Jul 2026 13:15:26 +0530 Subject: [PATCH] docs: expand runtime lifecycle architecture documentation --- docs/RUNTIME_LIFECYCLE.md | 1441 +++++++++++++++++++++++++++++++++++-- 1 file changed, 1390 insertions(+), 51 deletions(-) diff --git a/docs/RUNTIME_LIFECYCLE.md b/docs/RUNTIME_LIFECYCLE.md index b0a5135..a77f54f 100644 --- a/docs/RUNTIME_LIFECYCLE.md +++ b/docs/RUNTIME_LIFECYCLE.md @@ -1,63 +1,1402 @@ # Runtime Lifecycle Subsystem -The `nx9-auth` runtime lifecycle subsystem provides an enterprise-grade, lock-free, deterministic architecture for application startup, dependency assembly, operational observability, background worker coordination, prioritized shutdown hooks, and graceful HTTP server termination. +> **Project:** NX9-Auth\ +> **Release baseline:** v0.3.0\ +> **Scope:** Application runtime, startup, serving, signals, workers, +> shutdown hooks, cancellation, metrics, and resource termination\ +> **Primary implementation:** `src/runtime/` -## Architecture Overview +------------------------------------------------------------------------ -``` -CLI Commands / binary entrypoint (main.rs) - │ - ▼ -ApplicationBuilder - │ - ├── Database Initialization (SQLite / PostgreSQL) - ├── Repository Provider Assembly - ├── AppState Construction - └── Router Construction (Axum API + SPA UI) - │ - ▼ -Application Container (Lifecycle) - │ - ├── AtomicRuntimeState Machine - ├── SignalManager (SIGINT / SIGTERM) - ├── ShutdownCoordinator (CancellationToken Hierarchy) - ├── WorkerManager (Task Groups) - ├── HookRegistry (Prioritized Shutdown Hooks) - └── RuntimeMetrics - │ - ▼ -axum::serve (HTTP Server) +## 1. Purpose + +The NX9-Auth runtime lifecycle subsystem provides the orchestration +boundary between the CLI entrypoint and the long-running HTTP +application. It coordinates configuration, dependency assembly, database +initialization, Axum router construction, listener binding, background +workers, operating-system signals, graceful shutdown, prioritized +cleanup hooks, runtime metrics, and final resource release. + +The subsystem is intentionally explicit. Runtime state is represented by +a monotonic state machine rather than inferred from scattered booleans +or task state. Shutdown is coordinated rather than ad hoc, and worker +termination precedes final resource cleanup. + +------------------------------------------------------------------------ + +## 2. Design Goals + +The lifecycle architecture is designed around these goals: + +1. **Deterministic startup** --- dependencies are initialized in a + defined order before serving traffic. +2. **Deterministic shutdown** --- termination follows an explicit + sequence from draining through resource closure. +3. **Monotonic runtime state** --- normal lifecycle transitions move + forward only. +4. **Low synchronization overhead** --- lifecycle state uses atomic + operations rather than a mutex. +5. **Explicit ownership** --- the `Application` container owns or + coordinates the principal runtime components. +6. **Graceful HTTP termination** --- in-flight requests are given an + opportunity to complete. +7. **Bounded worker shutdown** --- background tasks are cancelled and + joined with timeout handling. +8. **Ordered cleanup** --- shutdown hooks execute by defined priority. +9. **Signal-aware operation** --- normal termination signals initiate + graceful shutdown; escalation can force termination. +10. **Testability** --- state transitions, worker handling, builder + behavior, and hook ordering are independently testable. + +------------------------------------------------------------------------ + +## 3. Architectural Principles + +The runtime follows several engineering principles. + +### 3.1 Explicit lifecycle over implicit lifecycle + +Startup and shutdown are represented by named states. This makes +operational behavior observable and prevents lifecycle logic from being +distributed invisibly across unrelated modules. + +### 3.2 Construction before exposure + +The HTTP server is not considered running until configuration, database +initialization, migrations, application state, and router assembly have +completed. + +### 3.3 Cancellation before destruction + +Background work is asked to terminate before resources it may depend on +are released. + +### 3.4 Ordered cleanup + +Cleanup operations with dependencies can be expressed through +shutdown-hook priority instead of relying on incidental registration +order. + +### 3.5 Idempotent shutdown initiation + +Multiple shutdown triggers must not cause multiple independent teardown +sequences. + +------------------------------------------------------------------------ + +## 4. High-Level Architecture + +``` mermaid +flowchart TD + CLI["CLI entrypoint"] --> CFG["Load configuration"] + CFG --> BUILDER["ApplicationBuilder"] + BUILDER --> DB["Initialize database"] + DB --> MIG["Run migrations"] + MIG --> STATE["Build AppState"] + STATE --> ROUTER["Build Axum router"] + ROUTER --> APP["Application"] + APP --> LISTENER["Bind TCP listener"] + LISTENER --> SERVER["Axum HTTP server"] + APP --> WORKERS["WorkerManager"] + APP --> SIGNALS["SignalManager"] + APP --> CANCEL["ShutdownCoordinator"] + APP --> HOOKS["HookRegistry"] + APP --> METRICS["RuntimeMetrics"] ``` -## Lifecycle States (`RuntimeState`) +This diagram deliberately uses simple Mermaid labels so that it renders +consistently on GitHub. -The state machine is lock-free and driven by `AtomicU8` with `compare_exchange` transitions. +------------------------------------------------------------------------ -| State | Value | Description | -| :--- | :--- | :--- | -| `Initializing` | 0 | Runtime configuration loading and dependency assembly. | -| `Starting` | 1 | Database connection pool init, migrations, router assembly. | -| `Running` | 2 | HTTP server bound and actively serving requests. | -| `Draining` | 3 | Shutdown signal received; server stops accepting new connections, draining existing HTTP requests. | -| `StoppingWorkers` | 4 | Cancelling and joining active background worker tasks. | -| `ExecutingHooks` | 5 | Executing registered shutdown hooks in priority order (`First` -> `Normal` -> `Last`). | -| `ClosingResources` | 6 | Closing database connection pools and flushing logs. | -| `Stopped` | 7 | All resources released cleanly; runtime process exits with status 0. | +## 5. Runtime Component Map -## Startup Sequence + ------------------------------------------------------------------------------- + Component Primary location Responsibility + ----------------------- ------------------------------- ----------------------- + `Application` `src/runtime/application.rs` Central lifecycle + container and + orchestration -1. `main()` parses CLI flags and loads configuration via `Config::find_and_load()`. -2. `run_server()` invokes `Application::builder(config).build().await`. -3. `ApplicationBuilder` creates `Application` and executes `initialize()`. -4. `initialize()` transitions state to `Starting`, connects database pool, executes migrations, and builds `Router`. -5. `app.start().await` transitions state to `Running`, binds `TcpListener`, prints `Listening on `, and awaits `axum::serve`. + `ApplicationBuilder` `src/runtime/builder.rs` Dependency assembly and + application + initialization -## Graceful Shutdown Sequence + `AtomicRuntimeState` `src/runtime/state.rs` Lock-free lifecycle + state tracking -1. `SIGINT` (Ctrl+C) or `SIGTERM` signal received by `SignalManager` or `ShutdownCoordinator`. -2. `axum::serve` completes its graceful shutdown loop, stopping the TCP listener. -3. State transitions to `Draining`. -4. State transitions to `StoppingWorkers`; `WorkerManager` cancels and joins task groups. -5. State transitions to `ExecutingHooks`; `HookRegistry` executes registered hooks. -6. State transitions to `ClosingResources`; `PoolHandle` closes the database pool. -7. State transitions to `Stopped`; application returns `Ok(())` with exit status 0. + `SignalManager` `src/runtime/signals.rs` SIGINT/SIGTERM handling + and escalation + + `ShutdownCoordinator` `src/runtime/cancellation.rs` Cancellation + propagation + + `WorkerManager` `src/runtime/workers.rs` Background task + grouping, tracking, and + termination + + `HookRegistry` `src/runtime/hooks.rs` Prioritized + asynchronous shutdown + hooks + + `RuntimeMetrics` `src/runtime/metrics.rs` Runtime-level + operational counters + + `Lifecycle` `src/runtime/lifecycle.rs` Lifecycle abstraction + implemented by the + application + ------------------------------------------------------------------------------- + +------------------------------------------------------------------------ + +## 6. Ownership Model + +`Application` acts as the lifecycle ownership boundary. + +``` mermaid +flowchart TD + APP["Application"] + APP --> CONFIG["Config"] + APP --> PROVIDER["Repository provider"] + APP --> POOL["Database pool handle"] + APP --> ROUTER["Axum router"] + APP --> STATE["AtomicRuntimeState"] + APP --> WORKERS["WorkerManager"] + APP --> SIGNALS["SignalManager"] + APP --> CANCEL["ShutdownCoordinator"] + APP --> HOOKS["HookRegistry"] + APP --> METRICS["RuntimeMetrics"] +``` + +The model avoids global mutable lifecycle state. Shared runtime +coordination is performed through purpose-specific primitives such as +atomics, cancellation tokens, task sets, and shared application state. + +------------------------------------------------------------------------ + +## 7. Dependency Assembly + +The builder separates construction from execution. + +``` mermaid +sequenceDiagram + participant Main + participant Builder + participant Database + participant State + participant Router + participant App + + Main->>Builder: build + Builder->>Database: initialize provider + Database-->>Builder: provider and pool + Builder->>Database: run migrations + Builder->>State: construct AppState + State-->>Builder: application state + Builder->>Router: build router + Router-->>Builder: router + Builder->>App: assemble runtime + App-->>Main: initialized application +``` + +This boundary is important because the runtime can fail during +construction without exposing a partially serving HTTP process. + +------------------------------------------------------------------------ + +## 8. Lifecycle State Model + +NX9-Auth defines the following lifecycle states. + + ------------------------------------------------------------------------ + State Value Meaning + --------------------- ---------------------------- --------------------- + `Initializing` 0 Configuration and + initial runtime + construction + + `Starting` 1 Database, migrations, + state, router, and + server preparation + + `Running` 2 HTTP listener is + active and the + service is + operational + + `Draining` 3 Shutdown has started + and HTTP traffic is + being drained + + `StoppingWorkers` 4 Background workers + are being cancelled + and joined + + `ExecutingHooks` 5 Registered shutdown + hooks are executing + + `ClosingResources` 6 Long-lived resources + are being released + + `Stopped` 7 Lifecycle termination + is complete + ------------------------------------------------------------------------ + +------------------------------------------------------------------------ + +## 9. Deterministic State Machine + +``` mermaid +stateDiagram-v2 + [*] --> Initializing + Initializing --> Starting + Starting --> Running + Running --> Draining + Draining --> StoppingWorkers + StoppingWorkers --> ExecutingHooks + ExecutingHooks --> ClosingResources + ClosingResources --> Stopped + Stopped --> [*] +``` + +Normal transitions are strictly forward. A runtime state must not +regress from a later lifecycle phase to an earlier phase. + +------------------------------------------------------------------------ + +## 10. Runtime State Invariants + +The lifecycle design is intended to preserve these invariants: + +- `Running` cannot be reached before startup initialization succeeds. +- Normal state transitions are monotonic. +- Shutdown initiation is guarded against duplicate execution. +- Worker termination occurs before final resource closure. +- Shutdown hooks execute after worker shutdown begins and before final + resource release. +- `Stopped` represents completion of the lifecycle sequence. +- Emergency state advancement is explicit rather than silently + violating transition rules. + +------------------------------------------------------------------------ + +## 11. Atomic Runtime State + +`AtomicRuntimeState` stores lifecycle state using `AtomicU8`. + +The compact representation is suitable because the state space is small +and fixed. Atomic operations allow state inspection and transitions +without introducing a mutex solely for lifecycle coordination. + +The implementation uses atomic memory ordering, including `Acquire` and +`AcqRel`, to coordinate visibility of lifecycle changes between +asynchronous execution contexts. + +### Why an atomic state machine? + +- State reads are inexpensive. +- Transition attempts can use compare-and-exchange semantics. +- Duplicate transition attempts can be detected. +- Shutdown initiation can be guarded without a global lock. +- The state machine remains independent of Tokio scheduling. + +`force_set()` exists for exceptional or escalation paths and should not +replace validated normal transitions. + +------------------------------------------------------------------------ + +## 12. Startup Lifecycle + +The normal startup path is: + +``` mermaid +flowchart TD + A["Process starts"] --> B["Parse CLI"] + B --> C["Load configuration"] + C --> D["Create ApplicationBuilder"] + D --> E["Initialize database provider"] + E --> F["Run migrations"] + F --> G["Construct AppState"] + G --> H["Construct Axum router"] + H --> I["Transition to Starting"] + I --> J["Bind TCP listener"] + J --> K["Transition to Running"] + K --> L["Serve HTTP"] +``` + +------------------------------------------------------------------------ + +## 13. Startup Sequence in Detail + +### Phase 1 --- Entrypoint + +The binary entrypoint parses CLI arguments. The `serve` command enters +the long-running server path. + +### Phase 2 --- Configuration + +Configuration is discovered and loaded before runtime dependencies are +assembled. + +A configuration failure prevents startup. + +### Phase 3 --- Application construction + +`Application::builder(config).build().await` creates the runtime +container and initializes dependencies. + +### Phase 4 --- Database initialization + +The configured SQLite or PostgreSQL backend is initialized through the +database provider abstraction. + +Pending migrations are applied before the service begins serving +requests. + +### Phase 5 --- Application state + +Database and repository dependencies are incorporated into `AppState`. + +### Phase 6 --- Router construction + +The Axum API and SPA routes are assembled using the initialized +application state. + +### Phase 7 --- Listener binding + +The runtime binds a `TcpListener` to the configured host and port. + +Listener binding is a startup boundary: a bind failure must abort +startup rather than incorrectly reporting `Running`. + +### Phase 8 --- Serving + +After successful binding, the runtime transitions to `Running` and +executes `axum::serve` with graceful shutdown integration. + +------------------------------------------------------------------------ + +## 14. Startup Failure Boundaries + +``` mermaid +flowchart TD + START["Startup"] --> CFG{"Configuration valid"} + CFG -- No --> FAIL["Return startup error"] + CFG -- Yes --> DB{"Database initialized"} + DB -- No --> FAIL + DB -- Yes --> MIG{"Migrations successful"} + MIG -- No --> FAIL + MIG -- Yes --> ROUTER{"Router constructed"} + ROUTER -- No --> FAIL + ROUTER -- Yes --> BIND{"Listener bound"} + BIND -- No --> FAIL + BIND -- Yes --> RUN["Running"] +``` + +A failed prerequisite must not leave the application falsely marked as +operational. + +------------------------------------------------------------------------ + +## 15. HTTP Server Lifecycle + +The HTTP server is created only after dependency assembly. + +``` mermaid +flowchart LR + LISTENER["TcpListener"] --> AXUM["axum serve"] + ROUTER["Router"] --> AXUM + SHUTDOWN["Graceful shutdown future"] --> AXUM + AXUM --> REQUESTS["HTTP requests"] +``` + +The graceful shutdown future connects external termination events to +Axum's server-draining behavior. + +------------------------------------------------------------------------ + +## 16. Shutdown Architecture + +Shutdown is not a single cancellation call. It is a sequence of +lifecycle phases. + +``` mermaid +flowchart TD + SIGNAL["Shutdown trigger"] --> GUARD["Initiate shutdown once"] + GUARD --> CANCEL["Cancel root token"] + CANCEL --> DRAIN["Drain HTTP server"] + DRAIN --> STOP["Stop workers"] + STOP --> HOOKS["Execute shutdown hooks"] + HOOKS --> CLOSE["Close resources"] + CLOSE --> DONE["Stopped"] +``` + +------------------------------------------------------------------------ + +## 17. Shutdown Triggers + +Graceful shutdown may originate from: + +- `SIGINT` +- `SIGTERM` +- programmatic cancellation through the shutdown coordinator +- another controlled runtime termination path + +The first shutdown event initiates graceful termination. + +A subsequent signal can be treated as escalation when graceful shutdown +is no longer appropriate. + +------------------------------------------------------------------------ + +## 18. Signal Management + +`SignalManager` is responsible for translating operating-system +termination signals into runtime shutdown events. + +``` mermaid +sequenceDiagram + participant OS + participant Signals + participant Runtime + participant Server + + OS->>Signals: first termination signal + Signals->>Runtime: initiate graceful shutdown + Runtime->>Server: begin draining + OS->>Signals: second termination signal + Signals->>Runtime: escalate termination +``` + +The distinction between first-signal graceful termination and subsequent +escalation prevents a stuck cleanup operation from indefinitely blocking +process termination. + +------------------------------------------------------------------------ + +## 19. Cancellation Architecture + +The shutdown coordinator uses `tokio_util::sync::CancellationToken`. + +``` mermaid +flowchart TD + ROOT["Root cancellation token"] + ROOT --> HTTP["HTTP shutdown"] + ROOT --> WG1["Worker group A"] + ROOT --> WG2["Worker group B"] + ROOT --> AUX["Auxiliary subsystem"] +``` + +Child cancellation tokens provide hierarchical propagation while +allowing subsystems to respond independently. + +Cancellation is a request to stop. It does not by itself prove that a +task has terminated; worker joining remains necessary. + +------------------------------------------------------------------------ + +## 20. Worker Management + +`WorkerManager` groups asynchronous tasks and tracks their lifecycle. + +The implementation uses `tokio::task::JoinSet` for task coordination. + +Responsibilities include: + +- registering background work, +- grouping related tasks, +- propagating cancellation, +- waiting for worker completion, +- enforcing bounded shutdown, +- aborting tasks that exceed the permitted shutdown window. + +------------------------------------------------------------------------ + +## 21. Worker Shutdown + +``` mermaid +flowchart TD + A["StoppingWorkers"] --> B["Signal cancellation"] + B --> C["Wait for worker completion"] + C --> D{"Completed before timeout"} + D -- Yes --> E["Worker group stopped"] + D -- No --> F["Abort remaining tasks"] + F --> E + E --> G["ExecutingHooks"] +``` + +Bounded shutdown is essential. Graceful termination must not become an +unlimited wait on a non-responsive task. + +------------------------------------------------------------------------ + +## 22. Worker Hierarchy + +``` mermaid +flowchart TD + WM["WorkerManager"] + WM --> G1["TaskGroup A"] + WM --> G2["TaskGroup B"] + G1 --> T1["Task 1"] + G1 --> T2["Task 2"] + G2 --> T3["Task 3"] +``` + +Task grouping allows cancellation and joining to be reasoned about at +subsystem boundaries rather than as unrelated spawned tasks. + +------------------------------------------------------------------------ + +## 23. Shutdown Hooks + +The runtime supports asynchronous shutdown hooks through the +`ShutdownHook` abstraction. + +``` rust +#[async_trait::async_trait] +pub trait ShutdownHook: Send + Sync { + fn name(&self) -> &'static str; + + fn priority(&self) -> ShutdownPriority { + ShutdownPriority::Normal + } + + async fn shutdown(&self) -> Result<()>; +} +``` + +Hooks are intended for cleanup operations that must participate in +lifecycle ordering. + +------------------------------------------------------------------------ + +## 24. Hook Priorities + +The defined order is: + +1. `First` +2. `Normal` +3. `Last` + +``` mermaid +flowchart LR + FIRST["First"] --> NORMAL["Normal"] + NORMAL --> LAST["Last"] +``` + +Priority is semantic. A cleanup dependency should be represented by the +appropriate priority rather than relying on accidental registration +order. + +------------------------------------------------------------------------ + +## 25. Hook Execution Model + +``` mermaid +sequenceDiagram + participant Runtime + participant Registry + participant First + participant Normal + participant Last + + Runtime->>Registry: execute hooks + Registry->>First: shutdown + First-->>Registry: complete + Registry->>Normal: shutdown + Normal-->>Registry: complete + Registry->>Last: shutdown + Last-->>Registry: complete + Registry-->>Runtime: hook phase complete +``` + +A robust shutdown implementation should preserve cleanup progression +even when an individual hook reports an error, while ensuring failures +remain observable. + +------------------------------------------------------------------------ + +## 26. Resource Closure + +The `ClosingResources` phase follows worker shutdown and hook execution. + +Resources may include: + +- database connection pools, +- repository/provider state, +- runtime-owned communication primitives, +- logging or telemetry buffers, +- other long-lived infrastructure owned by the application. + +The ordering prevents dependent background work from using resources +after they have been destroyed. + +------------------------------------------------------------------------ + +## 27. Complete Shutdown Sequence + +``` mermaid +sequenceDiagram + participant Signal + participant State + participant Server + participant Workers + participant Hooks + participant Resources + + Signal->>State: initiate shutdown + State->>State: Running to Draining + State->>Server: graceful shutdown + Server-->>State: HTTP drained + State->>State: Draining to StoppingWorkers + State->>Workers: cancel and join + Workers-->>State: workers stopped + State->>State: StoppingWorkers to ExecutingHooks + State->>Hooks: execute by priority + Hooks-->>State: hooks complete + State->>State: ExecutingHooks to ClosingResources + State->>Resources: close + Resources-->>State: released + State->>State: ClosingResources to Stopped +``` + +------------------------------------------------------------------------ + +## 28. Shutdown Failure Strategy + +Shutdown differs from startup failure handling. + +During startup, a failed mandatory prerequisite normally aborts startup. + +During shutdown, a cleanup failure should generally be recorded while +allowing later cleanup phases to proceed where safe. Otherwise, one +failed hook could prevent unrelated resources from being released. + +Important failure categories include: + +- worker timeout, +- worker panic, +- shutdown hook error, +- resource-close error, +- signal escalation, +- cancellation races. + +The exact error policy should remain explicit in implementation and +tests. + +------------------------------------------------------------------------ + +## 29. Concurrency Model + +The lifecycle subsystem is built on Tokio asynchronous execution. + +Primary concurrency mechanisms include: + +- Tokio tasks, +- `JoinSet`, +- `CancellationToken`, +- atomic lifecycle state, +- shared application state, +- asynchronous shutdown hooks. + +The lifecycle state machine itself does not depend on a blocking mutex. + +------------------------------------------------------------------------ + +## 30. Thread-Safety Model + +Runtime coordination should avoid mutable global state. + +Cross-task lifecycle communication is expressed through primitives +designed for concurrent access: + +``` mermaid +flowchart LR + TASK1["Task A"] --> ATOMIC["Atomic state"] + TASK2["Task B"] --> ATOMIC + SIGNAL["Signal task"] --> TOKEN["Cancellation token"] + TOKEN --> TASK1 + TOKEN --> TASK2 +``` + +Atomic state answers the question "which lifecycle phase are we in?" + +Cancellation answers the question "should this work stop?" + +Task joining answers the question "has this work actually stopped?" + +These are separate concerns and should remain separate. + +------------------------------------------------------------------------ + +## 31. Memory Ordering + +The state implementation uses atomic memory ordering such as `Acquire` +and `AcqRel`. + +At a conceptual level: + +- **Acquire** prevents subsequent operations from being reordered + before an observed synchronization point. +- **Release** publishes prior operations before a synchronization + point. +- **AcqRel** combines both properties for read-modify-write + operations. + +The runtime should use the weakest ordering that still preserves its +required synchronization semantics, but lifecycle correctness takes +priority over micro-optimization. + +Changes to atomic ordering require careful review because a state +transition is part of inter-task coordination rather than merely a +numeric assignment. + +------------------------------------------------------------------------ + +## 32. Why `AtomicU8` + +The lifecycle has a small finite state space, making an integer-backed +atomic representation appropriate. + +Benefits include: + +- compact storage, +- constant-time reads, +- compare-and-exchange transitions, +- no lifecycle mutex contention, +- explicit conversion between stored values and semantic states. + +The numeric representation is an implementation detail. Callers should +reason in terms of `RuntimeState`, not raw integers. + +------------------------------------------------------------------------ + +## 33. Observability + +Runtime transitions should be visible in structured logs. + +A typical startup trace includes: + +``` text +Initializing +Starting +Running +Listening on 0.0.0.0:8655 +``` + +Structured transition logging should identify both the previous and next +state. + +This allows operators to distinguish: + +- configuration failure, +- database startup failure, +- listener failure, +- successful service readiness, +- graceful shutdown, +- worker shutdown delays, +- final termination. + +------------------------------------------------------------------------ + +## 34. Runtime Metrics + +`RuntimeMetrics` provides a lifecycle-level location for operational +measurements such as: + +- request counts, +- error counts, +- active worker counts. + +Metrics should describe runtime behavior without becoming an alternate +source of lifecycle truth. `AtomicRuntimeState` remains authoritative +for lifecycle phase. + +------------------------------------------------------------------------ + +## 35. Lifecycle and Request Processing + +``` mermaid +flowchart TD + RUN["Running"] --> LISTENER["TCP listener"] + LISTENER --> ROUTER["Axum router"] + ROUTER --> MW["Middleware"] + MW --> HANDLER["API or UI handler"] + HANDLER --> RESPONSE["HTTP response"] + SIGNAL["Shutdown signal"] --> DRAIN["Draining"] + DRAIN --> LISTENER +``` + +During normal operation, requests flow through the listener and router. + +When graceful shutdown begins, the server stops accepting new work +according to Axum's graceful shutdown behavior and allows in-flight +requests to complete before later teardown phases proceed. + +------------------------------------------------------------------------ + +## 36. Database Lifecycle + +Database initialization is part of startup, not lazy runtime discovery. + +``` mermaid +flowchart LR + CONFIG["Database configuration"] --> INIT["Initialize provider"] + INIT --> MIG["Run migrations"] + MIG --> POOL["Pool ready"] + POOL --> STATE["AppState"] + STATE --> RUN["Running"] +``` + +On shutdown, database resources are retained until worker and hook +phases that may depend on them have completed. + +------------------------------------------------------------------------ + +## 37. Router Lifecycle + +The router is assembled after its required state is available. + +This provides a clean dependency direction: + +``` text +configuration + -> database/provider + -> AppState + -> router + -> listener/server +``` + +The HTTP layer therefore consumes initialized domain infrastructure +instead of being responsible for constructing it. + +------------------------------------------------------------------------ + +## 38. ApplicationBuilder Responsibilities + +`ApplicationBuilder` should remain focused on dependency assembly. + +Its responsibilities include: + +- receiving validated runtime configuration, +- initializing database infrastructure, +- assembling provider/repository dependencies, +- constructing `AppState`, +- constructing the Axum router, +- returning an initialized `Application`. + +It should not become a second application runtime. + +------------------------------------------------------------------------ + +## 39. Application Responsibilities + +`Application` owns the execution lifecycle after construction. + +Its responsibilities include: + +- state transitions, +- listener binding, +- HTTP serving, +- graceful shutdown integration, +- worker coordination, +- shutdown-hook execution, +- resource closure, +- lifecycle metrics and observability. + +This division keeps construction and execution conceptually distinct. + +------------------------------------------------------------------------ + +## 40. Lifecycle Trait + +The lifecycle abstraction exposes the major phases through operations +such as: + +``` text +initialize +start +shutdown +``` + +The abstraction makes lifecycle behavior testable independently from the +CLI command parser. + +The CLI should initiate lifecycle operations rather than duplicate their +implementation. + +------------------------------------------------------------------------ + +## 41. Security Considerations + +Lifecycle correctness contributes directly to security. + +### 41.1 Controlled startup + +The service must not report itself as operational before mandatory +security-sensitive dependencies are ready. + +### 41.2 Controlled shutdown + +Sessions, audit operations, database work, and background tasks should +not be abandoned arbitrarily during ordinary termination. + +### 41.3 No duplicate teardown + +Duplicate shutdown sequences can cause double-close behavior, +inconsistent logging, or races between cleanup operations. + +### 41.4 Bounded termination + +An attacker or malformed task should not be able to prevent process +termination indefinitely by blocking graceful worker shutdown. + +### 41.5 Signal escalation + +Forced termination provides an operational escape path when graceful +shutdown cannot complete. + +------------------------------------------------------------------------ + +## 42. Performance Characteristics + +The lifecycle subsystem is not expected to dominate request-path +performance. + +Approximate structural complexity: + + Operation Characteristic + ----------------------------- ---------------------------------------- + Runtime state read O(1) + Atomic transition O(1) + Cancellation propagation proportional to cancellation tree + Worker shutdown O(n) relative to tracked tasks + Hook execution O(n) relative to registered hooks + Startup dependency assembly proportional to initialized components + +Database initialization and migration cost dominate lifecycle startup +more than the state machine itself. + +------------------------------------------------------------------------ + +## 43. Graceful Shutdown Timing + +Shutdown timing should be treated as bounded operational policy. + +``` mermaid +flowchart LR + T0["Shutdown requested"] --> T1["HTTP drain"] + T1 --> T2["Worker timeout window"] + T2 --> T3["Hooks"] + T3 --> T4["Resource close"] + T4 --> T5["Exit"] +``` + +Timeout values should be configuration-driven where operational +requirements justify it. + +A timeout is not an error by itself; it is a boundary after which the +runtime may need to escalate from graceful waiting to forced task +abortion. + +------------------------------------------------------------------------ + +## 44. Extension Model + +New runtime subsystems should integrate through existing lifecycle +primitives rather than creating independent shutdown mechanisms. + +A new subsystem should determine: + +1. Who owns it? +2. When is it initialized? +3. Does it spawn background work? +4. Which cancellation token does it observe? +5. Must its tasks be joined? +6. Does it require a shutdown hook? +7. Which hook priority is correct? +8. Which resources must remain alive until it stops? +9. Which metrics and logs expose its lifecycle? + +------------------------------------------------------------------------ + +## 45. Example Custom Shutdown Hook + +``` rust +use nx9_auth::runtime::{ShutdownHook, ShutdownPriority}; + +struct CustomCleanupHook; + +#[async_trait::async_trait] +impl ShutdownHook for CustomCleanupHook { + fn name(&self) -> &'static str { + "custom_cleanup" + } + + fn priority(&self) -> ShutdownPriority { + ShutdownPriority::First + } + + async fn shutdown(&self) -> anyhow::Result<()> { + println!("Flushing custom buffer before shutdown..."); + Ok(()) + } +} +``` + +Registration belongs in application assembly where lifecycle ownership +is visible. + +------------------------------------------------------------------------ + +## 46. Example Runtime Bootstrap + +``` rust +use nx9_auth::config::Config; +use nx9_auth::runtime::{Application, Lifecycle}; + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + let config = Config::find_and_load()?; + let mut app = Application::builder(config).build().await?; + app.start().await?; + Ok(()) +} +``` + +The actual binary may perform additional CLI handling, logging +initialization, or command dispatch around this lifecycle. + +------------------------------------------------------------------------ + +## 47. Testing Strategy + +The lifecycle subsystem is verified by +`tests/runtime_lifecycle_test.rs`. + +The documented test areas include: + +- application builder lifecycle, +- state transitions, +- shutdown hook execution order, +- worker manager lifecycle. + +The broader NX9-Auth test suite additionally exercises application +behavior that depends on successful runtime assembly. + +------------------------------------------------------------------------ + +## 48. Lifecycle Test Model + +``` mermaid +flowchart TD + TESTS["Runtime lifecycle tests"] + TESTS --> BUILDER["Builder initialization"] + TESTS --> STATES["State transitions"] + TESTS --> HOOKS["Hook ordering"] + TESTS --> WORKERS["Worker lifecycle"] +``` + +The tests should verify behavior rather than merely implementation +structure. + +For example, hook tests should verify observable execution order instead +of assuming internal container ordering. + +------------------------------------------------------------------------ + +## 49. Recommended Additional Tests + +The following are valuable future lifecycle tests if not already +present: + +- invalid state transition rejection, +- duplicate shutdown initiation, +- worker timeout followed by abort, +- shutdown-hook failure while later hooks still execute, +- listener bind failure, +- migration failure during initialization, +- second-signal escalation, +- cancellation propagation to child task groups, +- shutdown with zero workers, +- shutdown with zero hooks, +- repeated state reads during concurrent shutdown. + +These are recommendations, not claims about the current v0.3.0 test +suite. + +------------------------------------------------------------------------ + +## 50. Operational Diagnostics + +When investigating startup problems, check the last lifecycle state +reached. + +### Stops during `Initializing` + +Investigate configuration discovery and initial runtime construction. + +### Stops during `Starting` + +Investigate: + +- database connection, +- migrations, +- application state assembly, +- router construction, +- listener binding. + +### Reaches `Running` but does not serve + +Investigate: + +- bound address, +- listener configuration, +- router behavior, +- reverse proxy configuration, +- firewall/network path. + +### Hangs during `StoppingWorkers` + +Investigate worker cancellation handling and timeout behavior. + +### Hangs during `ExecutingHooks` + +Investigate shutdown hooks and external dependencies used by cleanup +operations. + +### Hangs during `ClosingResources` + +Investigate database and other long-lived resource termination. + +------------------------------------------------------------------------ + +## 51. Logging Recommendations + +Lifecycle logs should be: + +- structured, +- concise, +- emitted at state boundaries, +- free of credentials and secrets, +- sufficient to reconstruct startup and shutdown progression. + +Recommended fields include: + +``` text +event +from_state +to_state +component +duration +error +``` + +Sensitive configuration values must never be included merely to improve +lifecycle diagnostics. + +------------------------------------------------------------------------ + +## 52. Anti-Patterns + +### 52.1 Detached unmanaged tasks + +Avoid spawning long-running tasks that are invisible to `WorkerManager`. + +### 52.2 Independent shutdown flags + +Avoid subsystem-specific boolean shutdown flags when cancellation tokens +already express the lifecycle signal. + +### 52.3 Closing resources before workers + +Workers may still require those resources. + +### 52.4 Blocking indefinitely + +Graceful shutdown requires bounded waits. + +### 52.5 State regression + +Do not move from a later lifecycle state back to an earlier state to +represent retries. + +### 52.6 Using `force_set()` as normal control flow + +Forced state mutation is an exceptional mechanism. + +### 52.7 Hiding startup failure + +Do not transition to `Running` if listener binding or mandatory +initialization failed. + +------------------------------------------------------------------------ + +## 53. Review Checklist for Runtime Changes + +Before merging changes to `src/runtime/`, verify: + +- [ ] lifecycle state transitions remain valid, +- [ ] no state regression was introduced, +- [ ] shutdown remains single-initiation, +- [ ] new workers observe cancellation, +- [ ] workers are tracked and joined, +- [ ] shutdown waits are bounded, +- [ ] resources outlive dependent workers, +- [ ] hook ordering remains deterministic, +- [ ] signal handling still supports graceful termination, +- [ ] new logs contain no sensitive information, +- [ ] lifecycle tests cover new behavior, +- [ ] `cargo fmt` passes, +- [ ] `cargo clippy` passes with project warning policy, +- [ ] workspace tests pass. + +------------------------------------------------------------------------ + +## 54. Source Layout + +``` text +src/runtime/ +├── application.rs +├── builder.rs +├── cancellation.rs +├── hooks.rs +├── lifecycle.rs +├── metrics.rs +├── mod.rs +├── signals.rs +├── state.rs +└── workers.rs +``` + +The exact module set should remain aligned with the repository as the +runtime evolves. + +------------------------------------------------------------------------ + +## 55. Related Source Areas + +The lifecycle subsystem interacts with: + +``` text +src/main.rs +src/config/ +src/db/ +src/api/ +src/middleware/ +tests/runtime_lifecycle_test.rs +``` + +The runtime coordinates these areas but should not absorb their domain +responsibilities. + +------------------------------------------------------------------------ + +## 56. Architecture Summary + +``` mermaid +flowchart TD + ENTRY["CLI"] --> BUILD["Build application"] + BUILD --> READY["Dependencies ready"] + READY --> RUN["Running"] + RUN --> SIGNAL["Shutdown trigger"] + SIGNAL --> DRAIN["Draining"] + DRAIN --> WORKERS["Stopping workers"] + WORKERS --> HOOKS["Executing hooks"] + HOOKS --> RESOURCES["Closing resources"] + RESOURCES --> STOP["Stopped"] +``` + +NX9-Auth's lifecycle architecture is centered on a simple rule: +**construction, operation, and destruction are explicit phases with +explicit ownership**. + +The state machine makes those phases observable. Cancellation tokens +propagate intent. Worker management proves task termination. Hook +priorities provide deterministic cleanup ordering. Resource closure +occurs only after dependent work has been stopped. + +------------------------------------------------------------------------ + +## 57. Verification Commands + +Runtime lifecycle tests: + +``` bash +cargo test --test runtime_lifecycle_test +``` + +Complete project test suite: + +``` bash +cargo test --all-features +``` + +Formatting: + +``` bash +cargo fmt --all -- --check +``` + +Clippy: + +``` bash +cargo clippy --workspace --all-targets --all-features -- -D warnings +``` + +Release build: + +``` bash +cargo build --release +``` + +For the WebAssembly administration UI, use the project's dedicated WASM +build process rather than assuming a native host build is equivalent. + +------------------------------------------------------------------------ + +## 58. Document Maintenance + +Update this document whenever a change modifies: + +- lifecycle states, +- state transition rules, +- startup ordering, +- shutdown ordering, +- signal semantics, +- worker management, +- cancellation hierarchy, +- hook priority semantics, +- resource ownership, +- runtime metrics, +- lifecycle test guarantees. + +Architecture documentation should describe implemented behavior. +Proposed behavior should be explicitly identified as a recommendation or +future design rather than presented as current implementation. + +------------------------------------------------------------------------ + +## 59. Status + +For the v0.3.0 architecture baseline, the runtime lifecycle consists of +explicit application construction, atomic lifecycle state tracking, +signal-driven graceful shutdown, hierarchical cancellation, managed +background workers, prioritized shutdown hooks, runtime metrics, and +final resource cleanup. + +The lifecycle progression is: + +``` text +Initializing + -> Starting + -> Running + -> Draining + -> StoppingWorkers + -> ExecutingHooks + -> ClosingResources + -> Stopped +``` + +This sequence forms the operational backbone of the NX9-Auth server +runtime.