Deployment & Operations

Deployment & Operations

Operational reference for the Rust streaming backend (streaming-backend) — covering process startup, environment-based configuration, database migrations, Redis initialization, dual-protocol serving (Axum HTTP + gRPC-Web), background maintenance jobs, and the failure modes operators must plan for.

Purpose and Scope

This page documents how the Rust backend is configured, started, and operated in a deployment environment. It covers the full boot sequence in streaming-backend/src/main.rs, the environment-variable contract in streaming-backend/src/config.rs, the runtime topology (PostgreSQL, Redis, embedded SFU, background tasks), and operational concerns such as fail-fast validation, secrets handling, and periodic cleanup.

Intentionally left to sibling pages in the rust-backend catalog: the HTTP router/middleware internals, the gRPC service definitions and handlers, the data model and persistence layer, and the per-service business logic (auth, chat, signals, media). Those pages describe what the servers do; this page describes how they are brought up and kept running.

Overview

The backend is a single Rust binary that, at startup, connects to PostgreSQL and Redis, runs database migrations, constructs the shared application state, and then serves traffic on two protocols simultaneously:

• Axum HTTP server on HOST:PORT (default 0.0.0.0:8080) — REST/WebSocket-style traffic through router::build_router.
• gRPC-Web server on HOST:GRPC_PORT (default 0.0.0.0:50051) — started via grpc::server::serve.

Everything is configured through environment variables (with a .env file loaded first as a convenience for local development). The process is designed to fail fast: required settings such as DATABASE_URL panic at startup if missing, and any connectivity failure to PostgreSQL or Redis aborts the boot before traffic is accepted.

Beyond serving, the process owns several operational responsibilities that run in background Tokio tasks:

• Identity key backfill — runs once at startup to derive deterministic identity keys for users missing a key bundle.
• Status purger — every 600 seconds, expires stale status entries via StatusService::purge_expired_statuses.
• SFU signal relay — forwards embedded SFU (str0m) signals to the call relay over an unbounded channel.

The key design intent: a single process owns the whole signaling/streaming lifecycle, so deployment is one binary plus two external dependencies (PostgreSQL, Redis). There is no external SFU or sidecar process to orchestrate; the media router is embedded.

Architecture

The following diagram shows the runtime topology and the startup dependency graph as implemented in main.rs and config.rs:

Component roles

The design deliberately shares one AppStateData across both serving stacks so gRPC handlers and HTTP handlers see the same in-memory services (Snowflake ID generator, refresh tokens, call tickets, relay, SFU, storage) — there is no duplicated state and no cross-process synchronization needed.

Core Flow — Process Startup Sequence

The boot sequence is strictly ordered so that the process never accepts traffic before its dependencies are ready. main() awaits each initialization step and propagates errors with ?, so any failure aborts startup and the process exits non-zero (fail-fast).

Step-by-step walkthrough

1. Load .env (best-effort) — dotenv().ok() loads a local .env file but ignores the error when it is absent. This keeps local development ergonomic while production uses real environment variables injected by the platform (main.rs).

2. Initialize logging — tracing_subscriber::fmt::init() installs the tracing formatter; all subsequent info! output (e.g., "Starting Secure Mesh Backend", "HTTP → 0.0.0.0:8080") goes to stdout via this subscriber (main.rs).

3. Read configuration — Config::from_env() resolves every setting. Missing DATABASE_URL or non-numeric PORT/GRPC_PORT/SNOWFLAKE_MACHINE_ID panics immediately — the operator is told what is wrong before anything else happens (config.rs).

4. PostgreSQL pool + migrations — db::create_pool(&cfg.database_url) opens the pool, then db::run_migrations_safely(&db_pool) applies schema migrations. Migrations run on every boot, so a deploy and a migration are the same operation — no separate migration step is required (main.rs).

5. Redis connection — the fred client is built from REDIS_URL and explicitly initialized with redis_client.init().await?. A Redis outage therefore blocks startup rather than causing mid-flight failures (main.rs).

6. Snowflake ID service — SnowflakeService::verify_and_init(&cfg.database_url, cfg.snowflake_machine_id) verifies/initializes the machine ID against the database. The comment in source warns the service must be instantiated exactly once — the single Arc is shared with RefreshTokenService (main.rs).

7. Service construction — call tickets and relay are built against Redis; the embedded SFU (str0m) is created with an unbounded MPSC channel whose receiver forwards every signal to the call relay in a dedicated task (main.rs).

8. Shared state — all services are packed into Arc<AppStateData> together with jwt_secret and start_time (used for uptime/health reporting) (main.rs).

9. Background tasks — identity key backfill is spawned once; the status purger loops every 600 seconds calling StatusService::purge_expired_statuses (main.rs).

10. Start servers — the gRPC-Web server is spawned as a Tokio task on GRPC_PORT, then main() binds the Axum listener on PORT and blocks in axum::serve(...), which keeps the process alive until the HTTP server terminates (main.rs).

Environment-Driven Configuration

Config is a plain Clone + Debug struct; there is no config file, no CLI parser, and no runtime reload. Every field maps 1 to an environment variable, which makes container orchestration straightforward: the image is immutable, and everything environment-specific is injected at deploy time.

#[derive(Clone, Debug)]
pub struct Config {
    pub host: String,
    pub port: u16,
    pub grpc_port: u16,
    pub database_url: String,
    pub redis_url: String,
    pub snowflake_machine_id: i32,
    pub jwt_secret: String,
    #[allow(dead_code)]  // used to select storage backend (local vs s3)
    pub storage_backend: String,
    pub local_upload_dir: String,
}

Source: config.rs

Two distinct defaulting strategies are used, and the difference matters operationally:

• Soft defaults (unwrap_or_else) — HOST, PORT, GRPC_PORT, REDIS_URL, SNOWFLAKE_MACHINE_ID, JWT_SECRET, STORAGE_BACKEND, LOCAL_UPLOAD_DIR fall back to values tuned for local development.
• Hard requirement (expect) — DATABASE_URL has no default; the process refuses to start without it. This is deliberate: every other service depends on the database, so there is no point booting into a half-configured state.

The default JWT_SECRET ("super-secret-default-key-change-in-prod") is explicitly named as a development placeholder — an operator who deploys without overriding it ships a well-known signing key. The STORAGE_BACKEND field carries #[allow(dead_code)] with a comment indicating it selects between local and S3 storage, signaling an intended extension point that is not yet wired into the boot path.

Usage Examples

Example 1 — Full boot sequence (entry point)

This is the entire startup path an operator observes: configuration resolution, dependency connection, migration, service construction, and server launch.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Load .env file first — ignores error if file is absent (production uses real env vars)
    dotenv().ok();

    tracing_subscriber::fmt::init();

    let cfg = Config::from_env();
    info!("Starting Secure Mesh Backend");

    let db_pool = db::create_pool(&cfg.database_url).await?;
    db::run_migrations_safely(&db_pool).await?;

    // Build fred RedisClient and connect
    let fred_config = FredConfig::from_url(&cfg.redis_url)?;
    let redis_client = Builder::from_config(fred_config).build()?;
    redis_client.init().await?;

    let snowflake = Arc::new(SnowflakeService::verify_and_init(&cfg.database_url, cfg.snowflake_machine_id).await?);

Source: main.rs

Example 2 — Dual-protocol serving

Note the asymmetry: gRPC is spawned as a fire-and-forget Tokio task, while the Axum HTTP server is bound first and then awaited on the main task, keeping the process alive.

    // Spawn gRPC (gRPC-Web) server
    let grpc_addr = format!("{}:{}", cfg.host, cfg.grpc_port).parse()?;
    tokio::spawn(crate::grpc::server::serve(state.clone(), grpc_addr));

    let cleanup_state = state.clone();
    tokio::spawn(async move {
        let mut interval = tokio::time::interval(std::time::Duration::from_secs(600));
        loop {
            interval.tick().await;
            let _ = crate::modules::status::service::StatusService::purge_expired_statuses(&cleanup_state).await;
        }
    });

    // Spawn Axum HTTP service on PORT
    let app = router::build_router(state);
    let http_addr = format!("{}:{}", cfg.host, cfg.port);
    let listener = TcpListener::bind(&http_addr).await?;
    info!("HTTP  → {}", http_addr);
    axum::serve(listener, app).await?;

    Ok(())
}

Source: main.rs

Example 3 — Configuration resolution patterns

This excerpt shows the two defaulting strategies and the validation errors operators will see:

    pub fn from_env() -> Self {
        Self {
            host: env::var("HOST").unwrap_or_else(|_| "0.0.0.0".into()),
            port: env::var("PORT")
                .unwrap_or_else(|_| "8080".into())
                .parse()
                .expect("PORT must be a number"),
            grpc_port: env::var("GRPC_PORT")
                .unwrap_or_else(|_| "50051".into())
                .parse()
                .expect("GRPC_PORT must be a number"),
            database_url: env::var("DATABASE_URL")
                .expect("DATABASE_URL must be set"),
            redis_url: env::var("REDIS_URL")
                .unwrap_or_else(|_| "redis://127.0.0.1:6379".into()),
            snowflake_machine_id: env::var("SNOWFLAKE_MACHINE_ID")
                .unwrap_or_else(|_| "0".into())
                .parse()
                .expect("SNOWFLAKE_MACHINE_ID must be a number"),
            jwt_secret: env::var("JWT_SECRET")
                .unwrap_or_else(|_| "super-secret-default-key-change-in-prod".into()),
            storage_backend: env::var("STORAGE_BACKEND")
                .unwrap_or_else(|_| "local".into()),
            local_upload_dir: env::var("LOCAL_UPLOAD_DIR")
                .unwrap_or_else(|_| "./uploads".into()),
        }
    }

Source: config.rs

Configuration Options

Configuration semantics

• Validation is fail-fast and synchronous. Three fields (PORT, GRPC_PORT, SNOWFLAKE_MACHINE_ID) use .parse().expect(...) with descriptive panic messages, and DATABASE_URL uses .expect(...) directly. Operators get immediate, actionable errors during image startup instead of runtime misbehavior.
• .env is a convenience, not a deployment contract. dotenv().ok() swallows the missing-file error, so production deployments can rely purely on injected environment variables.
• Secrets travel in env vars. JWT_SECRET is read from the environment and stored in AppStateData for the request pipeline; no secret material is written to disk by the application itself.

Failure Modes, Edge Cases & Concurrency

Startup failure modes (fail-fast)

The consistent pattern: every dependency error is a startup error. There are no retry loops in main() for database or Redis — the orchestrator (container platform / supervisor) is expected to restart the process. Retry-on-restart keeps the boot path simple and deterministic.

Concurrency model

• Single shared state. AppStateData is wrapped in Arc and cloned into the gRPC task, the HTTP router, and the background tasks. All in-process handlers observe the same service instances — no per-request copies, no split-brain between protocols (main.rs).
• Mutex-protected services. FileDetectService is constructed as Arc<Mutex<FileDetectService>>, indicating interior mutability that must be serialized across handlers (main.rs).
• Unbounded channel for SFU signals. The SFU-to-relay path uses tokio::sync::mpsc::unbounded_channel (main.rs). Unbounded means the producer never blocks, but under a signal burst the queue can grow without backpressure — a consideration for capacity planning under heavy call volume.
• Single-instance Snowflake. The comment in source explicitly warns verify_and_init must never be called twice; the service is constructed once and shared (main.rs). Operators must not run two replicas with the same SNOWFLAKE_MACHINE_ID against the same database without understanding the ID-uniqueness contract.
• Background task isolation. The status purger runs in its own task with a fixed 600-second interval and discards errors (let _ = ...), so a transient purge failure never crashes the process (main.rs).

Edge cases

• Missing .env is fine — dotenv().ok() ignores the error, so containers without a .env file boot normally (main.rs).
• Default JWT_SECRET is a live risk — the shipped default "super-secret-default-key-change-in-prod" is a known value; any production instance that forgets to override it has a publicly known signing key.
• start_time uptime tracking — Instant::now() is captured at boot into AppStateData, giving handlers a stable reference point for uptime/health reporting without a separate clock service (main.rs).

Performance & Operational Considerations

• Migrations run on every boot — startup cost scales with schema size; keep migrations idempotent (run_migrations_safely) so concurrent replicas or redeploys do not conflict.
• One binary, two ports, one process — the HTTP server is the lifecycle owner (main() awaits axum::serve); the gRPC server is a spawned task. When the HTTP server terminates, main() returns and the whole process — including the gRPC task — shuts down. There is no separate lifecycle management for the two protocol stacks.
• Logging — tracing_subscriber::fmt::init() writes formatted tracing output to stdout; collect container stdout as the primary log stream (main.rs).
• Periodic maintenance — the status purger runs every 600 seconds; consider aligning observability alerts with this cadence when monitoring table growth.
• Startup ordering matters for probes — because the HTTP listener is bound last (after DB, Redis, migrations, and service construction), a successful bind on PORT implies the entire dependency chain is healthy. A TCP readiness probe on PORT is therefore a meaningful signal; HTTP responses on GRPC_PORT confirm the gRPC stack.
• No graceful-shutdown handler observed — axum::serve is awaited directly without an explicit tokio::signal handler in main(). Termination relies on the runtime/container platform’s default SIGTERM behavior. (Verified only for main(); a shutdown hook may exist deeper in the router or server modules, which are outside this page’s evidence.)

Extension Points

• Storage backend selection — STORAGE_BACKEND is declared with #[allow(dead_code)] and a comment noting it selects local vs S3; this is the intended seam for plugging in a new storage implementation without changing the boot contract (config.rs).
• Module-based feature growth — the crate is organized into modules/ (e.g., modules::status::service) plus grpc/, middleware/, services/, and router/ (main.rs). New operational jobs follow the same pattern: construct in main(), share via AppStateData, spawn as a Tokio task.
• Configuration surface is env-driven by design — adding a new operational knob means adding a field to Config and a line in from_env(); no config-file schema or CLI migration is required.

Related Links

Source files (primary evidence for this page)

• streaming-backend/src/main.rs — process entry point, full boot sequence, dual-protocol serving, background tasks
• streaming-backend/src/config.rs — Config struct and from_env() environment contract
• streaming-backend/src/db.rs — create_pool and run_migrations_safely (called from main())
• streaming-backend/src/grpc/server.rs — gRPC-Web server launched as a Tokio task
• streaming-backend/src/router.rs — Axum router built by router::build_router (file referenced via mod router in main.rs)
• streaming-backend/src/state.rs — AppStateData definition (referenced in main.rs)

Related catalog pages

• HTTP Router & Middleware (rust-backend.http-router-middleware) — request pipeline, auth and rate-limit middleware that consume AppStateData.
• gRPC Services (rust-backend.grpc-services) — service handlers served on GRPC_PORT, including auth, chat, history, and signals.
• Data Models & Persistence (rust-backend.data-models) — entities migrated by run_migrations_safely at boot.
• Realtime Core — the browser-side counterpart (realtime-core/src/) that connects to these servers; its transport and signaling behavior determines the operational load profile.