# Configuration reference (https://sockudo.io/docs/reference/configuration) > Reference major Sockudo configuration sections and the deployment decisions they control. Sockudo's TOML configuration is organized by runtime responsibility: one table per subsystem, plus a handful of top-level keys for the listener and process. This page describes each section, what it controls, and when you would change it. Configuration resolves in layers, and each later layer wins: 1. compiled defaults, which apply to every key you omit 2. the configuration file (`config/config.toml`, `config/config.json`, or the path given with `--config`), with `${VAR}` and `${VAR:-default}` references expanded from the environment 3. supported environment-variable overrides 4. validation, which rejects inconsistent settings at startup Only explicitly mapped settings have an environment variable; nested policy such as app definitions and channel namespaces lives in the file or the app manager. For loading details, JSON equivalence, and guidance on what belongs in files, environment overrides, or secret stores, see [Static configuration](/docs/deployment/static-configuration). The complete [environment variable reference](/docs/reference/environment-variables) lists every runtime variable parsed by the server and push subsystem. Defaults on this page are the compiled defaults used when a key is omitted. The repository's `config/config.toml` is a development profile that overrides several of them, for example `mode = "development"`, `shutdown_grace_period = 2`, and `connection_recovery.enabled = true`. Start production files from the defaults you need, not from the development profile. ## Top-level Process-wide keys that sit outside any table. Change these to set the bind address, connection limits, timeouts, and graceful-shutdown behavior. | Key | Type | Default | Description | | --- | --- | --- | --- | | `host` | string | `"0.0.0.0"` | Bind host for WebSocket and HTTP API. Use a specific interface to avoid listening on every network. | | `port` | integer | `6001` | Bind port for WebSocket and HTTP API. | | `mode` | string | `"production"` | Runtime mode. `"production"` rejects node-local memory push drivers unless explicitly allowed and warns about risky paths such as a Unix socket under `/tmp`. Set `"development"` for local work. | | `debug` | boolean | `false` | Enables verbose diagnostics for local development, including source locations in logs. | | `max_connections` | integer | `0` | Per-node WebSocket connection limit. When exceeded the server returns close code 4100 so clients reconnect with backoff. `0` disables the limit. Set it to protect each node's memory and to drive `/ready` capacity gating. | | `activity_timeout` | integer (seconds) | `120` | Pusher protocol activity timeout announced to clients in `connection_established`. | | `user_authentication_timeout` | integer (seconds) | `3600` | How long a connection may stay open without completing user authentication when the app requires it. | | `websocket_max_payload_kb` | integer (KiB) | `64` | Legacy maximum inbound WebSocket payload size. | | `health_check_timeout_ms` | integer (ms) | `2000` | Timeout for each subsystem check in `/up` (adapter, cache, queue, app manager). | | `shutdown_grace_period` | integer (seconds) | `10` | Seconds the listener stays open after SIGTERM/SIGINT while `/ready` reports 503. Raise it when load balancers need longer to drain a node. | | `shutdown_close_delay_ms` | integer (ms) | `0` | Milliseconds to wait after the shutdown signal before closing existing WebSockets with 4200, so load balancers see the node draining first. | | `shutdown_close_spread_ms` | integer (ms) | `0` | Milliseconds over which existing WebSockets are closed with 4200, evenly in small batches, so clients do not all reconnect at once. `0` closes all at once. Delay plus spread must not exceed `shutdown_grace_period`. | | `server_role` | string | `"default"` | Deployment topology role. `"default"` runs the full server. `"api"` runs HTTP publish, history, mutable messages, and push — without WebSocket listeners, cluster heartbeats, or presence sync. | | `path_prefix` | string | `"/"` | Parsed for compatibility but not currently applied to any route. Leave it at `"/"` and put a reverse proxy in front for path rewriting. | ## Section index Every other setting lives in a table. Use this index to find the section that owns a behavior. | Group | Sections | Controls | | --- | --- | --- | | Listener and transport | `[ssl]`, `[unix_socket]`, `[cors]`, `[http_api]`, `[http_api.readiness]`, `[http_api.accept_traffic]`, `[websocket]`, `[instance]` | TLS, Unix socket listener, browser CORS, HTTP body limit and operator endpoints, probes and admission, per-connection buffers and heartbeats, and node identity. | | Apps and limits | `[app_manager]`, `[channel_limits]`, `[event_limits]`, `[presence]` | Where apps are stored, plus global channel, event, and presence limits that apps inherit. | | Runtime backends | `[adapter]`, `[cache]`, `[queue]`, `[rate_limiter]`, `[database]`, `[database_pooling]`, `[cleanup]` | Cross-node fanout, shared cache, background work, rate limits, database connections, and disconnect cleanup. | | Protocol features | `[connection_recovery]`, `[history]`, `[presence_history]`, `[versioned_messages]`, `[annotations]`, `[delta_compression]`, `[tag_filtering]`, `[idempotency]`, `[ephemeral]`, `[echo_control]`, `[event_name_filtering]` | Protocol V2 continuity, durable state, and delivery controls. | | Integrations | `[webhooks]`, `[push]`, `[[push_rules]]`, `[ai_transport]`, `[mcp]`, `[ably_compat]` | Outbound webhooks, push notifications, AI Transport, the MCP server, and the Ably facade. | | Observability | `[metrics]`, `[opentelemetry]`, `[logging]` | Prometheus, OTLP export, and log formatting. | ## Server role `server_role` controls which subsystems the process starts. Use `"api"` to scale HTTP publishing and history independently of WebSocket capacity. ```toml server_role = "api" [adapter] driver = "redis-cluster" fallback_to_local = false ``` API pods publish to the horizontal broadcast stream and serve HTTP endpoints (publish, history, mutable messages, annotations, push). They skip WebSocket listeners, adapter subscriptions, heartbeats, presence sync, and request/reply — so they never appear in peer accounting and WS pods never await replies from them. The server rejects startup when `server_role = "api"` is combined with `adapter.driver = "local"` or `adapter.fallback_to_local = true`. For endpoint availability details see [HTTP endpoints — availability by server role](/docs/reference/http-endpoints#availability-by-server-role). For capacity and operational metrics see [Capacity planning — API pod topology](/docs/deployment/capacity-planning#api-pod-topology). ## Capacity readiness Configure this when a load balancer should stop sending new connections to a full node. `GET /ready` runs the same dependency checks as `/up` and also reports `503 CAPACITY` when live native WebSocket connections reach the high watermark of `max_connections`. It stays unavailable until occupancy reaches the low watermark, avoiding repeated load-balancer changes near the limit. Disconnect cleanup backlog is excluded. Existing connections stay open; this endpoint does not rebalance them. ```toml max_connections = 10000 [http_api.readiness] high_watermark = 0.95 low_watermark = 0.85 ``` Watermarks must satisfy `0 <= low_watermark < high_watermark <= 1`. A zero `max_connections` disables capacity gating. Use `/ready` for load-balancer admission, `/up` for dependency health and rollout checks, and `/live` for process liveness. Capacity does not change `/up`, so a full pod can still pass deployment health checks. Standalone Redis adapter health probes reuse a dedicated reconnecting connection, separate from command, event, and blocking queue connections. Probes still execute `PING` and fail on dependency errors or timeouts; they do not open a new connection on every probe. There is no adapter connection-pool size to tune. ## Memory-pressure connection admission Memory-pressure admission control is opt-in under `[http_api.accept_traffic]`. Enable it when nodes run under a container memory limit and you would rather refuse new connections than be OOM-killed: ```toml [http_api.accept_traffic] enabled = true memory_threshold = 0.90 # memory_limit_bytes = 2147483648 # optional; otherwise discover the cgroup limit sample_interval_ms = 500 ``` On Linux, Sockudo samples its process RSS on `sample_interval_ms`, independently of Prometheus scrapes. It compares RSS with `memory_limit_bytes` when configured, otherwise with the finite cgroup v2 `memory.max` or cgroup v1 `memory.limit_in_bytes` value. At or above `memory_threshold`, native and Ably WebSocket upgrades return `503`, and the post-upgrade admission gate returns protocol close code `4100` if pressure changes during the upgrade. Existing connections remain open. The sampler fails open when RSS or a finite limit is unavailable, including on non-Linux systems. `GET /accept-traffic` exposes the admission decision for a load balancer; `/up` deliberately remains a dependency/readiness check and does not fail because of memory pressure. Use `sockudo_memory_pressure_shedding` and `sockudo_memory_pressure_rejections_total` to alert on the current state and rejected connection count. ## App manager The app manager stores app credentials, limits, and policy. The default `memory` driver reads apps from `[[app_manager.array.apps]]`; switch to a database driver when apps are created at runtime or shared by several nodes. | Section | Purpose | | --- | --- | | `[app_manager]` | Select app storage driver: `memory`, `mysql`, `pgsql` (aliases `postgres`, `postgresql`), `dynamodb`, `surrealdb` (alias `surreal`), or `scylladb` (alias `scylla`). `availability_check_interval_seconds` (default `15`, `0` disables) sets how often each node closes live connections of apps that were disabled (close code 4003) or deleted (4001). The `[app_manager.cache]` TTL adds to that delay. | | `[app_manager.array]` | Static in-config app definitions. | | `[[app_manager.array.apps]]` | App ID, key, secret, enabled flag, limits, and policy. | ## Channel namespaces Namespaces apply per-prefix channel rules within an app: naming patterns, client permissions, annotations, and history overrides. Configure namespace rules per app under `policy.channels.channel_namespaces` (`[[app_manager.array.apps.policy.channels.channel_namespaces]]` in TOML). For example, `private-chat:room-1` selects the namespace named `chat`: Sockudo strips the channel type prefix and uses the text before the first `:`. The first entry with that exact name wins. The regex validates that namespace's channel names; it does not select a namespace. A channel containing `:` with no matching namespace is rejected. Channels without `:` do not match a namespace. The three `allow_*_for_client` permission flags apply to **Protocol V2** client operations. **Unset or `null` means allowed**, preserving the existing default. Set `allow_publish_for_client = false` explicitly for server-published namespaces. These flags do not grant access by themselves: app-level client-message settings, private/presence channel authentication, subscription requirements, and connection capabilities still apply. They do not restrict authenticated server HTTP publishes and do not replace Protocol V1's Pusher-compatible authorization rules. | Field | Meaning and unset behavior | | --- | --- | | `name` | Required, non-empty namespace name, without `:`. Exact match after stripping a channel type prefix. | | `channel_name_pattern` | Optional Rust regular expression applied to the logical channel name, including `name:`, without `private-`, `private-encrypted-`, or `presence-` and without a `#user1,user2` suffix. Unset adds no regex restriction; use `^` and `$` for a whole-name match. | | `max_channel_name_length` | Positive limit in bytes for the complete wire channel name, including prefix and user suffix. Unset inherits `policy.limits.max_channel_name_length`, then `200`. | | `allow_user_limited_channels` | Allows V2 `#user1,user2` channel syntax. Unset means **denied**. The authenticated user's identity must also satisfy the channel access check. | | `allow_subscribe_for_client` | V2 client subscription permission. Unset means **allowed**; `false` rejects client subscriptions. | | `allow_publish_for_client` | V2 client publish permission. Unset means **allowed**; `false` rejects client publishes. | | `allow_presence_for_client` | V2 presence subscription permission. Unset means **allowed**; `false` rejects subscriptions to `presence-` channels in this namespace. `allow_subscribe_for_client = false` also denies presence subscriptions. | | `annotations_enabled` | Inherits `policy.channels.annotations_enabled`, then `false`. Enabling annotations requires the global annotation feature and implies durable message history for the namespace. | | `history` | Optional overrides: `rewind_enabled`, `retention_window_seconds`, `max_messages_per_channel`, `max_bytes_per_channel`. Each omitted field inherits app `policy.history`, then global `[history]`. There is no namespace `history.enabled` field. | | `presence_history` | Optional overrides: `enabled`, `retention_window_seconds`, `max_events_per_channel`, `max_bytes_per_channel`. Each omitted field inherits app `policy.presence_history`, then global `[presence_history]`. | An explicit read-only client policy for a server-published namespace: ```toml [[app_manager.array.apps.policy.channels.channel_namespaces]] name = "broadcast" channel_name_pattern = "^broadcast:[A-Za-z0-9._-]+$" max_channel_name_length = 200 allow_user_limited_channels = false allow_subscribe_for_client = true allow_publish_for_client = false allow_presence_for_client = false [app_manager.array.apps.policy.channels.channel_namespaces.history] rewind_enabled = true retention_window_seconds = 3600 max_messages_per_channel = 1000 max_bytes_per_channel = 10485760 ``` The equivalent app JSON nests `channel_namespaces` inside `policy.channels`. JSON does not support comments; the shipped JSON example spells out the permissive client flags. Set its publish flag to `false` wherever clients must only receive server-authored events. The dashboard exposes the same settings in the app's channel namespace editor. ## Ably compatibility keys Only relevant for binaries built with the `ably-compat` Cargo feature. `App.key` and `App.secret` are always the implicit primary credential. When the server is built with `ably-compat`, `[ably_compat]` can opt an app into additional Ably key names without creating sibling apps: ```toml [ably_compat] enabled = true realtime_admission = "accept" attach_timeout_ms = 10000 max_token_ttl_ms = 86400000 token_request_timestamp_skew_ms = 900000 nonce_ttl_seconds = 900 stats_fixture_ingest_enabled = false stats_queue_capacity = 4096 stats_flush_interval_ms = 10 stats_retention_seconds = 34560000 stats_max_scan_entries = 100000 stats_cas_retries = 8 [[ably_compat.keys]] app_id = "app-id" key_name = "app-key-readonly" secret = "replace-with-a-secret" capability = '{"chat:*":["subscribe","history"]}' revocable_tokens = true enabled = true rotation_id = "2026-07" ``` Each entry supports optional `created_at_ms`, `expires_at_ms`, and `revoked_at_ms` rotation boundaries. Disabled, expired, or revoked keys reject new key authentication. Tokens from a key with `revocable_tokens = true` are also rejected after that key is disabled, revoked, expired, or replaced with a different `rotation_id`. Secrets and bearer tokens are never written to logs. TokenRequest `timestamp` and `ttl` values accept JSON integers or decimal integer strings, matching values commonly forwarded from auth URL query parameters. Timestamps are accepted only within the configured skew. Nonces are claimed atomically in the configured cache for `nonce_ttl_seconds`, and issued token records use the same cache, allowing issuance on one node and use on another. Use a shared cache driver such as Redis in multi-node deployments. The server's `ably-compat` Cargo feature explicitly enables native AI Transport, recovery, delta, and push support. Runtime `[ably_compat].enabled` and the key registry remain the opt-in boundary; Protocol V1 routes and frames are unchanged. `realtime_admission = "placement_constraint"` is an operator/drain mode for a listener: authenticated Ably WebSocket upgrades receive `DISCONNECTED` error `50320`, prompting the SDK to try its configured fallback hosts over WebSocket. It does not enable Comet, polling, streaming, SSE, or any other realtime transport. The default is `accept`. `attach_timeout_ms` bounds the complete native presence/recovery/history attach path. On expiry Sockudo removes the partial subscription and returns `DETACHED`/`50003`; the default is 10 seconds. Ably realtime data queues use `[websocket].max_messages` and `[websocket].max_bytes`. When a slow session exceeds either configured bound, Sockudo marks its continuity lost and requires recovery instead of allowing the session to buffer beyond the operator-configured limit. The shipped configuration uses a 4 MiB byte bound so a healthy short burst of maximum-size application events is not mistaken for a stalled reader. Ably `delta=vcdiff` is negotiated per realtime channel and uses a fixed bounded compatibility cache; native `[delta_compression]` settings continue to configure only Sockudo's Pusher/V2 delta contract and do not select the Ably VCDIFF mode. Compatibility statistics use canonical UTC minute buckets in the configured cache and derive hour, day, and month results when queried. Sum counters and peak gauges retain their distinct rollup semantics. In multi-node or restart-safe deployments, use a shared persistent cache such as Redis; the memory cache is node-local and does not survive a restart. Redis-backed entries retain their configured TTL across an individual Sockudo node's graceful shutdown; use the explicit cache administration APIs for destructive prefix cleanup. The recording worker is bounded by `stats_queue_capacity`, combines at most 256 observations per write batch, and waits up to `stats_flush_interval_ms` to form a batch. REST and realtime publishes that receive an application ACK wait for the canonical bucket merge first. Outbound delivery accounting uses the nonblocking bounded path, so an abrupt process crash can lose observations that were still queued; queue saturation is reported as a drop rather than allowing unbounded memory growth. Retention, bounded query scans, and compare-and-swap retries are controlled by `stats_retention_seconds`, `stats_max_scan_entries`, and `stats_cas_retries`. `stats_fixture_ingest_enabled` exposes authenticated `POST /stats` only for conformance fixture provisioning and defaults to `false`. Fixture intervals are validated and written through the same minute-store and rollup contract as live observations; the HTTP handler does not return canned fixture values. Compatibility device registrations and channel subscriptions use `[push].storage_driver`, so clustered or restart-safe deployments should select the same durable push store used by native push APIs. ## Runtime backends These sections choose the infrastructure Sockudo coordinates through. A single node works with the defaults; any multi-node deployment needs a shared adapter and cache. | Section | Default driver | Purpose | | --- | --- | --- | | `[adapter]` | `local` | Cross-node fanout. Use `redis`, `redis-cluster`, `nats`, or another broker for more than one node. | | `[cache]` | `memory` | Shared cache and coordination. Use `redis` or `redis-cluster` with any non-local adapter. | | `[queue]` | `redis` | Webhook and push background work. The compiled default expects Redis; set `memory` explicitly for a single development node. | | `[rate_limiter]` | `memory` | Request, connection, event, and push limits. Enabled by default; use a shared driver so limits apply across nodes. | `adapter.driver = "local"` and `cache.driver = "memory"` keep all state inside one process. Running several nodes with these defaults silently splits channels and presence between nodes. Choose a shared adapter and cache before scaling out. ## Rate limiter client IP Set this when Sockudo runs behind a proxy or load balancer, otherwise every client shares the proxy's rate-limit bucket. `rate_limiter.api_rate_limit.trust_hops` and `rate_limiter.websocket_rate_limit.trust_hops` select the client's **one-based position from the right** of `X-Forwarded-For`, not a number of proxy machines. The Helm equivalents are `config.rateLimiter.apiTrustHops` and `wsTrustHops`. | Path directly in front of Sockudo | Header after the proxy appends its entries | Value | | --- | --- | --- | | NGINX using `$proxy_add_x_forwarded_for` | `, ` | `1` | | Google external Application Load Balancer | `, , ` | `2` | | No trusted proxy | Headers ignored; socket peer used | `0` (default) | Only enable a positive value when network access to Sockudo is restricted to trusted proxies, and each proxy sanitizes or appends the appropriate forwarding headers. Extra intermediaries can change the required value. Choosing too large an offset can select client-supplied data; hop counting does not authenticate the immediate peer. With a positive value, Sockudo uses `X-Real-IP` only when `X-Forwarded-For` is absent. The trusted proxy must overwrite `X-Real-IP`. A malformed, duplicated, or too-short `X-Forwarded-For` header falls back directly to the socket peer, even if `X-Real-IP` is present. Invalid or duplicated `X-Real-IP` also falls back. A warning is emitted once per extractor and contains only the configured offset, entry count, and failure reason. All accepted IPs are normalized before use as rate-limit keys. **Migration:** `0` now ignores **both** forwarding headers. Earlier versions honored `X-Real-IP` even at `0`, allowing directly connected clients to choose their rate-limit bucket. Deployments relying on a trusted proxy's `X-Real-IP` must explicitly configure a positive value for each enabled limiter and restrict direct access to the server. Without this configuration, proxied clients share the socket peer's bucket. ## OMQ adapter Choose OMQ when you want horizontal fanout without operating a broker. `adapter.driver = "omq"` enables brokerless horizontal fanout over OMQ PUB/SUB. Each node binds one subscriber endpoint and connects its publisher to every other node's subscriber endpoint. ```toml [adapter] driver = "omq" fallback_to_local = false [adapter.omq] bind_endpoint = "tcp://0.0.0.0:5556" connect_endpoints = [ "tcp://sockudo-1.internal:5556", "tcp://sockudo-2.internal:5556", ] prefix = "sockudo_adapter" request_timeout_ms = 5000 nodes_number = 3 io_threads = 1 send_hwm = 100000 recv_hwm = 100000 ``` OMQ uses transport topics derived from `prefix`: broadcast, request, response, and per-node request topics. The adapter is not persistent; if a process or network path is unavailable, messages in flight are not retained by a broker. ## Apache Iggy adapter and queue `adapter.driver = "iggy"` and `queue.driver = "iggy"` use Apache Iggy (build with the `iggy` Cargo feature). Sockudo ships the Iggy Rust SDK 0.11, which speaks the Iggy 0.9 server wire protocol: run Iggy server 0.9 or newer. Older 0.8.x servers are not wire-compatible, and Iggy 0.9 starts from a fresh data directory. An Iggy 0.9 server that binds a wildcard address (`0.0.0.0`) also needs `node.advertised_address` (`IGGY_NODE_ADVERTISED_ADDRESS`) set to the address clients dial. The same settings work for a single Iggy server and for a Viewstamped Replication (VSR) cluster. Against a cluster, the SDK reads the cluster roster after sign-in, follows the current leader, fails over to the surviving replicas when its node disappears, routes consumer polls and offset commits to the partition primary, and rejoins consumer groups after membership loss. No Sockudo setting is needed for any of that. ```toml [adapter] driver = "iggy" [adapter.iggy] connection_string = "iggy://iggy:iggy@iggy-1.internal:8090" # Extra replicas tried in order when the connection_string node is down at startup. cluster_seeds = ["iggy-2.internal:8090", "iggy-3.internal:8090"] connect_timeout_ms = 2000 failover_timeout_ms = 15000 consumer_name = "" stream = "sockudo" topic_prefix = "sockudo-adapter" request_timeout_ms = 5000 poll_interval_ms = 5 poll_batch_size = 100 partitions_count = 1 partition_id = 0 auto_create = true start_from_latest = true durability = "replicated" consumer_offset_durability = "replicated" [queue] driver = "iggy" [queue.iggy] connection_string = "iggy://iggy:iggy@iggy-1.internal:8090" cluster_seeds = ["iggy-2.internal:8090", "iggy-3.internal:8090"] connect_timeout_ms = 2000 failover_timeout_ms = 15000 stream = "sockudo" queue_topic_prefix = "sockudo-queue" consumer_group_prefix = "sockudo-workers" partitions_count = 3 durability = "persisted" consumer_offset_durability = "replicated" ``` | Key | Default | Purpose | | --- | --- | --- | | `connection_string` | `iggy://iggy:iggy@127.0.0.1:8090` | First node to dial. Its credentials and query options apply to every seed. | | `cluster_seeds` | `[]` | Extra `host:port` replicas, tried in order when an earlier one cannot be reached. They only bootstrap the connection; afterwards the SDK fails over using the roster the cluster advertises. | | `connect_timeout_ms` | `2000` | Bound on the TCP dial to each node while `cluster_seeds` is non-empty. A seed that refuses, cannot be resolved, or does not accept within this bound is skipped. Without seeds the SDK's own reconnection policy applies, as before. | | `failover_timeout_ms` | `15000` | How long a broadcast or queue publish, and the sign-in to a seed that accepted the dial, wait for the cluster to elect a leader or fail over before giving up. Keep it above Iggy's `cluster.heartbeat_timeout` (5 s by default) plus the election. Cross-node requests and health checks keep `request_timeout_ms`. | | `durability` | `replicated` | Message durability for topics Sockudo creates. `replicated` acknowledges after a VSR quorum commit; `persisted` also waits for recoverable stable-storage copies at that quorum. | | `consumer_offset_durability` | `replicated` | Same choice for consumer offsets stored on those topics. | Durability applies only to topics Sockudo creates (`auto_create = true`); it does not change existing topics. Single-node servers accept both values. The connection string's query options tune the SDK's failover: `reconnection_retries` (passes over the known nodes, default `unlimited`), `reconnection_interval` (default `1s`), `reestablish_after` (default `5s`), and `heartbeat_interval` (default `5s`), plus `tls`, `tls_domain`, `tls_ca_file`, and `nodelay`. For example, `iggy://iggy:secret@iggy-1:8090?reconnection_interval=500ms&heartbeat_interval=2s`. `nodes_number` is the number of **Sockudo** nodes the adapter waits on for request/response fan-in, not the number of Iggy replicas. A publish that runs out of time stops waiting, but its Iggy call keeps running to completion in the background. The SDK reconnects and signs in inside whichever call first meets a dead connection, and cancelling that call mid-sign-in would leave the shared client permanently disconnected. As a result, a publish reported as failed can still be delivered once the cluster recovers, and a retry after a reconnect can deliver a message twice: Iggy deduplicates retries only within one connection session. Protocol V2 clients drop duplicates by `message_id`. At most 10,000 abandoned calls run at once per component; beyond that, new calls fail immediately until the backlog drains. The adapter and queue health checks ping the connected Iggy node. The SDK keeps every client on the leader, so when the leader crashes, every Sockudo node's `/up` can return 503 until the election finishes (6 to 9 seconds with Iggy's defaults in testing). Size readiness probes to ride that out: Kubernetes' defaults (`periodSeconds: 10`, `failureThreshold: 3`) already do. Each component logs `cluster topology discovered` at startup with the cluster name, roster size, and current leader. `docker-compose.iggy-cluster.yml` runs a three-replica Iggy cluster with three Sockudo nodes for local failover testing. ## Redis connections, Sentinel, and TLS Shared by every Redis-backed component. Change it to point at your Redis deployment, enable TLS, or use Sentinel for automatic failover. `[database.redis]` configures the Redis connection used by the Redis adapter, cache, queue, rate limiter, and delta coordinator. Without Sentinel, `[database.redis.master_tls]` secures the direct Redis data connection for every consumer. When `sentinels` is non-empty, Sockudo connects through Redis Sentinel using a native Sentinel client (rather than a direct URL) and can secure both connection hops independently. Redis Cluster also reuses `master_tls` certificate material; the legacy `cluster.use_tls` switch continues to enable TLS with the default root store. `[database.redis.cluster]` is the canonical Redis Cluster topology for every cluster component — adapter, cache, rate limiter, delta coordinator, and queue. When `queue.driver = "redis-cluster"` and `queue.redis_cluster.nodes` is not explicitly set, the queue inherits the `database.redis.cluster` seeds (normalized with the shared cluster username/password and TLS settings). Set `queue.redis_cluster.nodes` only to override the queue's seeds independently of the rest of the deployment. | Key | Purpose | | --- | --- | | `sentinels` | List of `{ host, port }` Sentinel nodes. A non-empty list enables Sentinel mode. | | `name` | Monitored master (Sentinel service) name. | | `username` / `password` | Auth for the master/replica data connection. | | `sentinel_username` / `sentinel_password` | Auth for the Sentinel control connection. | | `[database.redis.sentinel_tls]` | TLS for the client→Sentinel control connection. | | `[database.redis.master_tls]` | TLS for direct Redis, Redis Cluster, or the Sentinel-resolved master/replica data connection. | Each TLS block accepts `enabled`, `accept_invalid_certs` (skips verification; never enable it in production), `ca_path` (PEM CA for private CAs), and `client_cert_path` + `client_key_path` (PEM pair for mutual TLS / client-certificate auth). For a direct Redis deployment signed by a private CA, no Sentinel configuration is required: ```toml [database.redis] host = "redis.internal" port = 6380 [database.redis.master_tls] enabled = true ca_path = "/etc/sockudo/tls/redis-ca.pem" ``` ```toml [database.redis] name = "mymaster" db = 0 username = "appuser" password = "app-secret" sentinel_username = "sentineluser" sentinel_password = "sentinel-secret" [[database.redis.sentinels]] host = "sentinel-1.internal" port = 26379 [[database.redis.sentinels]] host = "sentinel-2.internal" port = 26379 [database.redis.sentinel_tls] enabled = true ca_path = "/etc/sockudo/tls/ca.pem" client_cert_path = "/etc/sockudo/tls/client.pem" client_key_path = "/etc/sockudo/tls/client.key" [database.redis.master_tls] enabled = true ca_path = "/etc/sockudo/tls/ca.pem" client_cert_path = "/etc/sockudo/tls/client.pem" client_key_path = "/etc/sockudo/tls/client.key" ``` All Redis consumers inherit the native Sentinel topology when they do not define a URL override. An adapter, cache, queue, or rate-limiter URL override selects its standalone path; `master_tls` still applies to that direct data connection. Each Sentinel-backed consumer polls Sentinel once per second with `SENTINEL GET-MASTER-ADDR-BY-NAME` and follows a failover without a restart: when Sentinel promotes a new primary, cached connections are dropped and rebuilt against it, and the adapter's Pub/Sub listener resubscribes. Recovery normally completes about a second after the promotion, so health probes such as `/up` recover on their own. If you restrict the Sentinel user with ACLs, allow that command alongside the ones used for primary resolution (`SENTINEL MASTERS` and `ROLE`). Primary changes are logged at `info` as `redis sentinel primary changed, reconnecting`. ## Protocol features These sections enable and bound the Protocol V2 feature set. Most are off by default or bounded conservatively; enable a feature here, then opt apps or channel namespaces into it through policy where required. Protocol V1 delivery is unaffected by every section in this table. | Section | Purpose | | --- | --- | | `[connection_recovery]` | V2 replay buffers and resume behavior: `enabled` (default `false`), `buffer_ttl_seconds` (default `120`), and `max_buffer_size` (default `100`), which bounds both the per-channel hot buffer and one durable fallback replay. | | `[history]` | Durable channel history. | | `[versioned_messages]` | V2 mutable-message storage, version paging, and retention. Requires the `versioned-messages` Cargo feature (also enabled by `ai-transport`); without it the server logs a warning and disables this section. | | `[presence_history]` | Historical presence transitions and snapshots. | | `[annotations]` | V2 annotation publish/delete/summary surfaces. Requires the `versioned-messages` Cargo feature (also enabled by `ai-transport`); without it the server logs a warning and disables this section. | | `[delta_compression]` | Delta algorithms, per-socket state limits, full-message interval, and conflation behavior. Enabled by default with the `fossil` algorithm. | | `[tag_filtering]` | V2 event/tag/JMESPath subscription predicates and tag projection controls. Predicate source, AST, tree, and projected-message memory are hard-bounded. | | `[idempotency]` | HTTP publish deduplication: `enabled` (default `true`), `ttl_seconds` (default `120`), `max_key_length` (default `128`). | | `[ephemeral]` | Allows `extras.ephemeral` messages that skip the recovery buffer and webhooks. Enabled by default. | | `[echo_control]` | Lets V2 connections opt out of receiving their own publishes: `enabled` and `default_echo_messages` (both default `true`). | | `[event_name_filtering]` | V2 subscription event-name filters: `enabled` (default `true`), `max_events_per_filter` (default `50`), `max_event_name_length` (default `200`). | | `[webhooks]` | Webhook delivery, batching, and retry. | | `[ai_transport]` | AI Transport validation and session-channel matching. | | `[ai_transport.rollup]` | Append rollup egress coalescing and orphan tracking. | | `[[ai_transport.channels]]` | Channel prefixes where AI Transport validation applies. | | `[mcp]` | Embedded Model Context Protocol server: path or dedicated port, host/origin allow-lists, rate limits, tool hiding. Requires the `mcp` Cargo feature. | | `[[mcp.tokens]]` | Bearer tokens with `read` / `write` / `admin` scopes and app allow-lists. See [MCP server](/docs/server/mcp). | Annotations, including the optional Ably projection, require both global `[annotations].enabled = true` and app/channel policy opt-in. Set `annotations_enabled = true` under `[app_manager.array.apps.policy.channels]` or on the matching channel namespace. Enabling the Ably facade does not bypass either gate, and Protocol V1 never receives annotation frames. `[versioned_messages].driver` also selects the annotation authority. `postgres`, `mysql`, `dynamodb`, `scylladb`, and `surrealdb` each use their native atomic or conditional commit primitives for annotation serials, stable create IDs, canonical events, projections, replay, and retention. The corresponding Cargo feature and database configuration must be present. `memory` is a single-node development implementation; clustered startup rejects it. For any non-local adapter, use `cache.driver = "redis"` or `cache.driver = "redis-cluster"` for compatibility tokens, nonce replay, revocations, session-owner leases, idempotency receipts, and retained stats. Sockudo fails startup or the affected security operation when this coordination authority is unavailable; it does not fall back to a private per-process cache. ## Push Push requires the `push` Cargo feature plus a provider feature for each platform (`push-fcm`, `push-apns`, `push-webpush`, `push-hms`, `push-wns`). Configure storage, queue, workers, and per-provider switches in `[push]`. Provider credentials are stored through the [push credential API](/docs/reference/http-endpoints#credentials-and-templates) or supplied to workers through [provider environment variables](/docs/reference/environment-variables#push-providers); there is no `[push.providers.*]` table. | Section | Purpose | | --- | --- | | `[push]` | Storage and queue drivers, `fcm_enabled`, `apns_enabled`, `webpush_enabled`, `hms_enabled`, `wns_enabled`, optional `*_credential_ref` values, worker counts, fanout thresholds, backpressure, status retention, scheduler, repair, and cleanup. | | `[push.apns]` | APNs topic, endpoint, Live Activities, broadcast channels, connection pooling, and extra CA roots. | | `[push.retry]` | Provider delivery retry attempts, backoff, jitter, and `Retry-After` handling. | | `[push.circuit_breaker]` | Provider circuit-breaker failure threshold, cooldown, and half-open concurrency. | | `[push.default_quotas]` | Default per-app acceptance rate, daily delivery quota, fanout, and in-flight limits. | | `[push.payload_redaction]` | Redaction of payloads, template data, and provider overrides in logs. | | `[[push_rules]]` | Optional channel-publish to push-notification rules. | ## Capability tokens Protocol V2 capability tokens do not currently have a TOML `[auth.capability_tokens]` switch. They are accepted through the WebSocket `token` query parameter and refreshed with `sockudo:auth`. Limits are compiled constants in `sockudo-core/src/capability_token.rs`: HS256 only, max token size 8 KiB, max `client_id` 128 bytes, max `jti` 128 bytes, max lifetime 24 hours, and 30 seconds clock skew. Revocation uses signed HTTP `POST /apps/{appId}/revocations` and shared cache keys. ## AI Transport and Mutable Defaults Runtime config uses `[versioned_messages]`; `VersionStore` is the Rust trait/storage abstraction. There is no separate `[version_store]` TOML section in the current code. The following block is machine checked by `sockudo-core` tests. Update it whenever code defaults change. ```toml doc-defaults [versioned_messages] enabled = false driver = "memory" max_page_size = 100 retention_window_seconds = 0 purge_interval_seconds = 300 purge_batch_size = 1000 max_purge_per_tick = 100000 [history] enabled = false rewind_enabled = true backend = "postgres" retention_window_seconds = 86400 max_page_size = 100 writer_shards = 16 writer_queue_capacity = 4096 purge_interval_seconds = 300 purge_batch_size = 1000 max_purge_per_tick = 100000 [history.postgres] table_prefix = "sockudo_history" write_timeout_ms = 5000 [presence_history] enabled = false retention_window_seconds = 86400 max_page_size = 100 [annotations] enabled = false [ai_transport] enabled = false max_accumulated_message_bytes = 1048576 max_appends_per_message = 4096 max_transport_value_bytes = 256 max_steer_codec_message_ids_bytes = 2048 max_open_streaming_messages_per_channel = 1024 [ai_transport.rollup] enabled = true default_window_ms = 40 min_window_ms = 0 max_window_ms = 500 orphan_ttl_ms = 60000 wheel_tick_ms = 5 shards = 64 [push] storage_driver = "memory" queue_driver = "memory" allow_memory_drivers = false fcm_enabled = false apns_enabled = false webpush_enabled = false hms_enabled = false wns_enabled = false accept_worker_count = 1 planner_worker_count = 1 shard_worker_count = 1 dispatch_worker_count = 1 dispatch_max_outbound_requests = 32 feedback_worker_count = 1 retry_worker_count = 1 queue_partition_count = 1 channel_shard_count = 1 fanout_fast_threshold = 10000 fanout_shard_size = 100000 fanout_sync_threshold = 0 backpressure_lag_threshold_secs = 60 publish_status_ttl_days = 30 stale_device_max_age_days = 90 dry_run = false analytics_enabled = false analytics_retention_days = 30 scheduler_interval_secs = 5 repair_interval_secs = 30 repair_min_age_secs = 30 repair_batch_size = 100 cleanup_interval_secs = 300 cleanup_batch_size = 1000 cleanup_max_deleted_per_tick = 100000 [push.apns] topic = "com.example.app" endpoint = "https://api.push.apple.com" live_activities_enabled = false live_activity_topic = "" # derives .push-type.liveactivity bundle_id = "" # derives topic broadcast_enabled = false broadcast_endpoint = "https://api-broadcast.push.apple.com" management_endpoint = "https://api-manage-broadcast.push.apple.com:2196" default_broadcast_expiration_secs = 3600 # maximum 28800 connect_timeout_ms = 5000 request_timeout_ms = 10000 pool_idle_timeout_secs = 90 max_idle_connections_per_host = 128 tcp_keepalive_secs = 60 http2_keepalive_interval_secs = 30 http2_keepalive_timeout_secs = 10 ca_certificate_path = "" # optional extra PEM roots for private egress proxies or staging mocks [push.retry] max_attempts = 5 initial_backoff_ms = 1000 max_backoff_ms = 60000 max_elapsed_secs = 86400 jitter = true jitter_ratio_percent = 20 respect_retry_after = true [push.circuit_breaker] failure_threshold = 5 cooldown_secs = 60 half_open_max_inflight = 10 [push.default_quotas] acceptance_rps = 100 delivery_quota_daily = 0 fanout_max = 0 inflight_max = 1000 [push.payload_redaction] redact_payload = true redact_template_data = true redact_provider_overrides = true allow_debug_payload_logging = false [[push_rules]] enabled = true channel_pattern = "" event_filter = [] rate_limit_per_second = 100 [push_rules.payload_mapping] title_field = "title" body_field = "body" template_data_field = "data" include_remaining_fields = true [mcp] enabled = false path = "/mcp" # port = 6100 allowed_hosts = [] allowed_origins = [] allow_anonymous = false anonymous_scopes = ["read"] request_timeout_ms = 30000 max_body_bytes = 1048576 session_ttl_seconds = 1800 rate_limit_per_minute = 600 disabled_tools = [] # [[mcp.tokens]] # name = "ops-agent" # token = "${SOCKUDO_MCP_OPS_TOKEN}" # scopes = ["read", "write"] # apps = ["*"] ``` `max_messages_per_channel`, `max_bytes_per_channel`, `max_events_per_channel`, credential refs, and external key refs default to unset and are omitted from the checked block. `storage_driver = "memory"` and `queue_driver = "memory"` are node-local development drivers. In `mode = "production"`, Sockudo rejects them unless `allow_memory_drivers = true` is set explicitly. With `push.queue_driver = "memory"`, push uses a separate node-local queue and does not require `queue.driver` to match. Broker-backed push queue work runs on the shared `[queue]` backend, so `push.queue_driver` must name the same driver as `queue.driver`; Sockudo refuses to start when they differ. Push needs per-message delays for scheduled sends and retries, so supported broker drivers are `redis` (including Sentinel), `redis-cluster`, standard `sqs` and `pulsar`. Sockudo refuses to start push on `nats`, `rabbitmq`, `kafka`, `iggy`, `google-pubsub` and SQS FIFO queues (`queue.sqs.fifo = true`), which have no per-message delay. SQS caps delays at 15 minutes; longer push delays are re-deferred when the job arrives. Push publish admission also requires a healthy queue, a safe store, local pipeline workers, and local provider-worker capability. Raw provider-specific recipients require the matching provider worker before the request is accepted; shard-path publishes also require a local shard worker. `backpressure_lag_threshold_secs` limits the oldest actionable ready or inflight push queue item age; set it to `0` to rely only on queue-depth thresholds. Push repair runs in monolith workers when `repair_interval_secs > 0`; it scans stale queued durable publish logs and recreates missing `push.publish.v1` queue work after the publish has been queued for at least `repair_min_age_secs`. Push cleanup runs in monolith workers when `cleanup_interval_secs > 0`. It purges terminal publish statuses older than `publish_status_ttl_days`, delivery events and operator invalidations older than `analytics_retention_days`, expired idempotency records, expired scheduler locks, and any persisted dead-letter inspection rows written by a selected backend. Queue-native dead-letter metadata is inspected through `/apps/{appId}/push/deadLetters`. Cleanup is bounded by `cleanup_batch_size` per category and `cleanup_max_deleted_per_tick` overall. ## Observability Metrics are on by default; tracing export is opt-in. Change these sections to rename metrics, move the scrape listener, or ship telemetry to an OpenTelemetry collector. | Section | Purpose | | --- | --- | | `[metrics]` | Enables metrics and configures the Prometheus scrape endpoint. Defaults: `enabled = true`, `host = "0.0.0.0"`, `port = 9601`. | | `[metrics.prometheus]` | Prometheus metric naming options. | | `[metrics.tcp_exporter]` | Optional metrics-rs TCP event exporter for live clients and sidecars. | | `[opentelemetry]` | Optional OTLP traces, metrics, logs, resource identity, batching, and propagation. | | `[logging]` | Controls ANSI colors and tracing-target inclusion for text/JSON output. | | [logging environment](/docs/reference/logging) | Runtime filters, structured output, lifecycle fields, and data-safety policy. | OpenTelemetry is disabled by default and is additive to Prometheus and local logs: ```toml [opentelemetry] enabled = true traces_enabled = true metrics_enabled = true logs_enabled = true service_name = "sockudo" service_namespace = "realtime" deployment_environment = "production" resource_attributes = { "cloud.region" = "eu-central-1" } endpoint = "http://otel-collector:4317" export_timeout_ms = 10000 batch_scheduled_delay_ms = 5000 batch_max_queue_size = 2048 batch_max_export_batch_size = 512 metric_export_interval_ms = 60000 propagation_trace_context = true propagation_baggage = true ``` | Key | Default | Purpose | | --- | --- | --- | | `enabled` | `false` | Initializes OpenTelemetry and enables the selected signals. | | `traces_enabled` | `true` | Exports spans when OpenTelemetry is enabled. | | `metrics_enabled` | `true` | Exports Sockudo metrics through OTLP independently of the Prometheus listener. | | `logs_enabled` | `true` | Exports structured log records while retaining local output. | | `service_name` | `sockudo` | OpenTelemetry `service.name`. Must not be empty when enabled. | | `service_namespace` | unset | Optional `service.namespace`. | | `deployment_environment` | unset | Optional `deployment.environment.name`. | | `resource_attributes` | `{}` | Additional string-valued resource attributes. Never store secrets here. | | `endpoint` | unset | Optional common OTLP endpoint. When unset, standard SDK/exporter configuration applies. | | `export_timeout_ms` | `10000` | Timeout for one exporter request. | | `batch_scheduled_delay_ms` | `5000` | Maximum span/log batching delay. | | `batch_max_queue_size` | `2048` | Bounded batch processor queue capacity. | | `batch_max_export_batch_size` | `512` | Maximum export batch size; cannot exceed the queue capacity. | | `metric_export_interval_ms` | `60000` | Periodic metric export interval. | | `propagation_trace_context` | `true` | Extracts and injects W3C `traceparent` and `tracestate`. | | `propagation_baggage` | `true` | Extracts and injects W3C baggage. Do not place secrets in baggage. | Use standard `OTEL_*` environment variables for the OTLP protocol (`grpc`, `http/protobuf`, or `http/json`), signal-specific endpoints, headers, compression, sampling, and SDK limits. The complete mapping is in the [environment variable reference](/docs/reference/environment-variables#opentelemetry). HTTPS exporters use platform/web PKI roots; custom CA bundles and mTLS exporter configuration are not supported by the current Rust SDK integration. The shipped CORS configuration allows `traceparent`, `tracestate`, and `baggage`; retain those headers in a custom allowlist when browser clients participate in distributed traces. Collector failures are fail-open: export uses bounded queues and does not change request handling, `/live`, or `/up`. Invalid static settings are still rejected during startup. Stable traces, metrics, and logs are supported; OpenTelemetry profiles are not. With the defaults, `/metrics` is served on every interface at port `9601` without authentication. Bind `[metrics].host` to an internal interface or restrict the port with network policy. ## Memory cache expiration The `memory` cache driver honors the TTL supplied by each write. `cache.memory.ttl` (default `300` seconds) is both the default for writes with a zero TTL and the maximum for writes with a longer TTL. Set it to `0` to remove that maximum; a write with a zero TTL then creates a nonexpiring entry. Capacity eviction and process restarts can still remove entries. Overwriting an entry resets its expiry. Incrementing an existing counter with a zero TTL preserves its current expiry. Reading an entry does not extend its TTL. The same rules apply to the memory fallback used during a Redis outage. Recovery skips entries that expired while synchronization was in progress and copies live entries with their remaining TTL, rounded up to whole seconds for Redis. ## Example production skeleton A minimal multi-node starting point: shared Redis for fanout, cache, and queue, with metrics on the default port and durable push storage. ```toml title="config.toml" host = "0.0.0.0" port = 6001 debug = false [adapter] driver = "redis" [cache] driver = "redis" [queue] driver = "redis" [metrics] enabled = true port = 9601 [metrics.tcp_exporter] enabled = false host = "127.0.0.1" port = 5000 buffer_size = 1024 [push] storage_driver = "postgres" queue_driver = "redis" publish_status_ttl_days = 30 analytics_retention_days = 30 cleanup_interval_secs = 300 ``` ## Queue v2 reliability Tune `[queue.reliability]` to trade redelivery window against throughput. Queue v2 provides at-least-once delivery. Successful processors acknowledge a job; errors retry with bounded exponential backoff and jitter; exhausted jobs move to a dead-letter queue. Exactly-once execution is not promised, so handlers must remain idempotent. Redis, Redis Sentinel, and Redis Cluster additionally provide stable job IDs, deduplication, delayed jobs, renewable leases, stalled worker recovery, lifecycle events, and queue depth statistics through atomic Lua transitions. Redis workers preserve the configured callback concurrency while pipelining Redis I/O. `worker_prefetch` bounds the number of leased jobs buffered ahead of each callback worker; the default `16` amortizes claim, acknowledgement, and lease-renewal round trips without creating unbounded memory or failover exposure. Lower it for a smaller redelivery window, or raise it when Redis latency leaves callback workers idle. `queue.redis.response_timeout_ms` defaults to `5000`, allowing Sentinel clients to discard a stale primary connection promptly; `0` disables that deadline. For the legacy-equivalent high-throughput workload—batch enqueue, generated job IDs, immediate delivery, the configured default attempt count, no application deduplication key, and `event_retention = 0`—the Redis drivers use compact length-delimited frames and one raw atomic Redis transaction. Workers materialize each frame into the same leased Queue v2 state machine when they claim it, so retries, acknowledgements, stalled recovery, and dead lettering are unchanged while admission avoids per-job Redis writes. Supplying a stable `job_id`, delay, deduplication key, attempt override, or lifecycle-event retention selects the stricter per-job atomic path. Use that path when a caller must safely retry an enqueue whose network result was ambiguous. ```toml [queue] driver = "redis" # memory, redis, redis-cluster, nats, rabbitmq, kafka, iggy, pulsar, google-pubsub, sqs, sns [queue.reliability] max_attempts = 5 retry_base_delay_ms = 1000 retry_max_delay_ms = 60000 retry_jitter = 0.2 lease_duration_ms = 30000 lease_renew_interval_ms = 10000 stalled_batch_size = 100 worker_poll_interval_ms = 500 worker_prefetch = 16 shutdown_timeout_ms = 30000 completed_retention = 1000 failed_retention = 10000 event_retention = 10000 deduplication_ttl_ms = 300000 memory_capacity = 100000 max_batch_size = 1000 ``` `shutdown_timeout_ms` bounds the lifecycle/webhook drain after the listener grace period and also the subsequent queue worker shutdown. These are separate budgets; reserve time for both in your process supervisor. See [shutdown delivery](/docs/server/webhooks#shutdown-delivery). Broker-backed queues use native bulk/pipelined publishing rather than the compatibility loop: NATS and RabbitMQ pipeline acknowledgements, Kafka feeds librdkafka concurrently, Iggy and Pulsar use their producer batch APIs, Google Pub/Sub reuses its batching publisher, and SQS/SNS use provider batches of at most 10 entries. `max_batch_size` is an upper bound; a provider's smaller hard limit still applies. Topic, stream, queue, and publisher handles are cached after successful provisioning instead of issuing control-plane requests for every job. `worker_prefetch` also bounds concurrent callbacks for NATS, RabbitMQ, Pulsar, Google Pub/Sub, and standard SQS queues. SQS FIFO batches remain serial to preserve message-group order. `max_attempts` configures NATS delivery limits, Kafka and Iggy retry/DLQ transitions, RabbitMQ republish/DLQ transitions, and Pulsar's native dead-letter policy. SQS dead-lettering uses an operator-managed redrive policy; Sockudo never creates an SQS DLQ. Before consuming the `webhooks` queue, Sockudo reads its `RedrivePolicy`. If one exists, `maxReceiveCount` bounds delivery and the queue reports `dead_letter` capability. If none exists, Sockudo logs a warning at startup and deletes a failing job after `queue.reliability.max_attempts` receives. Set `queue.sqs.require_redrive_policy = true` (`QUEUE_SQS_REQUIRE_REDRIVE_POLICY=true`) to fail startup instead. Push stage queues are not validated. Terraform example (DLQ retention must exceed the source queue's: for standard queues, SQS keeps the original enqueue timestamp when it moves a message, so a shorter DLQ retention can expire it immediately): ```hcl resource "aws_sqs_queue" "webhooks_dlq" { name = "sockudo-webhooks-dlq" message_retention_seconds = 1209600 # 14 days } resource "aws_sqs_queue" "webhooks" { name = "sockudo-webhooks" message_retention_seconds = 345600 # 4 days redrive_policy = jsonencode({ deadLetterTargetArn = aws_sqs_queue.webhooks_dlq.arn maxReceiveCount = 3 }) } ``` FIFO note: with FIFO, the DLQ must also be FIFO, and one failing job blocks its message group (the whole queue with the default single `message_group_id`) until redrive or the fallback delete. Prefer a standard queue for webhooks. Queue v2 options are fail-closed. A broker that cannot honor per-job delay, deduplication, or attempt overrides returns an error instead of silently discarding the option. SQS standard queues support delays up to 15 minutes; SQS FIFO queues support stable IDs/deduplication but not per-message delay. Pulsar supports per-job delay through native delayed delivery (`deliverAt`) on the Shared subscription Sockudo's workers use; the broker's `delayedDeliveryEnabled` must stay on (the default). NATS, RabbitMQ, Kafka, Iggy and Google Pub/Sub reject per-job delay. The Redis drivers use a new `:v2:{queue-id}:*` key namespace. All keys for one logical queue share a Redis Cluster hash tag, so enqueue, claim, ack, retry, lease renewal, delayed promotion, and dead-letter transitions are single-slot and atomic. Existing RPUSH/BLPOP list keys are not deleted during startup or shutdown; drain them with the previous release before switching producers. When `database.redis.sentinels` is configured and `queue.driver = "redis"`, the queue resolves the current primary through native Sentinel support. Sentinel and primary authentication/TLS settings remain independent. A `queue.redis.url_override` selects standalone Redis/rediss instead and intentionally disables inherited Sentinel discovery for that queue. When `queue.driver = "redis-cluster"`, the queue resolves its cluster seeds in this order: explicit `queue.redis_cluster.nodes` (normalized with the shared cluster auth/TLS options), then `database.redis.cluster` when the queue nodes are unset, then a localhost dev fallback. Leaving `queue.redis_cluster.nodes` empty is the recommended way to share one Redis Cluster topology across the adapter, cache, rate limiter, delta coordinator, and queue. Primary promotion preserves at-least-once, not exactly-once, semantics. A hard primary loss can redeliver callbacks that were active or prefetched when the node failed; a planned Sentinel switchover can temporarily expose a larger duplicate window while the old primary is still reachable. Use the stable job ID as an idempotency key, and quiesce producers/workers before a planned switchover when duplicate callbacks must be minimized. SNS is producer-only. Configuring a consumer on SNS now fails instead of reporting a successful no-op. A requested backend that was not compiled into the binary also fails startup instead of silently falling back to memory. ## Next steps - [Environment variables](/docs/reference/environment-variables): Override these settings per environment and inject secrets. - [Static configuration](/docs/deployment/static-configuration): Loading order, TOML versus JSON, and secret handling. - [Scaling](/docs/server/scaling): Choose adapters, caches, and queues for horizontal fanout. - [Security](/docs/server/security): Harden auth, origins, rate limits, and operator endpoints.