# Sockudo Documentation (https://sockudo.io/docs)
> Build, operate, and extend Sockudo: the self-hosted Rust realtime server with Pusher compatibility, Protocol V2, clustered fanout, recovery, history, push, and official SDKs.
Sockudo is a self-hosted realtime server written in Rust. It speaks the Pusher Channels protocol, so existing Pusher clients and server libraries work against it, and it adds a native protocol for durable history, connection recovery, mutable messages, and other features Pusher does not offer. Start here whether you are replacing Pusher, building a new realtime product, or evaluating Sockudo for a clustered production deployment.
- [Install and verify](/docs/getting-started/installation): Run Sockudo locally, choose features, inspect health probes, and understand the default development app.
- [Connect the first loop](/docs/getting-started/first-connection): Subscribe from a client, publish from a trusted backend, add idempotency, and confirm metrics move.
- [Migrate from Pusher](/docs/getting-started/migration): Map pusher-js, Laravel Echo, Channels auth, event names, and publish calls onto Sockudo.
- [Design Protocol V2 streams](/docs/clients/protocol-v2): Use message IDs, serial continuity, rewind, deltas, tag filters, annotations, and recovery.
- [Deploy to production](/docs/deployment): Choose a topology, configure files and secrets, tune hosts, and deploy on VMs, containers, Kubernetes, or cloud platforms.
- [Reference the API](/docs/reference/http-endpoints): Find every server endpoint for publishing, channel state, history, mutations, annotations, push, and operations.
## Key terms
These terms appear on every page of the documentation.
| Term | Meaning |
| --- | --- |
| App | A tenant on the server. Each app has an `id`, a public `key` that clients use to connect, and a `secret` that only your backend holds and uses to sign requests. |
| Channel | A named stream that clients subscribe to. The name prefix (`private-`, `presence-`, `private-encrypted-`) decides whether a subscription needs your backend's approval. |
| Event | A named message published to one or more channels, such as `order.created`. |
| Protocol V1 | The Pusher-compatible wire protocol. It is the default, so `pusher-js`, Laravel Echo, and Pusher server SDKs keep working unchanged. |
| Protocol V2 | Sockudo's native wire protocol. Clients opt in, and V2 adds `message_id`, `serial`, and `stream_id` metadata plus recovery, rewind, deltas, filters, and more. |
| Adapter | The layer that relays broadcasts and presence between Sockudo nodes. The `local` adapter serves a single node; Redis, NATS, Kafka, and others connect a cluster. |
| App manager | Where app records live: inline in the config file (`memory`) or in a database such as PostgreSQL or MySQL. |
## Production topology
Sockudo sits between realtime clients and your trusted backends. Clients hold long-lived WebSocket connections and subscribe to public, private, presence, or encrypted channels. Backends never hold a socket; they call the HTTP API, usually through a server SDK, to publish events, sign channel auth, mutate V2 messages, manage push devices, and inspect operational state.
- **WebSocket ingress:** client connect, subscribe, auth, presence, and realtime delivery.
- **HTTP control plane:** trusted publishes, channel state, history reads, message mutations, annotations, and push workflows.
- **Horizontal fanout:** Redis, NATS, Kafka, RabbitMQ, Pulsar, Google Pub/Sub, Iggy, or another configured adapter.
Start with the in-memory local profile, then move app records, caches, queues, adapters, history, and push status into production backends as the deployment grows. Each move is a configuration change, not a code change in your clients.
## Recommended path
Follow these steps in order the first time. Each one builds on the previous page.
1. Run a local Sockudo server from [Installation](/docs/getting-started/installation).
2. Publish and receive your first event from [First connection](/docs/getting-started/first-connection).
3. Add private and presence channel auth from [Authentication](/docs/getting-started/authentication).
4. Choose Protocol V1 or V2 from [Compatibility](/docs/reference/compatibility).
5. Wire your client from [Realtime Clients](/docs/clients) and your backend from [Server SDKs](/docs/server-sdks).
6. Follow the [Deployment guide](/docs/deployment), then harden the runtime with [Scaling](/docs/server/scaling), [Security](/docs/server/security), and [Observability](/docs/server/observability).
## Developer quick loops
The snippets below form one complete loop: start a server, subscribe, publish, and confirm the event arrives. They all use the same local development app:
| Setting | Value |
| --- | --- |
| App ID | `app-id` |
| App key | `app-key` |
| App secret | `app-secret` |
| WebSocket and HTTP API | `127.0.0.1:6001` |
| Prometheus metrics | `127.0.0.1:9601` |
These values are public examples. Replace every credential before exposing a server to a network, and keep the app secret on trusted servers only.
### Run Sockudo and check health
Start the published container image with the development app defined through environment variables:
```bash
docker run --rm --name sockudo \
-p 6001:6001 -p 9601:9601 \
-e SOCKUDO_DEFAULT_APP_ID=app-id \
-e SOCKUDO_DEFAULT_APP_KEY=app-key \
-e SOCKUDO_DEFAULT_APP_SECRET=app-secret \
ghcr.io/sockudo/sockudo:latest
```
In a second terminal, check the health probe and the metrics endpoint:
```bash
curl -s http://127.0.0.1:6001/up
curl -s http://127.0.0.1:9601/metrics | head -n 3
```
`/up` prints `OK` when the server is healthy and at least one app is configured. The metrics endpoint returns Prometheus text whose series start with `sockudo_`.
Use `cargo run --release -p sockudo` instead when you want to validate a source build or a specific feature set. [Installation](/docs/getting-started/installation) covers every install path.
### Subscribe from a client
```ts title="client.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
enabledTransports: ["ws"],
protocolVersion: 2,
});
client.subscribe("orders").bind("order.created", (payload) => {
console.log("received", payload);
});
```
Set `protocolVersion: 2` when the client needs Sockudo-native recovery, rewind, message IDs, tags, deltas, annotations, or mutable messages. Without it, the client connects with Protocol V1. Leave existing Pusher-compatible clients on V1 until you are ready to adopt V2 behavior.
### Publish from a trusted backend
```ts title="publish.ts"
const sockudo = new Sockudo({
appId: "app-id",
key: "app-key",
secret: "app-secret",
host: "127.0.0.1",
port: 6001,
useTLS: false,
});
await sockudo.trigger(
"orders",
"order.created",
{ id: "ord_123", total: 4200 },
{ idempotency_key: "order-created-ord_123" },
);
```
The client from the previous step logs `received { id: 'ord_123', total: 4200 }`. Server SDKs sign requests, format payloads, and hide HMAC details. Use raw HTTP only when you are implementing or debugging an SDK; [First connection](/docs/getting-started/first-connection#publish-over-raw-http) shows a signed `curl` request.
### Turn local config into a production shape
```toml title="config/config.toml"
port = 6001
host = "0.0.0.0"
debug = false
[app_manager]
driver = "pgsql"
[adapter]
driver = "redis"
[cache]
driver = "redis"
[queue]
driver = "redis"
[metrics]
enabled = true
port = 9601
```
| Section | What it controls |
| --- | --- |
| `[app_manager]` | Where app IDs, keys, secrets, and policies are stored. `postgres` reads them from PostgreSQL instead of the config file. |
| `[adapter]` | How nodes share broadcasts and presence. `redis` lets every node deliver events published on any other node. |
| `[cache]` | Shared short-lived state such as idempotency keys and capability-token revocations. |
| `[queue]` | Background work such as webhook delivery. |
| `[metrics]` | The Prometheus endpoint, served on its own port. |
The important move is not a single backend choice. It is separating connection fanout, app records, cache state, queue work, durable history, push status, and metrics so each subsystem can fail, scale, and be observed independently.
Drivers such as `redis` and `postgres` must be compiled into the binary. The prebuilt Linux binaries and container images include them; source builds need the matching `--features`. See [Feature flags](/docs/getting-started/installation#feature-flags).
## Local baseline
Running the binary from a repository checkout loads `config/config.toml`, which defines the same development app inline:
```toml title="config/config.toml"
port = 6001
host = "0.0.0.0"
[app_manager]
driver = "memory"
[[app_manager.array.apps]]
id = "app-id"
key = "app-key"
secret = "app-secret"
enabled = true
```
For production, replace every example credential, enable TLS at the edge, use managed app storage, and keep app secrets only on trusted servers.
## Production readiness checklist
- **Compatibility:** decide which clients stay on Protocol V1 and which channels may use Protocol V2 features.
- **Authentication:** keep app secrets server-side, sign private and presence auth responses, and reject stale HTTP signatures.
- **Fanout:** choose an adapter, configure sticky sessions when required, and test cross-node broadcast delivery before launch.
- **Durability:** enable history, recovery, version storage, annotations, and push status only for channels that need those contracts.
- **Idempotency:** attach stable keys to retried publishes, message mutations, push requests, and backend workflows.
- **Observability:** scrape `/metrics`, alert on connection churn, publish failures, adapter errors, webhook retries, push provider outcomes, and recovery failures.
- **Operations:** document feature flags, runtime config, secret rotation, rollout order, and rollback behavior for every backend dependency.
## Next steps
- [Overview](/docs/getting-started/overview): The mental model: surfaces, protocols, channel types, and production shape.
- [Configuration reference](/docs/reference/configuration): Every config key, its default, and what it controls.
- [Troubleshooting](/docs/getting-started/troubleshooting): Diagnose connection, auth, publish, fanout, recovery, and push problems.
---
# Overview (https://sockudo.io/docs/getting-started/overview)
> Understand Sockudo's building blocks, its two wire protocols, channel types, and the path from a local server to a clustered production deployment.
Sockudo is a Rust realtime server that exposes the Pusher Channels protocol while adding a native Sockudo protocol for features that do not fit inside the Pusher compatibility contract. This page gives you the mental model behind the rest of the documentation: what the moving parts are, how the two protocols relate, and what a production deployment looks like. Read it before you install anything if you are new to realtime servers, or skim the tables if you are evaluating Sockudo against an existing stack.
## How a realtime server works
A realtime server keeps a persistent WebSocket connection open to every client so it can push data the moment something happens, instead of waiting for the client to poll. Your application never talks to those sockets directly. It sends an event to Sockudo over HTTP, and Sockudo delivers that event to every client subscribed to the target channel.
```mermaid
flowchart LR
A[Your backend] -->|HTTP publish| B[Sockudo]
B -->|WebSocket| C[Browser client]
B -->|WebSocket| D[Mobile client]
```
A few terms come up constantly:
- **App:** a tenant on the server, identified by an `id`. Clients connect with the app's public `key`; your backend signs requests with the app's `secret`, which never leaves trusted servers.
- **Channel:** a named stream, such as `orders` or `presence-room-42`. Clients subscribe to channels and receive the events published to them.
- **Event:** a named message on a channel, such as `order.created`, with a payload.
- **Socket ID:** the unique identifier Sockudo assigns to each connection when it opens. Auth signatures are bound to it.
- **Presence:** a channel type that tracks which users are subscribed and notifies members when someone joins or leaves.
## Mental model
Sockudo has four major surfaces:
| Surface | Purpose | Where to start |
| --- | --- | --- |
| WebSocket server | Accepts client connections, subscriptions, auth results, presence transitions, and realtime events. | [First connection](/docs/getting-started/first-connection) |
| HTTP API | Lets trusted backends publish events, inspect channels, read history, mutate messages, and manage push. | [HTTP API](/docs/server/http-api) |
| Client SDKs | Browser, mobile, desktop, .NET, and Python realtime clients that subscribe and react to events. | [Realtime Clients](/docs/clients) |
| Server SDKs | Backend libraries that sign auth responses, publish events, validate webhooks, and call REST endpoints. | [Server SDKs](/docs/server-sdks) |
The WebSocket server and the HTTP API run in the same process and share one port (`6001` by default). Prometheus metrics are served separately on port `9601`.
## Pick your path
| You are building | Start with | You should finish with |
| --- | --- | --- |
| A Pusher replacement | [Migration](/docs/getting-started/migration) and [Protocol reference](/docs/reference/protocol) | Existing clients connecting through Protocol V1 with unchanged auth and publish semantics. |
| A new realtime product | [First connection](/docs/getting-started/first-connection) and [Authentication](/docs/getting-started/authentication) | Public, private, and presence channels wired through client and server SDKs. |
| A clustered production deployment | [Deployment](/docs/deployment), [Scaling](/docs/server/scaling), and [Observability](/docs/server/observability) | Load-balanced nodes, shared adapter, health probes, metrics, and dependency-specific alerts. |
| A durable collaboration or AI workflow | [Protocol V2](/docs/clients/protocol-v2), [History and recovery](/docs/server/history-recovery), and [Mutable messages](/docs/server/mutable-messages) | Serial continuity, message IDs, rewind, annotations, and safe retry behavior. |
## Protocol modes
Every connection uses exactly one protocol version, chosen by the client when it connects. The client sends a `protocol` query parameter on the WebSocket URL: `protocol=2` selects Protocol V2, and any other value, including the `protocol=7` that `pusher-js` sends, selects Protocol V1.
| | Protocol V1 | Protocol V2 |
| --- | --- | --- |
| Purpose | Pusher compatibility | Sockudo-native features |
| Selected by | Default | `protocolVersion: 2` in Sockudo client SDKs |
| System event prefixes | `pusher:` and `pusher_internal:` | `sockudo:` and `sockudo_internal:` |
| Works with | `pusher-js`, Laravel Echo, Pusher server SDKs | Sockudo client SDKs |
Protocol V1 is the compatibility layer. Use it when you want existing `pusher-js`, Laravel Echo, or Pusher-shaped server SDK integrations to keep working with minimal changes.
Protocol V2 is the native layer. Use it when your application needs one or more of:
- connection recovery with `stream_id` and `serial`
- `message_id` on broadcasts
- subscribe-time rewind
- delta compression and conflation
- server-side tag filtering
- durable channel history
- versioned mutable messages
- annotations for reactions, receipts, moderation, and summaries
- push notification helper workflows
Three V2 fields carry most of these guarantees. `message_id` is a stable identity for each broadcast, so clients can drop duplicates. `serial` is the message's position in its stream, so a client knows exactly where it left off. `stream_id` is an opaque identifier for the stream those serials belong to. Together they let a reconnecting client ask for "everything after this point" and get a definitive answer.
```mermaid
flowchart TB
A[Existing Pusher clients] --> B[Protocol V1]
C[New Sockudo clients] --> D[Protocol V2]
B --> E{Need durable semantics?}
E -->|No| F[Keep V1 stable]
E -->|Yes| G[Move channels to V2]
D --> G
G --> H[Recovery, rewind, deltas]
```
Adopt V2 by channel or client cohort. Keeping V1 clients stable while enabling V2 for new experiences is the safest migration shape.
V1 and V2 clients can subscribe to the same channel. Sockudo strips V2-only metadata from V1 deliveries, so adding V2 clients does not change the payloads your existing Pusher clients receive.
## Channel types
Sockudo follows Pusher channel naming rules for public, private, presence, and encrypted channels. The prefix of the channel name decides whether a subscription needs approval from your backend.
| Channel | Prefix | Auth required | Typical use |
| --- | --- | --- | --- |
| Public | none | No | dashboards, public feeds, unauthenticated status |
| Private | `private-` | Yes | account data, order updates, team-only streams |
| Presence | `presence-` | Yes with user data | rooms, collaboration, online member lists |
| Encrypted | `private-encrypted-` | Yes with shared secret | payloads that should stay opaque to the transport |
[Authentication](/docs/getting-started/authentication) explains how your backend approves private, presence, and encrypted subscriptions.
## Production shape
A production deployment usually has:
1. One or more Sockudo nodes behind a load balancer.
2. A shared adapter such as Redis, Redis Cluster, NATS, Kafka, RabbitMQ, Pulsar, Google Pub/Sub, or Iggy, so an event published on one node reaches subscribers on every node.
3. A persistent app manager such as PostgreSQL, MySQL, DynamoDB, ScyllaDB, or SurrealDB when apps are not configured statically in the config file.
4. A cache and queue backend for rate limits, idempotency, webhooks, and optional push workflows.
5. Prometheus scraping and alerting for connection, fanout, push, history, and adapter health.
The default `local` adapter only delivers events to clients connected to the same node. Running two nodes with the `local` adapter behind a load balancer silently splits your subscribers: each publish reaches only the clients on the node that received it. Configure a shared adapter before you scale out.
Backend drivers are Cargo features. The prebuilt Linux binaries and container images ship with every production backend; source builds include only the features you select. See [Installation](/docs/getting-started/installation#feature-flags).
## What to read next
- [Install Sockudo](/docs/getting-started/installation): Docker, Cargo, feature flags, and local configuration.
- [Connect a client](/docs/getting-started/first-connection): Subscribe to a channel and publish from a backend.
- [Add authentication](/docs/getting-started/authentication): Private, presence, encrypted, and user authentication.
- [Migrate from Pusher](/docs/getting-started/migration): Compatibility notes and cutover strategy.
- [Deploy to production](/docs/deployment): Static configuration, Linux, Docker, Kubernetes, cloud platforms, and capacity planning.
---
# Installation (https://sockudo.io/docs/getting-started/installation)
> Run Sockudo with Docker, a prebuilt Linux binary, Cargo, or Helm, choose the right feature set, and verify the server with health probes.
This page gets a Sockudo server running and proves it is healthy. Pick the install path that matches where you are: a container for the quickest local evaluation, a prebuilt binary for Linux hosts, Cargo for other platforms or custom feature sets, and Helm for Kubernetes. Every path ends at the same [verification](#verify) step.
## Choose an install path
The paths differ mainly in which backends are compiled in. Sockudo uses Cargo features to include drivers such as Redis or PostgreSQL, so a binary can only use the backends it was built with.
| Path | Best for | Features included | Needs a Rust toolchain |
| --- | --- | --- | --- |
| [Container image](#pull-a-container-image) | Local evaluation, containers, CI | `full` and `monolith` | No |
| [Docker Compose](#docker-compose) | Local development with Redis from a repository checkout | `full` and `monolith` (built from the repository Dockerfile) | No |
| [Linux binary](#install-a-linux-binary) | VMs and bare-metal Linux hosts | `full` | No |
| [crates.io](#install-from-cratesio) | macOS, Windows, or other platforms | Defaults (`local`, `v2`, `opentelemetry`) unless you add `--features` | Yes |
| [Source](#run-from-source) | Contributors and custom feature sets | Whatever you select | Yes |
| [Helm](#kubernetes-with-helm) | Kubernetes clusters | Same as the container image | No |
All paths serve the WebSocket server and HTTP API on port `6001` and Prometheus metrics on port `9601` by default.
## Pull a container image
Sockudo publishes multi-architecture images to GitHub Container Registry and Docker Hub. GHCR is the
primary registry; Docker Hub mirrors the same release tags.
```bash
docker pull ghcr.io/sockudo/sockudo:latest
docker pull sockudo/sockudo:latest
```
Use versioned tags for production rollouts:
```bash
docker pull ghcr.io/sockudo/sockudo:5.2.0
docker pull sockudo/sockudo:5.2.0
```
Start a local container with an in-memory app and Prometheus metrics exposed:
```bash
docker run --rm --name sockudo \
-p 6001:6001 \
-p 9601:9601 \
-e HOST=0.0.0.0 \
-e PORT=6001 \
-e METRICS_PORT=9601 \
-e METRICS_ENABLED=true \
-e SOCKUDO_DEFAULT_APP_ID=demo-app \
-e SOCKUDO_DEFAULT_APP_KEY=demo-key \
-e SOCKUDO_DEFAULT_APP_SECRET=demo-secret \
ghcr.io/sockudo/sockudo:latest
```
| Variable | What it does |
| --- | --- |
| `HOST` | Interface the WebSocket and HTTP server binds to. `0.0.0.0` is required inside a container. |
| `PORT` | Port for the WebSocket server and HTTP API. Defaults to `6001`. |
| `METRICS_PORT` | Port for the Prometheus endpoint. Defaults to `9601`. |
| `METRICS_ENABLED` | Turns the Prometheus endpoint on or off. |
| `SOCKUDO_DEFAULT_APP_ID`, `SOCKUDO_DEFAULT_APP_KEY`, `SOCKUDO_DEFAULT_APP_SECRET` | Define one app from the environment. When any of these is set, it replaces the apps defined inline in the config file. |
The guides in this documentation use `app-id`, `app-key`, and `app-secret`. Set the three `SOCKUDO_DEFAULT_APP_*` variables to those values if you want to copy examples without editing them.
Mount a checked-in configuration file when you need the same settings locally, in CI, and in
production:
```bash
docker run --rm --name sockudo \
-p 6001:6001 \
-p 9601:9601 \
-v "$PWD/config/config.toml:/app/config/config.toml:ro" \
ghcr.io/sockudo/sockudo:latest \
sockudo --config /app/config/config.toml
```
## Docker Compose
Use Compose from a repository checkout when you want Redis and repeatable local dependencies:
```bash
git clone https://github.com/sockudo/sockudo.git
cd sockudo
docker compose up sockudo redis
```
The Compose file builds the image from the repository's `Dockerfile`, so the first `up` compiles Sockudo inside Docker. You do not need a local Rust toolchain, but expect the first build to take several minutes.
The default server listens on:
| Service | URL |
| --- | --- |
| HTTP and WebSocket API | `http://127.0.0.1:6001` |
| Prometheus metrics | `http://127.0.0.1:9601/metrics` |
The Compose service runs the Redis adapter, cache, queue, and rate limiter, and defines its app through environment variables:
| Setting | Value under Compose |
| --- | --- |
| App ID | `demo-app` |
| App key | `demo-key` |
| App secret | `demo-secret` |
| Client events | Enabled |
| User authentication | Disabled |
`docker-compose.yml` sets `SOCKUDO_DEFAULT_APP_ID=demo-app`, which replaces the `app-id` app from the config file. Use `demo-app`, `demo-key`, and `demo-secret` with Compose, or edit those variables to `app-id`, `app-key`, and `app-secret` to match the rest of the documentation.
## Install a Linux binary
The installer supports x86_64 and ARM64 Linux, detects GNU libc or musl, and verifies the release
archive against its published SHA-256 checksum. It installs to `~/.local/bin` by default.
```bash
curl --proto '=https' --tlsv1.2 -sSfL \
https://sockudo.io/install.sh | sh
```
`sockudo.io/install.sh` is the stable public URL. It redirects to the installer attached to the
latest GitHub release; the installer then downloads and verifies the selected version from GitHub.
On success it prints the installed version and path, and reminds you to add the directory to `PATH`
if it is not already there.
Pin the binary version in deployments:
```bash
curl --proto '=https' --tlsv1.2 -sSfL \
https://sockudo.io/install.sh \
| sh -s -- --version 5.2.0
```
Choose another writable installation directory with `--bin-dir`:
```bash
curl --proto '=https' --tlsv1.2 -sSfL \
https://sockudo.io/install.sh \
| sh -s -- --bin-dir "$HOME/bin"
```
| Installer option | Environment variable | What it does |
| --- | --- | --- |
| `--version ` | `SOCKUDO_VERSION` | Installs a specific release instead of the latest one. |
| `--bin-dir ` | `SOCKUDO_INSTALL_DIR` | Installs into this directory instead of `~/.local/bin`. |
Run the installed binary with a local configuration file:
```bash
sockudo --config ./config/config.toml
```
Only Linux binaries are published, and they are built with the `full` feature set. `cargo binstall sockudo` remains available for the same Linux
release assets; use a source build on macOS, Windows, and other platforms.
## Install from crates.io
Use `cargo install` when you prefer building the published crate on the target host or need a
platform without a prebuilt binary:
```bash
cargo install sockudo --locked
```
Pin a release or enable the same production-oriented features you would use from source:
```bash
cargo install sockudo --version 5.2.0 --locked
cargo install sockudo --locked --features "redis,postgres,push"
```
This path compiles on the machine where you run it, so it needs a Rust toolchain and any native
libraries required by the selected storage or adapter features.
A plain `cargo install sockudo` builds only the default features. It runs a single node with in-memory backends, and a config that selects `driver = "redis"` or `driver = "postgres"` needs a binary built with those features. Add every backend you plan to configure to `--features`.
## Run from source
The repository is a Cargo workspace with several binaries, so select the server package with `-p sockudo`:
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
git clone https://github.com/sockudo/sockudo.git
cd sockudo
cargo run --release -p sockudo
```
Run from the repository root, the server loads `config/config.toml`, which defines the `app-id` development app used throughout the guides. The compiled binary is written to `target/release/sockudo`.
The default feature set is intentionally local-friendly. Add only the backends your deployment needs:
```bash
cargo build --release -p sockudo --features "redis,postgres"
cargo build --release -p sockudo --features "redis-cluster,mysql"
cargo build --release -p sockudo --features "nats,postgres"
cargo build --release -p sockudo --features "kafka,postgres"
cargo build --release -p sockudo --features "iggy,postgres"
cargo build --release -p sockudo --features full
```
## Feature flags
| Feature | Enables |
| --- | --- |
| `local` | Default marker for the single-node development profile. The in-memory app manager, cache, queue, adapter, and rate limiter are always compiled in. |
| `v2` | Sockudo-native protocol features: `recovery`, `delta`, and `tag-filtering`. Enabled by default. |
| `recovery` | Serial continuity, `message_id`, replay buffers, resume events, and rewind. |
| `delta` | Fossil and Xdelta3/VCDIFF delta compression for V2 clients. |
| `tag-filtering` | Server-side tag filter expressions for V2 subscriptions. |
| `opentelemetry` | OTLP traces, metrics, and logs export. Enabled by default. |
| `redis` | Redis adapter, cache, queue, and rate limiter support. |
| `redis-cluster` | Cluster-aware Redis transport and adapter support. |
| `nats`, `kafka`, `rabbitmq`, `pulsar`, `google-pubsub`, `iggy` | Horizontal transport adapters. |
| `mysql`, `postgres`, `dynamodb`, `scylladb`, `surrealdb` | Persistent app manager backends, plus the matching durable history and push storage. |
| `push` | The push notification platform. Provider support is added with `push-fcm`, `push-apns`, `push-webpush`, `push-hms`, and `push-wns`. |
| `versioned-messages` | Versioned mutable messages. |
| `ai-transport` | AI Transport. Also enables `versioned-messages`. |
| `full` | All production backends and optional integrations. |
## Minimal Pusher-compatible build
If you only need a small Pusher-compatible server, build without default features:
```bash
cargo build --release -p sockudo --no-default-features
```
Add V2 features explicitly when needed:
```bash
cargo build --release -p sockudo --no-default-features --features "recovery,delta,tag-filtering"
```
## Configuration file
Sockudo prefers TOML configuration and resolves settings in this order:
1. If you pass `--config `, Sockudo loads that file. If it cannot be read or parsed, the server exits instead of falling back to defaults.
2. Without `--config`, Sockudo looks for `config/config.toml`, then `config/config.json`, relative to the working directory. If neither exists, it starts from built-in defaults.
3. Environment variables are applied on top, so `PORT`, `ADAPTER_DRIVER`, `SOCKUDO_DEFAULT_APP_ID`, and similar variables override the file.
Config files can reference the environment with `${VAR}` or `${VAR:-default}` placeholders, which keeps secrets out of checked-in files. A placeholder whose variable is unset and has no default is reported as an error at startup.
```toml title="config/config.toml"
port = 6001
host = "0.0.0.0"
debug = false
[app_manager]
driver = "memory"
[app_manager.array]
[[app_manager.array.apps]]
id = "app-id"
key = "app-key"
secret = "app-secret"
enabled = true
[app_manager.array.apps.policy.limits]
max_connections = 10000
[app_manager.array.apps.policy.features]
enable_client_messages = false
```
| Key | Default | What it does |
| --- | --- | --- |
| `port` | `6001` | Port for the WebSocket server and HTTP API. |
| `host` | `0.0.0.0` | Interface the server binds to. |
| `debug` | `false` | Turns on verbose debug-level logging. Keep it off in production. |
| `app_manager.driver` | `memory` | Where apps are stored. `memory` reads them from `app_manager.array.apps` in this file. |
| `apps.id`, `apps.key`, `apps.secret` | none | The app's identity. Clients connect with `key`; backends sign with `secret`. |
| `apps.enabled` | none | A disabled app rejects new WebSocket connections. |
| `apps.policy.limits.max_connections` | none | Maximum concurrent connections for this app. |
| `apps.policy.features.enable_client_messages` | `false` | Allows clients to send `client-` events directly to other subscribers on private and presence channels. |
For exact loading precedence, equivalent JSON, secret injection, and clustered examples, see
[Static configuration](/docs/deployment/static-configuration). Every key is listed in the [Configuration reference](/docs/reference/configuration).
## Kubernetes with Helm
The Helm chart is published alongside each release, and its version matches the Sockudo release version.
```bash
helm install sockudo oci://ghcr.io/sockudo/charts/sockudo --version 5.2.0 \
--set config.adapterDriver=redis \
--set redis.host=redis-master \
--set autoscaling.enabled=true \
--set pdb.enabled=true \
--set ingress.enabled=true \
--set serviceMonitor.enabled=true
```
| Value | What it does |
| --- | --- |
| `config.adapterDriver` | Adapter used to fan out between pods. Use a shared adapter such as `redis` for more than one replica. |
| `redis.host` | Redis host used by the Redis adapter and backends. |
| `autoscaling.enabled` | Creates a HorizontalPodAutoscaler. |
| `pdb.enabled` | Creates a PodDisruptionBudget so voluntary disruptions keep pods available. |
| `ingress.enabled` | Creates an Ingress for external traffic. |
| `serviceMonitor.enabled` | Creates a Prometheus Operator ServiceMonitor for metrics scraping. |
Use Kubernetes when you need autoscaling, service monitors, disruption budgets, secret-backed app credentials, and cluster-level rollout controls. Continue with the production
[Kubernetes and Helm guide](/docs/deployment/kubernetes).
## Verify
Check the health probe and the metrics endpoint:
```bash
curl -f http://127.0.0.1:6001/up
curl -f http://127.0.0.1:9601/metrics | head
```
A healthy server answers `/up` with HTTP `200` and the body `OK`, and the metrics endpoint returns Prometheus text whose series start with `sockudo_`:
```text
OK
# HELP sockudo_http_received_bytes Total amount of bytes that sockudo's REST API received
# TYPE sockudo_http_received_bytes counter
```
Sockudo exposes several probes on the main port. Use them for load balancers and orchestrators:
| Probe | Healthy response | What it checks |
| --- | --- | --- |
| `/up` | `200 OK` | The app manager responds and at least one app is configured, plus dependency health. Returns `503 ERROR` on failure and `503 DRAINING` during shutdown. A `200 DEGRADED` body means the server is up with a degraded dependency. |
| `/up/{app_id}` | `200 OK` | The same checks for one app. Returns `404 NOT_FOUND` for an unknown app ID. |
| `/ready` | `200 OK` | The `/up` checks plus capacity. Returns `503 CAPACITY` when the node should stop receiving new connections. |
| `/live` | `200` | The process is running. |
| `/accept-traffic` | `200 ACCEPTING` | Admission for new connections. Returns `503 MEMORY_PRESSURE` or `503 DRAINING`. |
`curl http://127.0.0.1:6001/up/app-id` confirms that a specific app exists. A `404` usually means the server loaded different credentials than you expected, for example `demo-app` under Docker Compose.
Started without a config file and without the `SOCKUDO_DEFAULT_APP_*` variables, Sockudo has no apps and `/up` returns `503 ERROR`. Define at least one app before pointing a load balancer at the server.
When health is green, continue with [First connection](/docs/getting-started/first-connection). Before
shipping, follow the [Deployment guide](/docs/deployment).
## Next steps
- [First connection](/docs/getting-started/first-connection): Subscribe a client and publish your first event.
- [Static configuration](/docs/deployment/static-configuration): Loading precedence, JSON equivalents, and secret injection.
- [Environment variables](/docs/reference/environment-variables): Every variable Sockudo reads and what it overrides.
---
# First connection (https://sockudo.io/docs/getting-started/first-connection)
> Connect a realtime client, subscribe to a channel, publish the first event from a trusted backend, and confirm delivery end to end.
This guide proves the full loop: Sockudo accepts a WebSocket connection, the client subscribes to a channel, and your backend publishes an event over the HTTP API that the client receives. Once this loop works, every other feature (auth, presence, Protocol V2, history, push) is an addition on top of it, so it is the best place to start debugging too.
## How the loop works
Clients and backends never talk to each other directly. The client holds a WebSocket connection to Sockudo; the backend sends a signed HTTP request; Sockudo delivers the event to every subscriber of the channel.
```mermaid
sequenceDiagram
participant C as Client
participant S as Sockudo
participant B as Backend
C->>S: Open WebSocket /app/app-key
S-->>C: connection_established with socket_id
C->>S: subscribe to orders
S-->>C: subscription_succeeded
B->>S: POST /apps/app-id/events (signed)
S-->>B: 200 with ok true
S-->>C: order.created event
```
The client connects with the app **key**, which is public. The backend signs its request with the app **secret**, which must stay on trusted servers.
## Run the loop
1. **Start Sockudo.** Run the container with the development app the guides use:
```bash
docker run --rm --name sockudo \
-p 6001:6001 -p 9601:9601 \
-e SOCKUDO_DEFAULT_APP_ID=app-id \
-e SOCKUDO_DEFAULT_APP_KEY=app-key \
-e SOCKUDO_DEFAULT_APP_SECRET=app-secret \
ghcr.io/sockudo/sockudo:latest
```
Confirm the app exists. The command prints `OK`:
```bash
curl -s http://127.0.0.1:6001/up/app-id
```
2. **Install the SDKs.** `@sockudo/client` is the realtime client; `sockudo` is the Node.js server SDK.
```bash
npm install @sockudo/client sockudo
```
3. **Subscribe from a client.** Run the [client code](#subscribe-from-javascript) in a browser app or a Node.js script and leave it running.
4. **Publish from your backend.** Run the [publish script](#publish-from-nodejs) in a second terminal.
5. **Check the client output.** The client logs the event as soon as Sockudo accepts the publish.
## Subscribe from JavaScript
```ts title="client.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
enabledTransports: ["ws"],
protocolVersion: 2,
});
client.connection.bind("connected", () => {
console.log("connected as", client.connection.socket_id);
});
const channel = client.subscribe("orders");
channel.bind("order.created", (payload) => {
console.log("received", payload);
});
```
| Option | What it does |
| --- | --- |
| `wsHost`, `wsPort` | Where the Sockudo server listens. The client requires either `wsHost` or `cluster`. |
| `forceTLS` | Use `wss://`. Set it to `false` for a local server without TLS. |
| `enabledTransports` | Limits the client to plain WebSockets, which keeps local tests predictable. |
| `protocolVersion` | `2` selects Sockudo's native Protocol V2. Omit it to connect with Protocol V1, the Pusher-compatible default. |
Protocol V1 clients use the same server with Pusher-compatible options:
```ts title="pusher-client.ts"
const pusher = new Pusher("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
cluster: "local",
enabledTransports: ["ws"],
});
pusher.subscribe("orders").bind("order.created", console.log);
```
`pusher-js` expects a `cluster` value, but the explicit `wsHost` decides where it connects.
## Publish from Node.js
```ts title="publish.ts"
const sockudo = new Sockudo({
appId: "app-id",
key: "app-key",
secret: "app-secret",
host: "127.0.0.1",
port: 6001,
useTLS: false,
});
await sockudo.trigger("orders", "order.created", {
id: "ord_123",
total: 42_00,
});
```
The client prints:
```text
connected as 1234567890.1234567890
received { id: 'ord_123', total: 4200 }
```
The socket ID differs on every connection.
## Publish over raw HTTP
Server SDKs are preferred because they handle signing, but the underlying API is small. Every HTTP API request carries Pusher-style HMAC authentication in its query string; an unsigned request is rejected:
```bash
curl -s -X POST "http://127.0.0.1:6001/apps/app-id/events" \
-H "Content-Type: application/json" \
-d '{"name": "order.created", "channel": "orders", "data": {"id": "ord_123", "total": 4200}}'
```
```json
{"error":"Missing authentication query parameters","code":"invalid_input","status":400}
```
To sign a request, compute an HMAC-SHA256 with the app secret over the method, the path, and the sorted query parameters, then append it as `auth_signature`. POST requests with a body must include the body's MD5 hash as `body_md5`:
```bash title="publish.sh"
APP_ID=app-id
APP_KEY=app-key
APP_SECRET=app-secret
BODY='{"name":"order.created","channel":"orders","data":"{\"id\":\"ord_123\",\"total\":4200}"}'
TIMESTAMP=$(date +%s)
BODY_MD5=$(printf '%s' "$BODY" | openssl dgst -md5 | sed 's/^.* //')
QUERY="auth_key=${APP_KEY}&auth_timestamp=${TIMESTAMP}&auth_version=1.0&body_md5=${BODY_MD5}"
SIGNATURE=$(printf 'POST\n/apps/%s/events\n%s' "$APP_ID" "$QUERY" \
| openssl dgst -sha256 -hmac "$APP_SECRET" | sed 's/^.* //')
curl -s -X POST "http://127.0.0.1:6001/apps/${APP_ID}/events?${QUERY}&auth_signature=${SIGNATURE}" \
-H "Content-Type: application/json" \
-d "$BODY"
```
A successful publish returns:
```json
{"ok":true}
```
Sockudo rejects a signed request whose `auth_timestamp` is more than 600 seconds away from the server clock. Generate the timestamp at request time and keep server clocks synchronized with NTP.
## What is on the wire
Seen from a raw WebSocket, the Protocol V1 exchange for this guide looks like this:
```json
{"event":"pusher:connection_established","data":"{\"socket_id\":\"9501381246.8633737996\",\"activity_timeout\":120}"}
{"event":"pusher_internal:subscription_succeeded","channel":"orders","data":"{}"}
{"event":"order.created","channel":"orders","data":"{\"id\":\"ord_123\",\"total\":4200}"}
```
Protocol V2 connections use the same flow with `sockudo:` and `sockudo_internal:` prefixes, and V2 messages can carry extra fields such as `message_id`. Client SDKs handle these frames for you; see the [Protocol reference](/docs/reference/protocol) when you need the details.
## Use idempotency for retries
When a backend may retry a publish after a timeout, attach an idempotency key. Sockudo deduplicates matching keys inside the configured window: the first request is broadcast, and a repeat with the same key returns the stored response without broadcasting again.
```ts
await sockudo.trigger(
"orders",
"order.created",
{ id: "ord_124" },
{ idempotency_key: "order-created-ord_124" },
);
```
| Setting | Default | What it does |
| --- | --- | --- |
| `[idempotency] enabled` | `true` | Turns publish deduplication on or off. |
| `[idempotency] ttl_seconds` | `120` | How long a key is remembered. Apps can override it in their `policy.idempotency` settings. |
| `[idempotency] max_key_length` | `128` | Longest accepted key. |
Raw HTTP clients can send the key as the `idempotency_key` body field or the `X-Idempotency-Key` header. The Node SDK can also generate keys for you with the `autoIdempotencyKey: true` constructor option.
Reusing a key with a different payload returns HTTP `400`. Derive keys from the business operation, such as `order-created-ord_124`, not from a random value per attempt, or retries will not be recognized as duplicates.
Keys are stored in the configured cache backend. With the in-memory cache each node remembers only its own keys, so use a shared cache such as Redis when several nodes receive publishes.
## Add push notification fanout
Realtime events and push notifications are complementary. Use WebSockets for connected clients, and push for devices that are offline, backgrounded, or outside the active channel session.
```ts
const accepted = await sockudo.publishPush({
recipients: [{ type: "channel", channel: "orders" }],
payload: {
title: "Order created",
body: "Order ord_124 is ready for review",
},
});
console.log(accepted.publish_id ?? accepted.publishId);
```
`publishPush` always submits the publish asynchronously. Sockudo answers `202 Accepted` with a publish ID as soon as the request is admitted; delivery to providers happens afterwards. Track progress with `sockudo.getPublishStatus(publishId)`.
Push requires a server built with the `push` feature (included in the prebuilt binaries and images) and configured provider credentials. Set it up with [Push notifications](/docs/server/push-notifications).
## Expected result
1. The client logs `received { id: 'ord_123', total: 4200 }`.
2. The HTTP publish returns `{"ok":true}`.
3. If push is configured, the push publish returns a publish ID.
4. Metrics under `/metrics` show connection and publish counters moving.
Check the counters directly:
```bash
curl -s http://127.0.0.1:9601/metrics \
| grep -E '^sockudo_(connected|new_connections_total|http_calls_received_total)'
```
`sockudo_connected` reports open sockets, `sockudo_new_connections_total` counts connection attempts, and `sockudo_http_calls_received_total` counts HTTP API calls, each labeled by `app_id`.
## Debug checkpoints
Use this table when the first loop does not behave as expected.
| Symptom | Check | Fix |
| --- | --- | --- |
| WebSocket connection fails | Browser devtools shows a failed request to `/app/app-key`. | Confirm `wsHost`, `wsPort`, `forceTLS`, and `enabledTransports` match the local server. |
| Connection closes immediately | The server may not know the app key. `curl http://127.0.0.1:6001/up/app-id` returns `404` for an unknown app. | Use the credentials the server actually loaded. Docker Compose uses `demo-app`, `demo-key`, and `demo-secret`. |
| Subscribe succeeds but no event arrives | The backend publish may target a different app, channel, or event name. | Reuse the same `app-key`, `app-id`, channel, and event name from the snippets above. |
| The publishing tab never sees its own event | Publishing with a `socket_id` parameter excludes that connection from delivery. | Omit `socket_id` while testing, or listen from a second client. |
| Private or presence channels reject auth | Auth signatures are generated on a trusted backend and include the exact socket ID and channel name. | Use a server SDK for channel auth before writing custom signing code. |
| Retried publishes create duplicates | The request does not include a stable idempotency key. | Derive keys from business identifiers such as `order-created-ord_123`. |
| Metrics do not move | You may be checking the wrong process, port, or app credentials. | Query `/up`, then scrape `:9601/metrics` while publishing a fresh event. |
When debugging compatibility, first prove the same event works with a public channel. Then add private auth, presence user data, Protocol V2 options, history, push, or mutations one feature at a time.
## Next steps
- [Add authentication](/docs/getting-started/authentication): Protect private, presence, and encrypted channels with your backend.
- [JavaScript client](/docs/clients/javascript): Every client option, runtime entrypoint, and recovery feature.
- [Node.js server SDK](/docs/server-sdks/node): Publishing, batching, channel state, history, and webhooks from Node.js.
---
# Authentication (https://sockudo.io/docs/getting-started/authentication)
> Protect private, presence, and encrypted channels, identify users, and issue Protocol V2 capability tokens from a trusted backend.
Sockudo never trusts client-provided access to protected channels. Clients request a subscription, your application server decides whether the user is allowed, then the server returns a signed response. This page covers every way your backend grants access, from Pusher-compatible channel authorization to Protocol V2 capability tokens. You need it as soon as you use a channel whose name starts with `private-` or `presence-`, or when you want Sockudo to know which user owns a connection.
## Who holds which credential
Every app has three credentials, and where they live is the foundation of Sockudo's security model:
| Credential | Visible to clients | Used for |
| --- | --- | --- |
| App ID | Yes | Identifies the app in HTTP API paths such as `/apps/app-id/events`. |
| App key | Yes | Identifies the app on the WebSocket URL. Shipping it in browser and mobile code is expected. |
| App secret | **Never** | Signs channel authorization, user authentication, capability tokens, HTTP API requests, and webhooks. |
Anything signed with the secret proves that your backend approved it. That is why every mechanism below runs on a server you control.
| Mechanism | Use it when | Protocol |
| --- | --- | --- |
| [Channel authorization](#channel-authorization-flow) | A client subscribes to a `private-`, `presence-`, or `private-encrypted-` channel. | V1 and V2 |
| [User authentication](#user-authentication) | You want Sockudo to know which user owns a connection, for user-targeted events and watchlists. | V1 and V2 |
| [Capability tokens](#protocol-v2-capability-tokens) | A V2 client should carry a short-lived, scoped grant for the whole connection. | V2 only |
| HTTP API signing | Your backend publishes or queries the HTTP API. Server SDKs do this for you. | Both |
## Channel authorization flow
1. Client subscribes to `private-orders` or `presence-room`.
2. The client SDK sends `socket_id` and `channel_name` to your auth endpoint.
3. Your backend validates the current user and channel policy.
4. A server SDK signs the response using the app secret.
5. Sockudo validates the signature and completes the subscription.
```mermaid
sequenceDiagram
participant C as Client
participant A as Your auth endpoint
participant S as Sockudo
C->>S: Connect and receive socket_id
C->>A: POST socket_id and channel_name
Note over A: Check session and channel policy
A-->>C: Signed auth response
C->>S: Subscribe with the signature
S-->>C: subscription_succeeded
```
The signature covers the exact `socket_id` and `channel_name`, plus the presence `channel_data` when present. A signature for one connection or channel cannot be replayed on another.
## Set up channel authorization
1. Add an auth endpoint to your backend that authenticates the user and signs the subscription with a server SDK.
2. Point the client's `channelAuthorization.endpoint` at it. The `@sockudo/client` default is `/sockudo/auth`.
3. Subscribe to a `private-` or `presence-` channel and confirm the subscription succeeds.
## Node.js auth endpoint
The client SDK sends the auth request as an HTML form post (`application/x-www-form-urlencoded`), so parse URL-encoded bodies. `requireUser` and `canAccessChannel` stand in for your own session lookup and authorization rules.
```ts title="auth-server.ts"
const app = express();
app.use(express.urlencoded({ extended: false }));
const sockudo = new Sockudo({
appId: process.env.SOCKUDO_APP_ID!,
key: process.env.SOCKUDO_APP_KEY!,
secret: process.env.SOCKUDO_APP_SECRET!,
host: "127.0.0.1",
port: 6001,
useTLS: false,
});
app.post("/sockudo/auth", (req, res) => {
const { socket_id, channel_name } = req.body;
const user = requireUser(req);
if (!canAccessChannel(user, channel_name)) {
return res.status(403).json({ error: "Forbidden" });
}
if (channel_name.startsWith("presence-")) {
return res.json(
sockudo.authorizeChannel(socket_id, channel_name, {
user_id: user.id,
user_info: { name: user.name, role: user.role },
}),
);
}
return res.json(sockudo.authorizeChannel(socket_id, channel_name));
});
```
`authorizeChannel` returns a different shape depending on the channel type:
| Channel | Response body |
| --- | --- |
| `private-` | `{ "auth": "app-key:" }` |
| `presence-` | `{ "auth": "...", "channel_data": "" }` |
| `private-encrypted-` | `{ "auth": "...", "shared_secret": "" }` |
For presence channels, `user_id` identifies the member, and `user_info` is shown to every other member of the channel.
## Client configuration
```ts title="client.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
channelAuthorization: {
endpoint: "/sockudo/auth",
},
});
client.subscribe("private-orders");
```
`channelAuthorization` also accepts `headers` and `headersProvider` when your endpoint expects a bearer token instead of a session cookie.
`@sockudo/client` posts to `/sockudo/auth` by default. If your backend still serves the endpoint your Pusher client used, set `channelAuthorization.endpoint` explicitly.
## User authentication
User authentication signs a connection identity. It powers watchlists and user-targeted server events. The client calls `client.signin()`, which posts the `socket_id` to your user-auth endpoint (default `/sockudo/user-auth`):
```ts
app.post("/sockudo/user-auth", (req, res) => {
const user = requireUser(req);
res.json(
sockudo.authenticateUser(req.body.socket_id, {
id: user.id,
user_info: { name: user.name },
}),
);
});
```
The response contains `auth` and `user_data`. Once a connection is signed in, your backend can reach every connection of that user, for example with `sockudo.sendToUser(userId, event, data)`.
User authentication is off unless the app enables it, either with `enable_user_authentication = true` under the app's `policy.features` or with `SOCKUDO_DEFAULT_APP_ENABLE_USER_AUTHENTICATION=true` for an app defined through environment variables.
When user authentication is enabled for an app, Sockudo closes any connection that has not signed in within `user_authentication_timeout` seconds (default `3600`) with close code `4009`. Call `client.signin()` right after connecting on every client of that app.
## Protocol V2 capability tokens
Protocol V2 clients may authenticate the WebSocket connection with a short-lived capability token: a JWT, signed with the app secret, that lists which channels the connection may use and for which operations. Instead of one auth round trip per channel, the client presents one token for the whole connection and refreshes it before it expires.
Configure the client with `authUrl` (or an `authCallback` function for full control over the request):
```ts title="client.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
protocolVersion: 2,
authUrl: "/sockudo/token",
});
```
The client sends a JSON `POST` to `authUrl` with `socket_id` and `reason` (`initial`, `reconnect`, `refresh`, or `expired`), and expects either a token string or an object with a `token` field in response.
The client rejects `token`, `authUrl`, and `authCallback` at construction unless `protocolVersion` is `2`.
Your backend issues the JWT after authenticating the user. Never ship the app secret to browsers, mobile apps, or untrusted clients.
Node.js example:
```ts title="token-endpoint.ts"
app.post("/sockudo/token", express.json(), (req, res) => {
const user = requireUser(req);
const now = Math.floor(Date.now() / 1000);
const capability = {
"private-orders:*": ["subscribe", "history"],
"presence-orders:*": ["subscribe", "presence"],
"private-orders:input": ["publish"],
};
const token = jwt.sign(
{
"x-sockudo-capability": JSON.stringify(capability),
"x-sockudo-client-id": user.id,
iat: now,
exp: now + 3600,
jti: crypto.randomUUID(),
},
process.env.SOCKUDO_APP_SECRET!,
{
algorithm: "HS256",
header: { kid: process.env.SOCKUDO_APP_KEY! },
},
);
res.json({ token });
});
```
Rust example:
```rust
use jsonwebtoken::{encode, Algorithm, EncodingKey, Header};
use serde::Serialize;
use std::collections::BTreeMap;
#[derive(Serialize)]
struct Claims {
#[serde(rename = "x-sockudo-capability")]
capability: String,
#[serde(rename = "x-sockudo-client-id")]
client_id: String,
iat: i64,
exp: i64,
jti: String,
}
fn issue_token(app_key: &str, app_secret: &str, client_id: &str) -> anyhow::Result {
let now = chrono::Utc::now().timestamp();
let mut capability = BTreeMap::new();
capability.insert("private-orders:*", vec!["subscribe", "history"]);
let mut header = Header::new(Algorithm::HS256);
header.kid = Some(app_key.to_owned());
Ok(encode(
&header,
&Claims {
capability: serde_json::to_string(&capability)?,
client_id: client_id.to_owned(),
iat: now,
exp: now + 3600,
jti: uuid::Uuid::new_v4().to_string(),
},
&EncodingKey::from_secret(app_secret.as_bytes()),
)?)
}
```
Token rules:
- Header `kid` must equal the Sockudo app key and `alg` must be `HS256`.
- `x-sockudo-capability` is a JSON map from channel pattern to operations; its stringified form is
accepted for compatibility.
- Operations include `publish`, `subscribe`, `history`, and `presence`; AI Transport also accepts
the explicit annotation, message-mutation, and push operations documented in
[AI Transport authentication](/docs/server/ai-transport-authentication).
- Patterns are exact names, namespace prefixes such as `orders:*`, or `*`; matching is case-sensitive.
- `x-sockudo-client-id` is the verified identity used for presence and user-limited channels.
- `iat`, `exp`, and `jti` are required. Tokens may live at most 24 hours; 1 hour or less is recommended.
- Clients refresh in place with `sockudo:auth` carrying `{ "token": "..." }`. Expired tokens emit `sockudo:token_expired` with code `40142` and close after a 30 second grace window.
| Limit | Value |
| --- | --- |
| Token size | 8 KiB |
| Channel patterns per token | 100 |
| `x-sockudo-client-id` and `jti` length | 1 to 128 bytes |
| Maximum lifetime (`exp` minus `iat`) | 24 hours |
| Clock skew tolerated on `iat`, `exp`, and the optional `nbf` | 30 seconds |
`@sockudo/client` refreshes provider-backed tokens automatically at 80% of their lifetime, so a client configured with `authUrl` or `authCallback` keeps its connection without interruption. To cut access before a token expires, revoke it by `jti` or `client_id` with `POST /apps/{appId}/revocations`; see [Security](/docs/server/security).
## Encrypted channels
Encrypted channels start with `private-encrypted-`. Payloads are encrypted end to end: Sockudo relays ciphertext and never sees the key. Your backend holds a 32-byte encryption master key, and when it signs a subscription it returns a per-channel shared secret derived from that key.
Generate a master key once and store it with your other secrets:
```bash
openssl rand -base64 32
```
Pass it to the server SDK:
```ts
const sockudo = new Sockudo({
appId: "app-id",
key: "app-key",
secret: "app-secret",
host: "127.0.0.1",
port: 6001,
encryptionMasterKeyBase64: process.env.SOCKUDO_ENCRYPTION_MASTER_KEY!,
});
```
The same SDK instance encrypts the payload when you `trigger` to an encrypted channel. An encrypted publish must target a single channel; the Node SDK rejects a multi-channel trigger that includes one.
Clients must use the encryption-capable build of the SDK to decrypt payloads:
```ts
```
Keep the master key outside the browser. Client SDKs only receive the derived shared secret for authorized encrypted channels.
## Push authorization boundary
Push registration and push publish helpers are also authentication boundaries. Mobile and browser clients should call your backend proxy, not Sockudo directly with app secrets.
```ts
app.post("/sockudo/push", async (req, res) => {
const user = requireUser(req);
await assertDeviceBelongsToUser(user, req.body.device_id);
const response = await sockudo.publishPush({
recipients: [{ type: "client", client_id: user.id }],
payload: req.body.payload,
});
res.status(202).json(response);
});
```
## Security checklist
- Read `socket_id` and `channel_name` from the request body; do not accept them from query parameters unless your framework requires it.
- Authenticate the user before signing anything.
- Enforce channel ownership and tenant boundaries before calling the SDK signing helper.
- Return presence user data that is safe for all channel members to see.
- Keep app secrets, webhook secrets, push provider credentials, and encryption master keys server-side only.
- Issue capability tokens from a backend endpoint only; do not mint them in client code.
- Use short request timeouts and structured audit logs for denied auth attempts.
## Next steps
- [Security](/docs/server/security): Origin checks, rate limits, token revocation, and hardening.
- [Server SDKs](/docs/server-sdks): Channel auth and user auth helpers in every supported language.
- [Troubleshooting](/docs/getting-started/troubleshooting): Diagnose rejected subscriptions and signature mismatches.
---
# Migration (https://sockudo.io/docs/getting-started/migration)
> Move from Pusher Channels or Soketi to Sockudo in phases, keeping existing clients on Protocol V1 and adopting Protocol V2 where it pays off.
Sockudo can be adopted in phases. Start with Protocol V1 compatibility, then opt selected clients into Protocol V2 when you want native Sockudo features. This page walks through both phases, the configuration differences from a hosted Pusher account, and a cutover and rollback plan that keeps production traffic safe throughout.
## Migration phases
The two phases are independent. Many teams finish phase one, run on it for weeks, and only then move individual features to V2.
```mermaid
flowchart LR
A[Pusher or Soketi] --> B[Sockudo with V1]
B --> C[Verify parity]
C --> D[Selected clients on V2]
```
| Phase | What changes | What stays the same |
| --- | --- | --- |
| Compatibility (V1) | Client host, ports, and TLS settings; backend host and credentials. | Client libraries, channel names, event names, auth endpoints, payloads, and webhooks. |
| Native (V2) | Selected clients move to Sockudo SDKs with `protocolVersion: 2`. | Backend publish calls and channel naming. V1 clients keep working alongside V2 clients. |
## What maps directly
Sockudo implements the Pusher Channels protocol and HTTP API, so most concepts carry over unchanged:
| Pusher concept | In Sockudo |
| --- | --- |
| App ID, key, and secret | An app record in the app manager, with the same three values. |
| Cluster hostname | Your own hostname, set with `wsHost` on clients and `host` on server SDKs. |
| `pusher:` system events | Unchanged on Protocol V1. Protocol V2 uses `sockudo:` instead. |
| Channel prefixes `private-`, `presence-`, `private-encrypted-` | Same names, same auth rules. |
| Publish endpoints `/apps/{appId}/events` and `/apps/{appId}/batch_events` | Same paths and HMAC request signing. |
| Channel state `/apps/{appId}/channels` and `/apps/{appId}/channels/{channelName}/users` | Same paths. |
| Webhooks signed with `X-Pusher-Key` and `X-Pusher-Signature` | Same headers, so existing webhook validation keeps working. |
## Compatibility-first migration
For existing Pusher clients, point the host and ports at Sockudo:
```ts title="pusher-client.ts"
const pusher = new Pusher(process.env.NEXT_PUBLIC_PUSHER_KEY!, {
wsHost: "realtime.example.com",
wsPort: 80,
wssPort: 443,
forceTLS: true,
enabledTransports: ["ws", "wss"],
cluster: "local",
});
```
Backend SDKs continue to publish through Pusher-compatible REST endpoints. Change host, port, TLS, and credentials, then run your existing private and presence auth tests.
Laravel applications keep using Laravel Echo with the Pusher connector; [Laravel](/docs/server-sdks/laravel#connect-laravel-echo) shows the broadcaster and Echo configuration.
The app key in client code, the app ID and secret in backend code, and the app record in Sockudo must all describe the same app. Mirror your existing Pusher credentials into Sockudo first, and the only change clients need is the host.
## Native migration
After compatibility is stable, migrate clients to Sockudo SDKs and enable Protocol V2:
```ts title="client.ts"
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
protocolVersion: 2,
connectionRecovery: true,
});
```
Protocol V2 is the right choice for recovery, rewind, deltas, filters, versioned messages, annotations, and push helper workflows.
Under Protocol V2, system events use the `sockudo:` prefix. Code that binds `pusher:subscription_succeeded` or other `pusher:` events must bind the `sockudo:` equivalents once a client switches to `protocolVersion: 2`. Your own event names, such as `order.created`, do not change.
`connectionRecovery: true` lets the client resume after a disconnect, but Sockudo only keeps the replay buffer when `[connection_recovery] enabled = true` is set on the server. It is off in the built-in defaults; the sample `config/config.toml` turns it on.
## Server configuration differences
A hosted Pusher account manages apps, fanout, and storage for you. With Sockudo you choose a backend for each of those jobs. Sockudo prefers TOML:
```toml title="config/config.toml"
[app_manager]
driver = "pgsql"
[adapter]
driver = "redis"
[cache]
driver = "redis"
[queue]
driver = "redis"
```
| Section | Role during migration |
| --- | --- |
| `[app_manager]` | Stores the app IDs, keys, and secrets you mirror from Pusher. |
| `[adapter]` | Shares broadcasts and presence between Sockudo nodes. Required as soon as you run more than one node. |
| `[cache]` | Holds idempotency keys and other short-lived shared state. |
| `[queue]` | Runs background work such as webhook delivery. |
Keep one source of truth for app credentials. If you previously hardcoded credentials in application config, move them into Secret Manager, Kubernetes Secrets, or the persistent app manager.
## Push notification migration
If your old realtime stack only handled WebSockets, plan push as part of the migration instead of after it. Sockudo's push platform covers:
- device registration and activation
- provider credentials for FCM, APNs, Web Push, HMS, and WNS
- channel push subscriptions
- async publish admission
- scheduled push
- provider delivery status callbacks
- push metrics and capacity planning
Keep push provider credentials in Sockudo or your trusted backend. Clients should only receive short-lived activation or identity tokens. Start with [Push notifications](/docs/server/push-notifications).
## Cutover checklist
1. Run Sockudo next to the existing realtime provider.
2. Mirror app credentials into Sockudo.
3. Point staging clients at Sockudo in Protocol V1 mode.
4. Verify public, private, presence, encrypted, webhook, and push flows.
5. Enable metrics and alerting before production traffic.
6. Move a small tenant, region, or traffic shard first.
7. Migrate selected applications to Protocol V2 only after V1 parity is verified.
## Rollback
Rollback is simplest when client configuration is environment-driven. Read the realtime host and protocol version from your own application settings rather than hardcoding them:
```bash title=".env"
REALTIME_HOST=realtime.example.com
REALTIME_PROTOCOL_VERSION=1
REALTIME_FORCE_TLS=true
```
These are your application's variables, not Sockudo settings. Switching `REALTIME_HOST` back to the previous provider, or `REALTIME_PROTOCOL_VERSION` back to `1`, then becomes a configuration deploy instead of a code change.
Keep idempotency keys on backend publishes during cutover. If a retry crosses providers, use an application-level operation ID so downstream consumers can suppress duplicates.
## Next steps
- [Compatibility reference](/docs/reference/compatibility): Exactly what Protocol V1 and V2 support, and where they differ.
- [Protocol V2](/docs/clients/protocol-v2): Message IDs, serials, recovery, rewind, and V2 system events.
- [Deployment](/docs/deployment): Topologies, configuration, and rollout on your infrastructure.
---
# Troubleshooting (https://sockudo.io/docs/getting-started/troubleshooting)
> Diagnose connection, authentication, publish, cluster fanout, recovery, push notification, and webhook problems one boundary at a time.
Start with the path that failed: connection, subscription, publish, fanout, history, or push. Sockudo is easier to debug when each boundary is checked separately, because each one fails in its own recognizable way. Use the table below to jump to the right section, then work through its checks in order.
## Find your symptom
| Symptom | Likely boundary | Section |
| --- | --- | --- |
| The client never reaches `connected` | Network, TLS, proxy, or app key | [Connection fails](#connection-fails) |
| `private-` or `presence-` subscriptions are rejected | Your auth endpoint or signature | [Private or presence subscription fails](#private-or-presence-subscription-fails) |
| Backend publishes return `400` or `401` | HTTP API request signing | [HTTP API requests are rejected](#http-api-requests-are-rejected) |
| Publishes succeed but clients see nothing | Channel, protocol, or cluster fanout | [Events publish but clients do not receive them](#events-publish-but-clients-do-not-receive-them) |
| Reconnecting clients miss messages | Recovery configuration | [Recovery or rewind is incomplete](#recovery-or-rewind-is-incomplete) |
| Push is accepted but never arrives | Push providers and device state | [Push notification publish is accepted but nothing arrives](#push-notification-publish-is-accepted-but-nothing-arrives) |
| Webhook signatures fail | Raw request body handling | [Webhooks are rejected](#webhooks-are-rejected) |
## Connection fails
Verify the server is reachable and healthy. A healthy server prints `OK`:
```bash
curl -f http://127.0.0.1:6001/up
```
Then confirm the app your client uses exists. An unknown app ID returns `404`:
```bash
curl -s http://127.0.0.1:6001/up/app-id
```
Check the client host options:
```ts
new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
enabledTransports: ["ws"],
});
```
To take the client SDK out of the picture, open a raw WebSocket with Node.js 22 or later. A working server answers with `connection_established` right away:
```bash
node -e 'const ws = new WebSocket("ws://127.0.0.1:6001/app/app-key?protocol=7"); ws.onmessage = (m) => { console.log(m.data); ws.close(); };'
```
```json
{"event":"pusher:connection_established","data":"{\"socket_id\":\"9501381246.8633737996\",\"activity_timeout\":120}"}
```
Common causes:
- using `forceTLS: true` against an HTTP-only local server
- forgetting `enabledTransports: ["ws"]` in local browser tests
- proxying WebSockets without `Upgrade` and `Connection` headers
- sending clients to one host while HTTP publish uses another app environment
- using `app-key` against Docker Compose, whose app key is `demo-key`
- an app with `enabled = false`, which rejects new connections
When the server closes the socket, the close code tells you why. `4001` means the app does not exist, `4004` means the app is over its connection quota, and `4009` means the connection was unauthorized. The full list is in [Errors](/docs/reference/errors).
## Private or presence subscription fails
Inspect the auth endpoint response. Private auth must return an `auth` value. Presence auth must return both `auth` and `channel_data`.
```json
{
"auth": "app-key:signature",
"channel_data": "{\"user_id\":\"42\",\"user_info\":{\"name\":\"Ada\"}}"
}
```
If signatures fail, verify:
- `socket_id` is exact
- `channel_name` is exact
- app key and secret match the Sockudo app
- presence `channel_data` is serialized before signing
- framework body parsers are not dropping form fields
`@sockudo/client` posts `socket_id` and `channel_name` as a URL-encoded form to `/sockudo/auth` by default. If your framework only parses JSON bodies, both fields arrive empty and every signature is wrong.
If connections close with `4009` about an hour after connecting, the app has user authentication enabled and the client never called `client.signin()`. See [User authentication](/docs/getting-started/authentication#user-authentication).
## HTTP API requests are rejected
Every HTTP API call must be signed with the app secret. Server SDKs do this for you, so these errors usually come from custom clients or misconfigured SDKs:
| Error | Cause | Fix |
| --- | --- | --- |
| `400` with `Missing authentication query parameters` or `Invalid authentication query parameters` | The query string is missing or lacks `auth_key`, `auth_timestamp`, or `auth_signature`. | Sign the request, or use a server SDK. |
| `401` with a timestamp error | `auth_timestamp` is more than 600 seconds from the server clock. | Generate the timestamp per request and sync clocks with NTP. |
| `401` with `body_md5 mismatch` | The body was changed after hashing, or re-serialized by a middleware. | Hash and send the exact same bytes. |
| `401` with any other message | Wrong key or secret, or a signature over a different method, path, or query. | Check the credentials and compare the signed string with the request. |
[First connection](/docs/getting-started/first-connection#publish-over-raw-http) has a working signed `curl` example to compare against.
## Events publish but clients do not receive them
Check the channel name and protocol mode first. Protocol V1 clients receive Pusher-shaped event prefixes. Protocol V2 clients receive Sockudo-native prefixes and V2 metadata.
Watch the counters while you publish a fresh event:
```bash
curl -s http://127.0.0.1:9601/metrics \
| grep -E '^sockudo_(connected|http_calls_received_total|ws_messages_sent_total)'
```
`sockudo_http_calls_received_total` should rise with each publish, and `sockudo_ws_messages_sent_total` should rise as Sockudo delivers to subscribers. If the first moves and the second does not, no connection on that node is subscribed to the channel.
Also check:
- The publish and the subscription use the same app, channel, and event name, including case.
- The publish does not include the subscriber's own `socket_id`; Sockudo excludes that connection from delivery.
In a cluster, verify all nodes share the same adapter and cache configuration. Local memory adapters do not fan out between nodes.
If delivery works with one node and becomes intermittent with two, the nodes are almost certainly running the default `local` adapter. Each publish then reaches only the clients connected to the node that handled it. Configure a shared adapter such as Redis on every node.
## Recovery or rewind is incomplete
Recovery depends on continuity. Sockudo should fail closed when the stream cannot be proven: if the replay buffer expired or the stream was reset, the client receives a resume failure and must rebuild its state instead of silently missing messages.
Check:
- `protocolVersion: 2`
- `connectionRecovery: true`
- replay buffer TTL and size
- channel-specific rewind limits
- adapter delivery latency and duplicate suppression metrics
The replay buffer is configured on the server:
| Key | Default | What it does |
| --- | --- | --- |
| `[connection_recovery] enabled` | `false` | Keeps the replay buffer that resume depends on. The sample `config/config.toml` sets it to `true`. |
| `[connection_recovery] buffer_ttl_seconds` | `120` | How long a disconnected client can be away and still resume. |
| `[connection_recovery] max_buffer_size` | `100` | Messages kept per channel for replay. |
The matching environment variables are `CONNECTION_RECOVERY_ENABLED`, `CONNECTION_RECOVERY_BUFFER_TTL`, and `CONNECTION_RECOVERY_MAX_BUFFER_SIZE`. For gaps longer than the buffer, use durable history; see [History and recovery](/docs/server/history-recovery).
## Push notification publish is accepted but nothing arrives
Push is asynchronous by design. A successful admission response means Sockudo accepted the publish, not that every provider delivered it.
Check in order:
1. The device registration exists and is active.
2. The device has a provider token for the target platform.
3. The channel push subscription exists when targeting a channel.
4. Provider credentials are configured for the platform.
5. The publish returned a publish ID.
6. The publish status shows how far the publish progressed.
7. Provider delivery callbacks are reaching Sockudo when configured.
Read the status with the server SDK, which signs the request for you:
```ts
const status = await sockudo.getPublishStatus("pub_123");
console.log(status);
```
The underlying endpoint is `GET /apps/{appId}/push/publish/{publishId}/status`. Its lifecycle state shows whether the publish is still queued, planning, or dispatching, or has finished as succeeded, partially succeeded, failed, expired, cancelled, or dead-lettered.
Common push causes:
- APNs topic or environment mismatch
- Web Push VAPID key mismatch
- FCM project credentials for the wrong app
- expired device provider token
- client registration stored under a different `client_id`
- a server built without the `push` feature or the provider's feature, such as `push-apns`
## Webhooks are rejected
Webhook validation uses the raw body, not a parsed JSON object. Sockudo signs the exact bytes it sends and puts the signature in the `X-Pusher-Signature` header.
```ts
app.post("/sockudo/webhook", express.raw({ type: "application/json" }), (req, res) => {
const webhook = sockudo.webhook({
rawBody: req.body.toString(),
headers: req.headers,
});
if (!webhook.isValid()) {
return res.status(401).send("invalid signature");
}
res.sendStatus(200);
});
```
Keep the raw request bytes from your framework before JSON parsing mutates whitespace or key order. The SDK also checks that `Content-Type` is `application/json`, so a proxy that rewrites that header makes every webhook invalid.
## Escalation bundle
When reporting an issue, include:
- Sockudo version (`sockudo --version`) and feature flags
- protocol version used by affected clients
- app manager, adapter, cache, and queue drivers
- relevant configuration with secrets redacted
- server logs around the failure
- metrics for connection, publish, fanout, history, and push
- a minimal client and server SDK reproduction
## Next steps
- [Errors](/docs/reference/errors): HTTP error codes, WebSocket close codes, and capability-token errors.
- [Observability](/docs/server/observability): Metrics, logs, and alerts for every subsystem.
- [Logging](/docs/reference/logging): Log levels, formats, and what each subsystem reports.
---
# Deployment guide (https://sockudo.io/docs/deployment)
> Choose a Sockudo production topology, configuration strategy, and rollout path.
This guide helps you pick a production shape for Sockudo before you pick a platform. It is for
anyone taking Sockudo beyond a laptop, from a first single-container deployment to a multi-zone
cluster. Read it first, then follow the platform guide that matches the topology you choose.
A good Sockudo deployment starts with the failure model, not the platform logo. Decide whether one
process is sufficient, which state must survive a restart, and whether a client may reconnect to a
different node. Those decisions determine the adapter, cache, queue, app manager, and history
backends.
- [Static configuration](/docs/deployment/static-configuration): Use TOML or JSON for structure, environment variables for overrides, and secret stores for credentials.
- [Linux host](/docs/deployment/linux): Run a binary under systemd, raise file limits correctly, tune the node, and configure a reverse proxy.
- [Docker and Compose](/docs/deployment/docker): Mount configuration, set container limits, handle secrets, and move from one host to several.
- [Kubernetes and Helm](/docs/deployment/kubernetes): Configure probes, disruption budgets, resources, ingress, autoscaling, and pod or node sysctls.
- [Cloud platforms](/docs/deployment/cloud-platforms): Apply the same architecture on AWS, Google Cloud, and Azure, including managed-node tuning examples.
- [Capacity planning](/docs/deployment/capacity-planning): Budget connections, find the real bottleneck, and run geographically honest benchmarks.
## How a Sockudo cluster fits together
Each Sockudo node holds the WebSocket connections that landed on it. Everything a node cannot
answer alone goes through a shared backend:
```mermaid
flowchart TB
C[Clients] --> LB[Load balancer]
LB --> A[Sockudo node A]
LB --> B[Sockudo node B]
A <--> AD[Horizontal adapter]
B <--> AD
A --> CQ[Cache and queue]
B --> CQ
A --> DS[Durable stores]
B --> DS
```
Each backend exists for a different reason, which is why they are configured separately:
| Backend | What it does | Why it must be shared across nodes |
| --- | --- | --- |
| Adapter | Fans a publish out to every node with matching subscribers, and answers cross-node queries such as channel occupancy and presence. | A client on node A must receive an event published through node B. |
| Cache | Holds coordination state such as idempotency keys and cached channel data. | A retried publish can land on a different node and must still be deduplicated. |
| Rate limiter | Counts requests against per-app and per-client limits. | Limits enforced per node multiply by the node count. |
| Queue | Carries webhook and push jobs. | Work enqueued on a node that dies should still be delivered. |
| App manager | Stores app IDs, keys, secrets, and policy. | Every node must agree on which apps exist and what they may do. |
| History and version stores | Persist Protocol V2 history, recovery data, and mutable-message versions. | Recovery and mutations must work when a client reconnects to another node. |
You can mix drivers. A common pattern is a NATS adapter for fanout with Redis for cache, queue,
and rate limits, plus PostgreSQL for apps and history.
## Choose a topology
| Deployment | Good fit | Runtime backends | Main limitation |
| --- | --- | --- | --- |
| One process, memory only | Local development, demos, CI | Local adapter, memory cache, queue, app manager, history | Restart loses process-local state; no failover. |
| One production VM | Small stable workload with a simple operational model | Local adapter; static apps or a durable app manager; optional durable history | The host is one failure domain. |
| Multiple nodes with Redis | Most regional production workloads | Redis adapter, cache, queue, and rate limiter; shared app manager when apps change dynamically | Redis availability and capacity become part of the realtime path. |
| Multiple nodes with NATS | High fanout or high subscription churn | NATS adapter, Redis cache, durable queue and app manager | NATS carries fanout, but coordination and durable features still need shared stores. |
| Durable Protocol V2 cluster | Recovery, history, mutable messages, annotations, AI Transport | Shared adapter and cache plus durable history and version stores | More write amplification and more dependencies to operate. |
| Managed/serverless container platform | Moderate socket counts with platform-managed rollout and scaling | Shared adapter and stores from the first replica | Platform connection duration, concurrency, and node-tuning limits apply. |
Memory drivers are not shared because several processes use identical configuration. Every
multi-node deployment needs a horizontal adapter. Any feature whose correctness crosses nodes also
needs its documented shared authority.
`adapter.fallback_to_local` defaults to `true`. If the configured horizontal adapter cannot be
initialized at startup, the node logs a warning and runs with the local adapter instead. In a
cluster that silently splits clients into isolated islands that never see each other's events. Set
`fallback_to_local = false` (or `ADAPTER_FALLBACK_TO_LOCAL=false`) on every multi-node deployment
so a broken adapter fails startup instead.
See [Scaling](/docs/server/scaling) for the cross-node semantics of each adapter.
## Recommended production baseline
A conventional regional cluster has:
- at least two Sockudo instances spread across failure domains
- a load balancer that supports WebSocket upgrades and has a deliberate idle timeout
- Redis or another supported horizontal adapter
- a shared Redis or Redis Cluster cache for coordination, idempotency, and distributed rate limits
- durable app storage if app records change at runtime
- a durable queue when webhook or push work must survive a node failure
- durable history and version storage when Protocol V2 recovery or mutable state must cross nodes
- metrics, structured logs, readiness checks, and a tested drain path
Sticky sessions are not required for basic pub/sub. With a shared adapter, any node can serve any
client. They can still reduce reconnect churn, but a Protocol V2 reconnect to a different node
recovers only when the replay state is available through a shared backend. Start with
[Scaling](/docs/server/scaling) for cross-node semantics and [Security](/docs/server/security) for
network and secret boundaries.
## Configuration strategy
Use three layers with distinct responsibilities:
1. Put stable structure in a version-controlled TOML or JSON file: enabled features, driver choices,
limits, queue reliability, history retention, and per-app policy.
2. Put environment-specific addresses and non-secret switches in deployment variables or a
generated config file.
3. Inject app secrets, database passwords, broker credentials, and provider keys from the
platform's secret store.
The split keeps reviews meaningful: a pull request that changes a driver or a limit is visible in
Git, while credentials rotate without a code change and never appear in the repository.
Sockudo does not treat environment variables as a complete alternative syntax for every nested
configuration field. Complex policy and multi-app definitions belong in a file or persistent app
manager. Config files can reference environment variables with `${VAR}` placeholders when a nested
value must come from the environment. See [Static configuration](/docs/deployment/static-configuration)
for exact precedence and equivalent TOML and JSON examples.
## Workload questions to answer first
Document these inputs before choosing instance sizes:
| Input | Why it matters |
| --- | --- |
| Peak concurrent sockets and reconnect surge | Sets file descriptor, memory, load-balancer, and admission budgets. |
| Messages per second and average payload | Sets CPU, network, adapter, and serialization load. |
| Subscribers per publish | Distinguishes ingest throughput from fanout throughput. |
| Public, private, and presence mix | Adds authentication, membership, and transition work. |
| Subscription churn | Can dominate otherwise quiet chat or room workloads. |
| Protocol V1 or V2 features | Recovery, deltas, filtering, history, and mutations change state and compute costs. |
| Retention and recovery objectives | Selects memory buffers versus durable stores. |
| Client geography | Changes handshake time, RTT, reconnect behavior, and apparent benchmark throughput. |
[Capacity planning](/docs/deployment/capacity-planning) turns these answers into per-node limits.
## Rollout sequence
1. Build or pin one exact Sockudo image or binary version and the required Cargo features.
Verify with `sockudo --version`, which prints the version, for example `sockudo 5.2.0`.
2. Validate the resolved config in staging. Startup must log `explicit configuration applied` and
`applied environment variable overrides`, and `curl -fsS http://:6001/up/` must
print `OK`.
3. Load test from realistic client regions and confirm the load generators are not saturated.
4. Set a per-node `SOCKUDO_MAX_CONNECTIONS` below the measured safe ceiling. Above 95% of that
value, `/ready` returns `503 CAPACITY` so a load balancer can stop routing new clients.
5. Deploy at minimum capacity before enabling autoscaling.
6. Remove a node while traffic is active and verify that its clients reconnect elsewhere and that
recovery succeeds.
7. Fail or pause each shared dependency and verify readiness, alerts, and backpressure.
8. Roll in small batches while watching connection churn, adapter errors, and recovery failures.
## Production gate
Do not call a deployment ready until all of these are true:
- `/live` is used for liveness and startup checks
- `/up`, `/up/`, or the capacity-aware `/ready` is used for readiness
- the load balancer idle timeout and Sockudo heartbeat behavior have been tested together
- file descriptor limits have been checked in the actual process or container
- credentials are absent from images, ConfigMaps, command lines, and logs
- every replica has a unique `INSTANCE_PROCESS_ID`, stable across restarts when logs, metrics, or
Iggy consumers depend on it
- `adapter.fallback_to_local` is `false` on every multi-node deployment
- multi-node drivers and feature flags pass Sockudo startup validation
- termination grace exceeds Sockudo's shutdown grace period plus its drain budgets
- load testing includes connect, steady-state, fanout, churn, and reconnect phases
- dashboards distinguish a Sockudo bottleneck from the load generator, load balancer, NAT, or broker
`/up` checks the app manager, adapter, cache, and queue. If a liveness probe uses it, a slow Redis
makes the orchestrator restart every healthy Sockudo process at once, and every client reconnects
into the same slow Redis. `/live` answers only whether the process is running.
## Next steps
- [Static configuration](/docs/deployment/static-configuration): Learn the exact loading order, environment overrides, and placeholder substitution.
- [Scaling](/docs/server/scaling): Understand adapters, duplicate delivery, recovery across nodes, and rolling deploys.
- [Capacity planning](/docs/deployment/capacity-planning): Turn workload numbers into per-node connection limits.
- [HTTP endpoints](/docs/reference/http-endpoints): Reference for /live, /up, /ready, /accept-traffic, and /metrics.
---
# Static configuration (https://sockudo.io/docs/deployment/static-configuration)
> Configure Sockudo with TOML or JSON, understand environment precedence, and keep secrets out of static files.
This page explains exactly how Sockudo resolves its configuration at startup: which file it reads,
how environment variables override it, and where secrets belong. Read it before writing a
production config for any platform; the Linux, Docker, and Kubernetes guides all build on it.
Sockudo supports a static TOML file, a legacy-compatible JSON file, and runtime environment
overrides. Use a file for the complete deployment shape. Use environment variables for the subset
of settings that vary per environment and for secret references supplied by the platform.
## Loading order
Configuration is resolved once, at startup, in this order:
1. Start from compiled defaults.
2. If `--config ` (short form `-c`) is passed, load that file and nothing else.
3. Otherwise, look in the process working directory for `config/config.toml`, then
`config/config.json`, and load the first one that exists. If neither exists, keep the compiled
defaults.
4. Apply supported environment-variable overrides on top of the loaded values.
5. Validate the final configuration.
An explicit `.toml` path is parsed as TOML. Any other filename extension is parsed as JSON. Prefer a
`.toml` or `.json` suffix so the format is obvious.
```bash
sockudo --config /etc/sockudo/config.toml
sockudo --config /etc/sockudo/config.json
```
The selected file becomes the complete file-based configuration. An explicit `--config` file is not
merged with an auto-discovered `config/config.toml`; fields missing from the file receive compiled
defaults, not values from another file.
If the selected file (explicit or auto-discovered) cannot be read, contains an unresolved
`${VAR}` placeholder, or fails to parse, Sockudo prints
`sockudo: failed to load configuration file '': ` to stderr and exits with status 1.
It does not fall back to defaults. A failed final validation also stops startup with
`Configuration validation failed: `. Treat a crash loop after a config change as a config
error first.
Environment overrides behave differently: if applying them fails, Sockudo logs
`environment variable overrides failed: ` and continues starting. Alert on that log line.
Use `--config` in every production deployment. Relying on auto-discovery makes the result depend on
the working directory. The `CONFIG_FILE` environment variable present in the published container
image does not select the file by itself; the image's default command passes an explicit
`--config` path.
Configuration is read only at startup. Change it through a rollout or service restart; there is no
general-purpose hot reload.
### Startup log lines
Once logging is initialized, Sockudo records which source it used. Expect one of these:
| Log message | Meaning |
| --- | --- |
| `explicit configuration applied: ` | The `--config` file loaded. |
| `loaded configuration file: config/config.toml` | Auto-discovery found a file (or `config/config.json`). |
| `no config/config.toml or config/config.json found; using defaults` | No file was found; only defaults and environment variables apply. |
| `applied environment variable overrides` | Environment overrides were applied. |
In production, require the `explicit configuration applied` line before sending traffic.
## TOML or JSON?
| Format | Prefer it when | Notes |
| --- | --- | --- |
| TOML | Humans maintain the file and comments are valuable. | Preferred for bare-metal, VM, and source deployments. |
| JSON | A chart, control plane, or program generates the file. | Useful for Helm `configJson` and legacy installations; comments are not allowed. |
| Environment only | A small, single-app deployment uses only supported overrides. | Not every nested policy has an environment equivalent. |
TOML and JSON deserialize into the same `ServerOptions` structure, so every key works in both.
The annotated reference file is `config/config.toml` in the repository.
## Environment placeholders in config files
Before parsing, Sockudo replaces `${VAR}` and `${VAR:-default}` placeholders in the file with values
from the process environment. This lets a committed file reference a secret without containing it,
including nested values that have no dedicated environment variable:
```toml title="config.toml"
[metrics]
port = ${METRICS_PORT:-9601}
[[app_manager.array.apps]]
id = "production"
key = "${PROD_APP_KEY}"
secret = "${PROD_APP_SECRET}"
enabled = true
```
Rules worth knowing:
- An environment variable that is set but empty is treated as unset, so `${VAR:-default}` uses the
default.
- A placeholder without a default whose variable is unset fails the load and lists every
unresolved variable in the error.
- Substitution is plain text over the whole file before parsing, including comments. A `${...}` in a
comment must also resolve.
- The value is inserted verbatim. Quote placeholders for string fields, and avoid values containing
`"` or `\` in TOML or JSON strings, because they would change the file's syntax.
Placeholders work for any key in the file, but they resolve only once, when the file is read. The
dedicated environment variables in the [environment variable reference](/docs/reference/environment-variables)
are applied after the file and always win. Use placeholders for secrets and nested values; use
dedicated variables for per-environment switches.
## Equivalent single-node files
This TOML profile keeps all state in one process and is suitable for a first production-like test,
not for failover:
```toml title="config.toml"
mode = "production"
host = "0.0.0.0"
port = 6001
debug = false
max_connections = 20000
activity_timeout = 120
shutdown_grace_period = 30
[adapter]
driver = "local"
[cache]
driver = "memory"
[queue]
driver = "memory"
[rate_limiter]
enabled = true
driver = "memory"
[metrics]
enabled = true
driver = "prometheus"
host = "0.0.0.0"
port = 9601
[app_manager]
driver = "memory"
[app_manager.array]
[[app_manager.array.apps]]
id = "example"
key = "replace-me"
secret = "replace-me"
enabled = true
[app_manager.array.apps.policy.limits]
max_connections = 20000
max_client_events_per_second = 100
[app_manager.array.apps.policy.features]
enable_client_messages = false
enable_user_authentication = true
[app_manager.array.apps.policy.channels]
allowed_origins = ["https://app.example.com"]
```
The top-level `max_connections` caps sockets on this node across all apps (`0` disables the cap).
The app's `policy.limits.max_connections` is a separate per-app quota.
The equivalent JSON is:
```json title="config.json"
{
"mode": "production",
"host": "0.0.0.0",
"port": 6001,
"debug": false,
"max_connections": 20000,
"activity_timeout": 120,
"shutdown_grace_period": 30,
"adapter": {
"driver": "local"
},
"cache": {
"driver": "memory"
},
"queue": {
"driver": "memory"
},
"rate_limiter": {
"enabled": true,
"driver": "memory"
},
"metrics": {
"enabled": true,
"driver": "prometheus",
"host": "0.0.0.0",
"port": 9601
},
"app_manager": {
"driver": "memory",
"array": {
"apps": [
{
"id": "example",
"key": "replace-me",
"secret": "replace-me",
"enabled": true,
"policy": {
"limits": {
"max_connections": 20000,
"max_client_events_per_second": 100
},
"features": {
"enable_client_messages": false,
"enable_user_authentication": true
},
"channels": {
"allowed_origins": ["https://app.example.com"]
}
}
}
]
}
}
}
```
Replace the example credentials before use. For a committed production file, replace the literal
key and secret with `${...}` placeholders, bootstrap the app from a secret-backed environment, or
use a persistent app manager.
## Clustered Redis profile
The stable structure can remain in TOML:
```toml title="config.toml" {9-10}
mode = "production"
host = "0.0.0.0"
port = 6001
max_connections = 50000
shutdown_grace_period = 30
[adapter]
driver = "redis"
# Fail startup instead of silently running as an isolated local node.
fallback_to_local = false
[cache]
driver = "redis"
[queue]
driver = "redis"
[rate_limiter]
enabled = true
driver = "redis"
[app_manager]
driver = "memory"
[metrics]
enabled = true
host = "0.0.0.0"
port = 9601
```
`fallback_to_local` defaults to `true`. Leaving it on lets a node that cannot reach Redis at startup
run alone, splitting clients into groups that never see each other's events.
Inject the shared address, node identity, and one immutable app at runtime:
```bash title="sockudo.env"
REDIS_URL=rediss://sockudo@redis.internal:6379/0
INSTANCE_PROCESS_ID=sockudo-a-01
SOCKUDO_DEFAULT_APP_ENABLED=true
SOCKUDO_DEFAULT_APP_ID=production
SOCKUDO_DEFAULT_APP_KEY=secret-store-value
SOCKUDO_DEFAULT_APP_SECRET=secret-store-value
SOCKUDO_DEFAULT_APP_ALLOWED_ORIGINS=https://app.example.com
SOCKUDO_DEFAULT_APP_MAX_CONNECTIONS=50000
```
`REDIS_URL` updates the Redis adapter, cache, queue, and rate limiter connection. Driver selection
still must be `redis`, either in the file or through `ADAPTER_DRIVER`, `CACHE_DRIVER`,
`QUEUE_DRIVER`, and `RATE_LIMITER_DRIVER`.
`INSTANCE_PROCESS_ID` names the node. When it is unset, Sockudo generates a random UUID at every
start, which is unique but changes on each restart. Set it explicitly when logs, metrics, or Iggy
consumers need a stable identity, and never put one shared value in a file that every replica
loads.
An app registered through `SOCKUDO_DEFAULT_APP_*` gets a per-app connection quota of `100` unless
`SOCKUDO_DEFAULT_APP_MAX_CONNECTIONS` is set. Clients beyond the quota are rejected even when the
node has capacity. Set it deliberately, as in the example above.
Use a persistent app manager instead of environment bootstrap when operators create, rotate, or
disable multiple applications without rebuilding the deployment.
## What belongs where
| Setting | Static file | Environment | Secret store |
| --- | --- | --- | --- |
| Driver selection and feature gates | Yes | Optional rollout override | No |
| Nested app/channel policy | Yes, or persistent app manager | Only the documented subset | Credentials only |
| Retention, buffer, and queue reliability | Yes | Use supported overrides sparingly | No |
| Per-environment hostnames and ports | Template, generated file, or `${VAR}` placeholder | Yes | Only if address contains credentials |
| App keys and secrets | Avoid literals in committed files; use `${VAR}` placeholders | Inject from secret reference | Yes |
| Redis, SQL, broker passwords | Avoid literals | Inject from secret reference | Yes |
| TLS private keys and push provider keys | File path or credential reference | Path/reference only | Yes |
Do not place secrets in image layers, Helm values committed to Git, Kubernetes ConfigMaps, process
arguments, or Terraform plan output.
## Environment override behavior
Environment variables are explicit mappings in the Sockudo configuration loader, not a generic
`SECTION_FIELD=value` translation. The [environment variable reference](/docs/reference/environment-variables)
is the authoritative list. An unknown variable name is ignored without a warning, so check spelling
against the reference.
Several app-bootstrap variables have deliberate replacement behavior:
- a complete `SOCKUDO_DEFAULT_APP_ID`, `SOCKUDO_DEFAULT_APP_KEY`, and
`SOCKUDO_DEFAULT_APP_SECRET` set registers the environment-backed default app
- setting any of those three, or `SOCKUDO_DEFAULT_APP_ENABLED=false`, removes the apps declared
inline in the file
- `SOCKUDO_DEFAULT_APP_ENABLED=false` disables the environment-backed registration
- a partial set (for example an ID and key without a secret) registers nothing and logs
`default app env configured but id, key, and secret were not all provided`
- `SOCKUDO_SKIP_INLINE_APPS=true` skips apps declared inside the file
- `APP_MANAGER_REGISTER_INLINE_APPS=false` also prevents inline registration
Avoid mixing an inline app and environment bootstrap accidentally. Choose one source for each app
and confirm the startup messages.
## Container and Kubernetes mounts
Mount a read-only file into the published container and replace the default command:
```bash
docker run --rm \
--mount type=bind,src="$PWD/config.toml",dst=/app/config/production.toml,readonly \
ghcr.io/sockudo/sockudo:5.2.0 \
sockudo --config /app/config/production.toml
```
In Kubernetes, mount non-secret structure from a ConfigMap and inject secrets separately:
```yaml
containers:
- name: sockudo
args: ["--config", "/etc/sockudo/config.toml"]
envFrom:
- secretRef:
name: sockudo-runtime
volumeMounts:
- name: config
mountPath: /etc/sockudo
readOnly: true
volumes:
- name: config
configMap:
name: sockudo-config
```
With `${VAR}` placeholders in the ConfigMap, the Secret supplies the values and the ConfigMap stays
safe to commit. The Helm chart also accepts `configJson` for advanced generated configuration and
`extraEnvFrom` for Secret or ConfigMap references; see [Kubernetes and Helm](/docs/deployment/kubernetes).
## Validate before rollout
1. Check syntax in CI so a typo never reaches a host:
```bash
taplo check config.toml
jq empty config.json
```
Both commands print nothing and exit `0` when the file parses.
2. Start the exact production binary with the same features, environment, working directory, and
mounted secrets:
```bash
sockudo --config /etc/sockudo/config.toml
```
The process must keep running. The logs must contain `explicit configuration applied` and
`applied environment variable overrides`, and must not contain
`environment variable overrides failed` or `Configuration validation failed`.
3. Check the process and app health from the same host:
```bash
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6001/live
curl -fsS http://127.0.0.1:6001/up/
```
Expect `200` from `/live`. `/up/` prints `OK` once shared dependencies are reachable,
`DEGRADED` (still `200`) when a non-critical check reports degradation, and fails with `503` or
`404` when a dependency is down or the app does not exist.
## Next steps
- [Linux host](/docs/deployment/linux): Run the validated config under systemd with correct file limits.
- [Docker and Compose](/docs/deployment/docker): Mount the config read-only and inject secrets into a container.
- [Kubernetes and Helm](/docs/deployment/kubernetes): Supply the config through chart values, a ConfigMap, or a Secret.
- [Configuration reference](/docs/reference/configuration): Every configuration key with its default.
---
# Linux VM and bare metal (https://sockudo.io/docs/deployment/linux)
> Run Sockudo under systemd with correct file limits, measured kernel tuning, and a WebSocket-aware proxy.
This guide runs Sockudo as a systemd service on a Linux VM or physical host, then tunes file limits,
the kernel, and the reverse proxy for long-lived WebSocket connections. Choose this path when you
want direct control over the process and host, or when connection density matters more than
orchestration features.
A Linux VM is often the simplest production deployment for a known, stable workload. It gives you
direct control over file limits, kernel settings, CPU scheduling, network queues, and the process
lifecycle. Start with one host only if a host restart is an acceptable outage; use at least two
hosts and a shared adapter for availability.
## Host layout
Use separate paths for the binary, configuration, secrets, and service account:
```text
/usr/local/bin/sockudo
/etc/sockudo/config.toml
/etc/sockudo/sockudo.env
/etc/systemd/system/sockudo.service
```
Root owns the binary and configuration so a compromised Sockudo process cannot rewrite them. The
`sockudo` group can read the configuration and secrets. Sockudo does not need root privileges when
it binds to ports `6001` and `9601`, because both are above 1024.
## Install the service
1. Create the service account and configuration directory:
```bash
sudo useradd --system --home /var/lib/sockudo --create-home --shell /usr/sbin/nologin sockudo
sudo install -d -o root -g sockudo -m 0750 /etc/sockudo
```
Verify with `id sockudo`, which prints the new user and group.
2. Install a pinned binary. With the release installer, download into a staging directory and then
install it system-wide:
```bash
curl --proto '=https' --tlsv1.2 -sSfL https://sockudo.io/install.sh \
| sh -s -- --version 5.2.0 --bin-dir "$HOME/sockudo-5.2.0"
sudo install -o root -g root -m 0755 "$HOME/sockudo-5.2.0/sockudo" /usr/local/bin/sockudo
```
From a source build, install `target/release/sockudo` the same way. Verify with
`/usr/local/bin/sockudo --version`, which prints `sockudo 5.2.0`.
3. Install the configuration and the secret environment file:
```bash
sudo install -o root -g sockudo -m 0640 config/production.toml /etc/sockudo/config.toml
sudo install -o root -g sockudo -m 0640 /dev/null /etc/sockudo/sockudo.env
sudoedit /etc/sockudo/sockudo.env
```
Keep `sockudo.env` at mode `0640` or stricter. Verify with `ls -l /etc/sockudo`: both files
show `-rw-r-----` and group `sockudo`.
4. Write the systemd unit from the next section to `/etc/systemd/system/sockudo.service`.
5. Start the service and enable it at boot:
```bash
sudo systemctl daemon-reload
sudo systemctl enable --now sockudo
sudo systemctl status sockudo
```
`status` must show `active (running)`. Confirm the configuration source and health:
```bash
journalctl -u sockudo -b | grep -E 'explicit configuration applied|applied environment variable overrides'
curl -fsS http://127.0.0.1:6001/up/
```
The `curl` command prints `OK`.
Do not put secrets directly in the systemd unit. Unit contents are commonly collected in
diagnostics and configuration management, and `systemctl show` exposes them to local users.
## systemd unit
```ini title="/etc/systemd/system/sockudo.service"
[Unit]
Description=Sockudo realtime server
Wants=network-online.target
After=network-online.target
[Service]
Type=simple
User=sockudo
Group=sockudo
WorkingDirectory=/var/lib/sockudo
EnvironmentFile=-/etc/sockudo/sockudo.env
ExecStart=/usr/local/bin/sockudo --config /etc/sockudo/config.toml
Restart=on-failure
RestartSec=2s
LimitNOFILE=1048576
TimeoutStopSec=120s
KillSignal=SIGTERM
NoNewPrivileges=true
PrivateTmp=true
PrivateDevices=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
RestrictSUIDSGID=true
LockPersonality=true
[Install]
WantedBy=multi-user.target
```
Why the key directives matter:
- `--config` is explicit, so the result does not depend on `WorkingDirectory`. A file that fails to
load stops startup, and `Restart=on-failure` retries it every two seconds; check
`journalctl -u sockudo` when the unit flaps.
- `LimitNOFILE` is the only reliable way to raise the open-file limit for a systemd service. Every
socket is a file descriptor.
- `TimeoutStopSec` is how long systemd waits after `SIGTERM` before it sends `SIGKILL`. A kill
during drain can lose buffered webhook and lifecycle events.
- The `Protect*`, `Private*`, and `Restrict*` directives remove filesystem and kernel access that
Sockudo does not need.
Set `TimeoutStopSec` above `shutdown_grace_period` plus twice
`queue.reliability.shutdown_timeout_ms / 1000`, with additional time for connection and backend
teardown. The two budgets cover [webhook draining](/docs/server/webhooks#shutdown-delivery) and
queue worker shutdown.
For example (illustrative): with `shutdown_grace_period = 30` and the default
`shutdown_timeout_ms = 30000`, the drain can take `30 + 2 × 30 = 90` seconds. Adding 30 seconds for
teardown gives the `TimeoutStopSec=120s` shown above. Recalculate when you change either value.
Add `ReadWritePaths=` only when a selected feature writes to local disk. A stricter security
directive may need adjustment for a provider SDK or a TLS key path; keep any exception narrow.
## File descriptor limits
Every accepted socket consumes a file descriptor, as do logs, shared-backend connections, metrics,
and internal listeners. When the process reaches its limit, new connections fail at `accept` even
though CPU and memory look healthy. Budget:
```text
required nofile >= planned sockets + backend connections + listeners + operational reserve
```
For example (illustrative): 100,000 planned sockets, 200 backend connections, and 10 listeners
with a 20% reserve need `(100,000 + 200 + 10) × 1.2 ≈ 120,252` descriptors. The unit's
`LimitNOFILE=1048576` covers that comfortably.
Use at least 10–20% reserve and never set `SOCKUDO_MAX_CONNECTIONS` equal to the hard `nofile`
limit.
`/etc/security/limits.d/*.conf` applies to PAM login sessions. It does not reliably change a
systemd service, so a shell showing a high `ulimit -n` proves nothing about Sockudo.
`LimitNOFILE=` in the unit is the relevant setting, and it cannot exceed the kernel's `fs.nr_open`.
Verify the running process rather than the shell:
```bash
systemctl show sockudo -p LimitNOFILE
cat /proc/"$(pidof sockudo)"/limits | grep -i 'open files'
ls /proc/"$(pidof sockudo)"/fd | wc -l
```
The first two commands should both report `1048576`; the third shows how many descriptors the
process holds right now.
Also check system-wide pressure:
```bash
sysctl fs.file-max fs.nr_open
cat /proc/sys/fs/file-nr
```
The first field of `file-nr` is the number of allocated file handles; it should stay well below
`fs.file-max`.
## Baseline kernel profile
Treat this as a starting profile for a dedicated Sockudo node, not a universal requirement:
```ini title="/etc/sysctl.d/99-sockudo.conf"
# Listen and SYN queues for connection bursts.
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
# Input backlog; raise only when packet drops show the default is too small.
net.core.netdev_max_backlog = 16384
# Detect dead peers eventually at the TCP layer. Sockudo protocol heartbeats
# remain the primary application-level liveness mechanism.
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 5
# Avoid restarting congestion control from the initial window after idle.
net.ipv4.tcp_slow_start_after_idle = 0
```
Apply and verify:
```bash
sudo sysctl --system
sysctl net.core.somaxconn net.ipv4.tcp_max_syn_backlog
sysctl net.core.netdev_max_backlog
sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
```
Each `sysctl` line must print the value from the file, for example `net.core.somaxconn = 65535`.
Why these are not all “set once and forget” values:
- a larger `somaxconn` helps only if the application's listen backlog and accept loop can use it
- a larger SYN queue helps connection bursts, such as every client reconnecting after a node
restart, not steady-state message fanout
- a larger `netdev_max_backlog` can trade drops for latency and memory when CPU is already saturated
- TCP keepalive does not replace Sockudo's protocol ping/pong and should not fire more aggressively
than necessary
- `tcp_slow_start_after_idle = 0` matters for realtime traffic because quiet sockets are idle most
of the time; without it, the first burst after a pause starts from a small congestion window
- modern Linux already autotunes TCP buffers; raising `rmem_max` and `wmem_max` without high
bandwidth-delay-product evidence can increase per-socket memory risk
Optional settings such as `tcp_fin_timeout`, socket buffer ceilings, IRQ affinity, CPU isolation,
busy polling, and NIC ring sizes should follow a measured bottleneck. Record the before/after
kernel counters and repeat the same workload.
Do not widen `ip_local_port_range` to increase inbound WebSocket capacity. Inbound sockets use the
server's listening port. The ephemeral range matters for high-volume outbound connections from
Sockudo, a NAT gateway, or the load generator.
## Inspect the network path
Useful counters during a connect or reconnect surge:
```bash
ss -s
ss -lnt
nstat -az | grep -E 'ListenOverflows|ListenDrops|TCPBacklogDrop|TCPSynRetrans'
ip -s link
ethtool -S eth0
```
Interpret them together:
- `ListenOverflows` or `ListenDrops` suggests the accept/listen path cannot keep up
- interface drops suggest host networking, vNIC, or CPU pressure
- SYN retransmits can be the client network, load balancer, firewall, or server
- no server-side drops with low achieved load usually points upstream or at the generator
## Reverse proxy example
Terminate TLS at a load balancer or a carefully configured reverse proxy. A WebSocket starts as an
HTTP/1.1 request with `Upgrade: websocket`; a proxy that does not forward those headers turns every
connection attempt into a plain HTTP request. NGINX needs HTTP/1.1 upgrade forwarding and long
enough timeouts:
```nginx title="/etc/nginx/conf.d/sockudo.conf"
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
upstream sockudo {
least_conn;
server 10.0.1.10:6001 max_fails=3 fail_timeout=10s;
server 10.0.2.10:6001 max_fails=3 fail_timeout=10s;
}
server {
listen 443 ssl;
server_name ws.example.com;
location / {
proxy_pass http://sockudo;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 180s;
proxy_send_timeout 180s;
proxy_buffering off;
}
}
```
`least_conn` sends each new connection to the backend with the fewest active connections, which
suits long-lived sockets better than round robin. `proxy_buffering off` stops NGINX from holding
realtime frames in a buffer.
The default Sockudo `activity_timeout` is 120 seconds. Set every proxy or load-balancer idle timeout
above the longest expected interval between application or heartbeat traffic, with margin. A
180-second starting value works with the default, but validate it with the actual client SDKs and
network path.
If the proxy closes a connection that has been quiet for less time than the client heartbeat
interval, idle clients disconnect and reconnect on a fixed cycle. The symptom is a steady sawtooth
in connection count with no errors on the Sockudo side.
### Client IP behind the proxy
If the proxy adds entries to `X-Forwarded-For`, configure `RATE_LIMITER_API_TRUST_HOPS` and
`RATE_LIMITER_WS_TRUST_HOPS` to the client's one-based position from the right of that header.
For the NGINX configuration above, use `1`: `$proxy_add_x_forwarded_for` appends the client's
address, so a client-supplied prefix is ignored. Other platforms can append more than one entry;
Google's external Application Load Balancer appends `, `, requiring `2`
when it connects directly to Sockudo. Validate the actual path rather than counting proxy machines.
Only enable forwarding-header trust when direct access to Sockudo is restricted to your proxies
and those proxies sanitize or append the headers. A hop count does not authenticate the socket
peer. With `0`, both `X-Forwarded-For` and `X-Real-IP` are ignored. Invalid or short chains fall
back to the socket peer, with a once-only warning containing no header values. See
[client IP extraction](/docs/reference/configuration#rate-limiter-client-ip) for migration details.
## One host versus several
| Concern | One host | Two or more hosts |
| --- | --- | --- |
| Adapter | `local` is valid | Use Redis, NATS, or another horizontal adapter |
| App definitions | Static memory app can work | All nodes need identical static apps, or use a shared app manager |
| Rate limits | Memory is per host | Use Redis or Redis Cluster for cluster-wide limits |
| Queue | Memory work is lost with the host | Use a durable shared queue for webhooks and push |
| History and mutations | Memory can be acceptable for tests | Use a supported durable backend |
| Load balancer | Optional reverse proxy | Required, with health checks and drain |
For multiple nodes, set `fallback_to_local = false`. The default is `true`, which lets a node that
cannot reach its horizontal adapter at startup run alone. A production cluster should fail startup
when its horizontal adapter is unavailable instead of silently splitting into isolated local
islands.
## Drain and upgrade
On `SIGTERM`, Sockudo marks itself as draining (`/up` and `/ready` return `503 DRAINING`), rejects
new WebSocket upgrades, closes existing connections with code 4200 so clients reconnect elsewhere,
and keeps the listener open for `shutdown_grace_period` seconds. See
[rolling deploys](/docs/server/scaling#rolling-deploys) for the full sequence and the
`shutdown_close_delay_ms` and `shutdown_close_spread_ms` pacing options.
1. Remove the host from load-balancer rotation. If you skip this, the balancer's health check
still sees `503 DRAINING` once Sockudo stops, but only after its probe interval.
2. Wait for the balancer's deregistration delay or connection drain policy.
3. Stop Sockudo with `sudo systemctl stop sockudo`. `TimeoutStopSec` must cover Sockudo's listener
grace, webhook drain, queue worker shutdown, and teardown.
4. Replace the binary and configuration atomically, for example with `install` to a new path
followed by `mv`.
5. Start the service with `sudo systemctl start sockudo`.
6. Wait until `curl -fsS http://127.0.0.1:6001/up/` prints `OK`, then restore
load-balancer membership.
Watch reconnect attempts, recovery success, adapter errors, and connection distribution throughout
the rollout.
## Production checklist
- `/usr/local/bin/sockudo --version` reports the intended release on every host
- `/proc//limits` shows the intended `Max open files`
- `SOCKUDO_MAX_CONNECTIONS` (or `max_connections`) is set below the measured safe ceiling
- `TimeoutStopSec` exceeds the drain budget calculated above
- the sysctl profile is applied after reboot (`sysctl --system` reads `/etc/sysctl.d`)
- proxy idle timeouts exceed the heartbeat interval, verified with a quiet client
- the metrics port `9601` is reachable only from your monitoring network
- `fallback_to_local = false` on every node of a multi-host deployment
## Next steps
- [Capacity planning](/docs/deployment/capacity-planning): Measure the safe connection ceiling before setting a production limit.
- [Observability](/docs/server/observability): Scrape metrics, ship logs, and set alerts.
- [Scaling](/docs/server/scaling): Add a second host with a shared adapter and plan rolling deploys.
- [Security](/docs/server/security): Harden secrets, origins, and network exposure.
---
# Docker and Compose (https://sockudo.io/docs/deployment/docker)
> Run Sockudo in containers with mounted configuration, correct limits, health checks, and shared backends.
This guide runs the published Sockudo image with `docker run` or Docker Compose, with production
limits, read-only configuration, and separate secrets. Choose it for a single container host, for
staging environments, or as the building block before moving to
[Kubernetes](/docs/deployment/kubernetes).
## What the image does by default
The published image is built from the repository `Dockerfile`:
- it runs as a non-root `sockudo` user with `/app` as the working directory
- its default command is `sockudo --config /app/config/config.json`
- it exposes `6001` (WebSocket and HTTP API) and `9601` (Prometheus metrics)
- it sets `HOST=0.0.0.0`, `PORT=6001`, and `METRICS_PORT=9601`
- it includes `curl` and a `HEALTHCHECK` that calls `http://localhost:${PORT}/${HEALTHCHECK_PATH}`
every 30 seconds, with `HEALTHCHECK_PATH=up`
The bundled `config.json` is a development example: `mode` is `development`, the adapter is
`local`, and `shutdown_grace_period` is `2` seconds. It lets a bare `docker run` start safely with
node-local drivers. For production, pin a release tag, mount or generate the exact configuration,
inject secrets separately, and set limits on the container itself.
The image is published to both `ghcr.io/sockudo/sockudo` and Docker Hub as `sockudo/sockudo`, with
the same version tags.
## Production-like docker run
1. Put the non-secret configuration in `config.toml` (see
[Static configuration](/docs/deployment/static-configuration)) and the secrets in
`sockudo.env`. Make the config readable by the container user and keep the env file private:
```bash
chmod 0644 config.toml
chmod 0600 sockudo.env
```
2. Start the container:
```bash
docker run -d \
--name sockudo \
--restart unless-stopped \
--init \
--stop-timeout 120 \
--ulimit nofile=262144:262144 \
--read-only \
--tmpfs /tmp:rw,noexec,nosuid,size=64m \
--mount type=bind,src="$PWD/config.toml",dst=/app/config/production.toml,readonly \
--env-file "$PWD/sockudo.env" \
-e HEALTHCHECK_PATH=up/production \
-p 6001:6001 \
-p 127.0.0.1:9601:9601 \
ghcr.io/sockudo/sockudo:5.2.0 \
sockudo --config /app/config/production.toml
```
3. Confirm the configuration source and health:
```bash
docker logs sockudo 2>&1 | grep 'explicit configuration applied'
curl -fsS http://127.0.0.1:6001/up/production
docker inspect sockudo --format '{{.State.Health.Status}}'
```
The log line names `/app/config/production.toml`, `curl` prints `OK`, and the health status
becomes `healthy` after the first check.
Why each flag is there:
| Flag | Purpose |
| --- | --- |
| `sockudo --config ...` after the image | Arguments after the image replace its default command, so include both `sockudo` and `--config`. |
| `--init` | Runs a minimal init as PID 1 that forwards signals and reaps child processes. |
| `--stop-timeout 120` | Time between `SIGTERM` and `SIGKILL`; it must cover Sockudo's drain (see below). |
| `--ulimit nofile=...` | Raises the open-file limit; every WebSocket is a file descriptor. |
| `--read-only` and `--tmpfs /tmp` | Makes the image filesystem immutable while leaving a small scratch area. Sockudo logs to stdout. |
| `-e HEALTHCHECK_PATH=up/production` | Points the image's built-in health check at your app. |
| `-p 127.0.0.1:9601:9601` | Publishes metrics only on loopback. |
The `CONFIG_FILE` environment value embedded in the image is descriptive; the server selects a
non-default file through `--config`.
Docker's default stop timeout is 10 seconds. After `SIGTERM`, Sockudo keeps its listener open for
`shutdown_grace_period`, then drains webhooks and queue work, each phase bounded by
`queue.reliability.shutdown_timeout_ms` (30 seconds by default). With
`shutdown_grace_period = 30` that is up to `30 + 2 × 30 = 90` seconds before teardown, so the
examples use 120 seconds. A `SIGKILL` during the drain can lose buffered lifecycle events.
## Compose profile
This example keeps the Sockudo process immutable and connects it to an external or separately
managed Redis:
```yaml title="compose.yaml"
name: sockudo
services:
sockudo:
image: ghcr.io/sockudo/sockudo:5.2.0
command: ["sockudo", "--config", "/app/config/production.toml"]
restart: unless-stopped
init: true
stop_grace_period: 120s
read_only: true
tmpfs:
- /tmp:size=64m,mode=1777
ulimits:
nofile:
soft: 262144
hard: 262144
ports:
- "6001:6001"
- "127.0.0.1:9601:9601"
volumes:
- ./config/production.toml:/app/config/production.toml:ro
env_file:
- ./secrets/sockudo.env
environment:
HOST: "0.0.0.0"
PORT: "6001"
METRICS_HOST: "0.0.0.0"
METRICS_PORT: "9601"
INSTANCE_PROCESS_ID: "compose-1"
healthcheck:
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:6001/up/production"]
interval: 10s
timeout: 3s
start_period: 20s
retries: 3
logging:
driver: local
options:
max-size: "20m"
max-file: "5"
```
Start it and check the result:
```bash
docker compose up -d
docker compose ps
```
The `sockudo` service shows `running (healthy)` once `/up/production` returns `OK`.
`METRICS_HOST` is `0.0.0.0` inside the container so the published port can reach the metrics
listener; the `127.0.0.1:9601` host binding is what keeps it private.
For local development, adding Redis to the same Compose project is convenient. In production, a
single-host Redis container has the same host failure domain as Sockudo. Use a managed or
independently operated Redis topology when the shared adapter, cache, queue, or rate limiter must
survive that host.
## Configuration and secrets
Prefer this separation:
- mount the complete non-secret TOML or JSON read-only
- supply secret-backed environment variables through the orchestrator
- reference those variables from the config with `${VAR}` placeholders when a nested value, such as
an inline app secret, has no dedicated environment variable
- mount TLS keys, provider credentials, and service-account JSON as read-only secret files
- reference secret file paths from Sockudo configuration or environment
An `env_file` is still a plaintext secret file on the Docker host. Limit its owner and mode, keep it
out of image build context and source control, and rotate it through the host's secret-management
workflow.
Docker build arguments and layers can remain visible in image metadata or caches. Inject
credentials at runtime only.
## Limits: host versus container
Three different layers can cap a container:
| Layer | Check | Configure |
| --- | --- | --- |
| Sockudo admission | `SOCKUDO_MAX_CONNECTIONS` | Static config or environment |
| Process `nofile` | `docker exec sockudo sh -c 'ulimit -n'` | `--ulimit` or Compose `ulimits` |
| Host kernel and service manager | `sysctl`, Docker daemon unit limits | Host image, sysctl, and daemon configuration |
The lowest of the three wins. Changing `/etc/security/limits.d` inside the image does not raise the
running container's file limit. Set it through the runtime and verify from inside the container.
Host-level connection queues and NIC settings remain host concerns. Namespaced sysctls may be set
per container, but only after confirming the runtime supports them and the value is safe for other
workloads. For a dedicated container host, apply the measured host guidance from
[Linux VM and bare metal](/docs/deployment/linux).
## CPU and memory
Start with a memory limit that leaves room for peak sockets, payload buffers, adapter queues, and
allocator fragmentation. An out-of-memory kill disconnects every client on that container at once,
and they all reconnect at the same moment.
CPU quotas can create latency cliffs during reconnect or fanout bursts. Set CPU reservations, watch
throttling metrics, and add a hard CPU limit only when multi-tenant isolation requires one and the
load test includes the same quota.
Measure:
```bash
docker stats sockudo
docker inspect sockudo --format '{{json .HostConfig.Ulimits}}'
docker exec sockudo sh -c 'ulimit -n; cat /proc/1/limits'
docker exec sockudo sh -c 'curl -fsS -o /dev/null -w "%{http_code}\n" http://127.0.0.1:6001/live'
```
`ulimit -n` must print `262144`, and the `/live` check prints `200`.
## Health and startup ordering
Use `/live` when the runtime needs to decide whether the process is alive. Use `/up` or
`/up/` when the load balancer needs dependency-aware readiness. `/up/` prints `OK`
with status `200` when healthy and returns `503` when a dependency check fails.
Compose `depends_on` can order startup, but it does not make a dependency permanently available.
Sockudo must still expose failed readiness and operators must alert on adapter, cache, queue, and
app-manager failures.
Plain Docker and Compose report a failing health check as `unhealthy` but do not restart the
container because of it. That is the right behavior for a dependency-aware check: restarting a
healthy process because Redis is briefly slow does not repair Redis and amplifies reconnect load.
If you add an auto-restart tool, point it at `/live`, not `/up`.
## Multi-host containers
Moving Compose services onto two VMs does not create Sockudo clustering automatically. Every
Sockudo replica must have:
- the same app identity and policy, or access to one shared app manager
- a unique `INSTANCE_PROCESS_ID`
- a shared horizontal adapter with `fallback_to_local = false`
- a shared cache for cross-node coordination and distributed limits
- durable queue and state backends where the enabled features require them
- a WebSocket-aware load balancer with readiness and connection drain
Do not place a local memory adapter behind a multi-host load balancer. Clients connected to
different hosts would occupy isolated realtime islands.
## Production checklist
- the image tag is a pinned release, never `latest`
- logs show `explicit configuration applied` with your mounted path
- `ulimit -n` inside the container matches the intended `nofile`
- the stop timeout exceeds `shutdown_grace_period` plus twice the queue shutdown timeout
- the metrics port is bound to loopback or a private network
- secrets live only in the env file or mounted secret files, never in the image or command line
- multi-host deployments use a shared adapter, cache, and queue
## Next steps
- [Kubernetes and Helm](/docs/deployment/kubernetes): Move from one container host to orchestrated replicas.
- [Linux VM and bare metal](/docs/deployment/linux): Tune the container host's kernel and network path.
- [Static configuration](/docs/deployment/static-configuration): Write the config file you mount into the container.
- [Scaling](/docs/server/scaling): Choose a shared adapter before adding a second host.
---
# Kubernetes and Helm (https://sockudo.io/docs/deployment/kubernetes)
> Deploy Sockudo with production Helm values, safe probes, disruption controls, ingress, and node tuning.
This guide deploys Sockudo to Kubernetes with the official Helm chart, then covers the settings
that matter for long-lived connections: probes, drain timing, autoscaling, ingress, and node
tuning. It is for platform teams running several replicas who want repeatable rollouts and
failure-domain placement.
Kubernetes is a good Sockudo platform when you need repeatable rollouts, failure-domain placement,
secret injection, metrics discovery, and several replicas. It does not remove the state model:
pods still need shared adapters and stores, and scale-down still disconnects their clients.
## Install the chart
The chart is published as an OCI artifact at `oci://ghcr.io/sockudo/charts/sockudo`. The release
pipeline packages it with the same version as the Sockudo release, so chart `5.2.0` deploys image
`5.2.0` by default.
1. Create the namespace and the Secrets that the [production values](#production-values-example)
reference. Write each value to a file first so it never appears in shell history or a manifest:
```bash
kubectl create namespace sockudo
kubectl create secret generic sockudo-app -n sockudo \
--from-file=default-app-id=./secrets/app-id \
--from-file=default-app-key=./secrets/app-key \
--from-file=default-app-secret=./secrets/app-secret
kubectl create secret generic sockudo-redis -n sockudo \
--from-file=redis-password=./secrets/redis-password
```
`--from-file` stores the exact bytes, including a trailing newline. Create the files without
one, for example `openssl rand -hex 32 | tr -d '\n' > secrets/app-secret`.
Verify with `kubectl get secret sockudo-app -n sockudo -o jsonpath='{.data}' | jq 'keys'`, which
lists `default-app-id`, `default-app-key`, and `default-app-secret`.
2. Save the [production values](#production-values-example) as `values-production.yaml` and adapt
hosts, Redis address, and limits.
3. Install or upgrade the release:
```bash
helm upgrade --install sockudo oci://ghcr.io/sockudo/charts/sockudo \
--version 5.2.0 \
--namespace sockudo \
--create-namespace \
--values values-production.yaml
```
4. Wait for the rollout:
```bash
kubectl rollout status deploy/sockudo -n sockudo
```
Expect `deployment "sockudo" successfully rolled out`. Then run the checks in
[Verify the deployed state](#verify-the-deployed-state).
Pin the chart and image version in GitOps or release automation. Do not deploy `latest`.
To work on the chart itself, install from a checkout instead: `helm upgrade --install sockudo
./charts/sockudo`.
## ArgoCD
The chart is published as an OCI artifact, so ArgoCD consumes it directly without cloning this
repository:
```yaml title="sockudo-application.yaml"
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sockudo
namespace: argocd
spec:
project: default
source:
repoURL: ghcr.io/sockudo/charts
chart: sockudo
targetRevision: 5.2.0
helm:
valuesObject:
replicaCount: 3
destination:
server: https://kubernetes.default.svc
namespace: sockudo
syncPolicy:
syncOptions:
- CreateNamespace=true
```
Note `repoURL` carries no `oci://` prefix: for a Helm chart source, ArgoCD takes the bare
registry path and infers the OCI protocol from `chart` being set.
## Configuration from a Secret
By default the chart renders the `config.*` values into a ConfigMap and mounts it at
`/app/config/config.json`. Under GitOps that ConfigMap is generated from values held in a
repository, so it can only carry configuration that is safe to commit.
Per-app webhooks are the case where it is not. Webhooks are expressible only in the config file,
never through environment variables, so an app that has them must carry its secret in the same
file. `defaultApp.existingSecret` does not help: it configures the environment-defined default
app, which has no webhook support.
Point `config.existingSecret` at a Secret to supply the whole file instead:
```yaml
config:
existingSecret: sockudo-config
existingSecretKey: config.json
```
The ConfigMap is then not rendered, and the `config` volume is backed by the Secret. Combining
`config.existingSecret` with `configJson` fails the render rather than silently ignoring one of
them. The `config.*` keys that the chart also emits as environment variables, such as the driver
selections and `shutdownGracePeriod`, still apply and override the file.
The content must be JSON: it is mounted as `config.json` and Sockudo selects its parser from the
file extension, even though the annotated reference configuration is TOML.
Helm never sees the Secret's content, so the `checksum/config` pod annotation is not emitted in
this mode. Rotating the Secret updates the mounted file but does not restart the pods, and Sockudo
reads its configuration only at startup - use a reloader controller or restart the Deployment.
This is the shape that pairs with a secret operator. With External Secrets Operator, an
`ExternalSecret` pulls the config file out of the backing store and the chart mounts the Secret it
produces:
```yaml title="sockudo-config-externalsecret.yaml"
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
name: sockudo-config
namespace: sockudo
spec:
refreshInterval: 5m
secretStoreRef:
kind: ClusterSecretStore
name: my-secret-store
target:
name: sockudo-config
creationPolicy: Owner
template:
engineVersion: v2
data:
config.json: "{{ .sockudoConfig }}"
data:
- secretKey: sockudoConfig
remoteRef:
key: SOCKUDO_CONFIG_FILE
```
Sockudo replaces `${VAR}` placeholders in the config file with environment values before parsing.
A `configJson` that contains `"secret": "${WEBHOOK_APP_SECRET}"` stays safe to commit, and an
`extraEnvFrom` Secret reference supplies the value. An unset variable without a `${VAR:-default}`
fallback stops startup with an error that names it. See
[environment placeholders](/docs/deployment/static-configuration#environment-placeholders-in-config-files).
## Production values example
The following is a regional three-pod baseline with one immutable app sourced from a Kubernetes
Secret and Redis used for fanout, cache, queue, and rate limits:
```yaml title="values-production.yaml"
replicaCount: 3
image:
repository: ghcr.io/sockudo/sockudo
tag: "5.2.0"
pullPolicy: IfNotPresent
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 25%
config:
mode: production
debug: false
logOutputFormat: json
rustLog: "info,sockudo=info"
shutdownGracePeriod: 40
adapterDriver: redis
appManagerDriver: memory
cacheDriver: redis
queueDriver: redis
rateLimiterDriver: redis
rateLimiter:
enabled: true
apiMaxRequests: 1000
apiWindowSeconds: 60
apiTrustHops: 1
wsMaxRequests: 100
wsWindowSeconds: 60
wsTrustHops: 1
metrics:
enabled: true
defaultApp:
enabled: true
existingSecret: sockudo-app
maxConnections: 50000
enableClientMessages: false
enableUserAuthentication: true
redis:
host: redis.internal
port: 6379
existingSecret: sockudo-redis
extraEnv:
# Chart default allocator tuning; keep it, because setting extraEnv replaces the list.
- name: _RJEM_MALLOC_CONF
value: "background_thread:true,dirty_decay_ms:2000,muzzy_decay_ms:0,tcache_max:4096,abort_conf:true"
- name: SOCKUDO_MAX_CONNECTIONS
value: "50000"
- name: ADAPTER_FALLBACK_TO_LOCAL
value: "false"
- name: SOCKUDO_DEFAULT_APP_ALLOWED_ORIGINS
value: "https://app.example.com"
- name: INSTANCE_PROCESS_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: "1"
memory: 1Gi
limits:
memory: 2Gi
readinessProbe:
httpGet:
path: /up/production
port: http
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
livenessProbe:
httpGet:
path: /live
port: http
periodSeconds: 20
timeoutSeconds: 3
failureThreshold: 3
startupProbe:
httpGet:
path: /live
port: http
periodSeconds: 5
timeoutSeconds: 3
failureThreshold: 30
terminationGracePeriodSeconds: 120
pdb:
enabled: true
minAvailable: 2
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: sockudo
serviceMonitor:
enabled: true
interval: 30s
scrapeTimeout: 10s
ingress:
enabled: true
className: nginx
annotations:
nginx.ingress.kubernetes.io/proxy-read-timeout: "180"
nginx.ingress.kubernetes.io/proxy-send-timeout: "180"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
hosts:
- host: ws.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: sockudo-tls
hosts:
- ws.example.com
```
What the less obvious values do:
- `defaultApp.existingSecret` reads the keys `default-app-id`, `default-app-key`, and
`default-app-secret`. If the ID or key is missing, the app is not registered.
- `redis.existingSecret` must contain `redis-password`; `redis-username` is optional.
- `defaultApp.maxConnections` becomes `SOCKUDO_DEFAULT_APP_MAX_CONNECTIONS`, the per-app quota.
`SOCKUDO_MAX_CONNECTIONS` is the separate per-pod cap across all apps.
- `ADAPTER_FALLBACK_TO_LOCAL=false` makes a pod that cannot reach Redis at startup fail instead of
running as an isolated node. The default is `true`.
- `INSTANCE_PROCESS_ID` from the pod name gives each pod a unique, readable identity. The chart sets
the same variable itself when `cluster.enabled` is true; do not define it twice in that case.
- `strategy` surges new pods before removing old ones; see [Rollouts and disruptions](#rollouts-and-disruptions).
Helm replaces lists instead of merging them. The chart's default `extraEnv` sets
`_RJEM_MALLOC_CONF` to tune the jemalloc allocator; any `extraEnv` you provide drops it unless you
copy it in, as the example does.
Adapt the topology label selector when the release name or chart labels differ. If application
records must change dynamically, select a durable app manager instead of `memory` and configure its
database credentials from a Secret.
For advanced nested configuration, use the chart's `configJson` value or mount your own static
file. Keep secret values out of `configJson`; combine it with `extraEnvFrom`, `${VAR}`
placeholders, or file-backed Secrets.
## Probes
Use different endpoints for different decisions:
| Probe | Endpoint | Meaning |
| --- | --- | --- |
| Startup | `/live` | The process has started. |
| Liveness | `/live` | The process runtime is alive. |
| Readiness | `/up` or `/up/` | Required apps and shared dependencies are available. |
| Capacity-aware readiness | `/ready` | Same dependency checks as `/up`, plus `503 CAPACITY` when the pod nears `SOCKUDO_MAX_CONNECTIONS`. |
| New-connection admission | `/accept-traffic` | The pod is not draining or shedding new connections under memory pressure. |
Do not use dependency-aware `/up` as liveness. Restarting every pod because Redis is slow creates a
reconnect storm and does not repair Redis.
`/ready` goes unavailable at the high watermark (default 95% of `SOCKUDO_MAX_CONNECTIONS`) and
recovers at the low watermark (default 85%), so a full pod stops receiving new clients without
flapping. Tune it with `config.httpApi.readiness.highWatermark` and `lowWatermark`, which the chart
writes to its generated ConfigMap; with `configJson` or `config.existingSecret`, set
`http_api.readiness` in your file instead. With a zero
connection limit the capacity gate is disabled. Use `/ready` for readiness when you want full pods
taken out of rotation; keep `/up` when rollout health must succeed even on a full pod.
Set probe timeout below its period and above normal dependency-check latency. Sockudo's
`health_check_timeout_ms` (default `2000`) bounds each dependency check separately.
## Resources and connection admission
Persistent connections do not map cleanly onto CPU-only autoscaling:
- memory rises with sockets, subscriptions, pending writes, and per-connection feature state
- CPU rises with handshake, auth, messages, fanout, compression, filtering, and reconnect bursts
- a pod can have low CPU while holding too many quiet sockets to remove safely
Set `SOCKUDO_MAX_CONNECTIONS` on every pod so a single replica cannot absorb more connections than
its measured memory and file-descriptor budget. Keep minimum replicas high enough for the peak
quiet-socket count even when CPU is low.
For a second guard based on actual memory use, enable the chart's memory-pressure admission control:
```yaml
config:
httpApi:
acceptTraffic:
enabled: true
memoryThreshold: 0.9
sampleIntervalMs: 500
```
With `memoryLimitBytes: null` (the default), Sockudo discovers the pod's finite cgroup memory limit.
When process RSS reaches the configured fraction, the pod rejects only new native and Ably
WebSocket connections with `503`/retry guidance while established sessions continue. It resumes
admission after a later sample falls below the threshold. The sampler is independent of metrics
scrapes and fails open if Linux RSS or cgroup data is unavailable.
For example (illustrative): with `limits.memory: 2Gi` and `memoryThreshold: 0.9`, new connections
are refused once RSS reaches about 1.8 GiB, leaving roughly 200 MiB for established sessions to
absorb a fanout burst before the OOM killer would disconnect all of them.
Keep `/up` as Kubernetes readiness so dependency failures remain visible. If the ingress or external
load balancer supports a separate backend-admission or network-watcher probe, point that probe at
`/accept-traffic`. Direct admission enforcement remains active even without such a probe.
CPU limits can throttle a pod during the exact reconnect or fanout burst it needs to process.
Requests plus a memory limit are a reasonable starting point on a dedicated node pool. Add a CPU
limit only when isolation requires it and benchmark with the same quota.
## Autoscaling
The chart can configure an HPA:
```yaml
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 12
targetCPUUtilizationPercentage: 65
targetMemoryUtilizationPercentage: 75
behavior:
scaleUp:
stabilizationWindowSeconds: 30
scaleDown:
stabilizationWindowSeconds: 600
policies:
- type: Pods
value: 1
periodSeconds: 120
```
When `autoscaling.enabled` is true, the chart omits `replicas` from the Deployment and the HPA owns
the replica count.
CPU and memory are fallback signals, not a complete realtime scaling policy. A stronger setup also
uses active connections per pod, connection admission rejections, event-loop or fanout latency,
adapter backlog, and reconnect rate through an external/custom metrics adapter.
`autoscaling.metrics` is passed through to the HPA verbatim and accepts any `autoscaling/v2` metric
type, so those signals can be expressed directly. `sockudo_connected` is the connection gauge:
```yaml
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 12
metrics:
- type: External
external:
metric:
name: sockudo_connected
target:
type: AverageValue
averageValue: "40000"
```
An `External` metric requires a metrics adapter, such as Prometheus Adapter or KEDA, that exposes
`sockudo_connected` to the Kubernetes external metrics API.
For example (illustrative): with `SOCKUDO_MAX_CONNECTIONS=50000`, a target of `40000` adds pods at
80% connection utilization, before `/ready` reports `CAPACITY` at 47,500 and well before admission
rejects at 50,000.
Setting it **replaces** the CPU and memory metrics generated from
`targetCPUUtilizationPercentage` and `targetMemoryUtilizationPercentage` rather than adding to
them. That is deliberate: an HPA acts on the highest recommendation from any metric, so leaving
CPU in place would keep overriding a connection-based policy. Put CPU back in the list explicitly
if you want both.
Scale up early. Scale down slowly and one pod at a time. Every removed pod makes its clients
reconnect, so an aggressive HPA can oscillate and manufacture load.
## Rollouts and disruptions
Set:
- a PodDisruptionBudget that leaves enough connection capacity online
- zone or host topology spread
- a termination grace period covering listener grace, webhook drain, queue worker shutdown,
and teardown
- a rollout `maxUnavailable` compatible with the PDB and remaining socket capacity
- load-balancer deregistration or endpoint propagation time inside the drain budget
On SIGTERM, Sockudo rejects new WebSocket upgrades, closes existing connections with code 4200 so
clients reconnect right away, and keeps the listener open for `SHUTDOWN_GRACE_PERIOD` seconds
while `/up` and `/ready` return 503 `DRAINING`:
```mermaid
sequenceDiagram
participant K as Kubelet
participant P as Old pod
participant L as Load balancer
participant C as Clients
participant N as Other pods
K->>P: SIGTERM
P-->>L: /ready returns 503 DRAINING
Note over P,L: wait shutdown_close_delay_ms
P->>C: Close frame 4200
C->>L: Reconnect
L->>N: Route to ready pods
Note over K,P: listener open until grace ends, then webhook and queue drain
```
By default the close happens immediately and to every connection at once.
`SHUTDOWN_CLOSE_DELAY_MS` waits first, so the load balancer can observe the 503 before anyone
reconnects, and `SHUTDOWN_CLOSE_SPREAD_MS` spreads the closes over a window so the remaining pods
are not hit by one simultaneous reconnect wave. Delay plus spread must fit inside
`SHUTDOWN_GRACE_PERIOD`; set them through `extraEnv`. See
[rolling deploys](/docs/server/scaling#rolling-deploys) for the full sequence.
Set `SHUTDOWN_GRACE_PERIOD` (chart value `config.shutdownGracePeriod`) above the readiness probe's
`periodSeconds × failureThreshold` plus endpoint propagation time, and keep
`terminationGracePeriodSeconds` above `SHUTDOWN_GRACE_PERIOD` plus twice
`QUEUE_SHUTDOWN_TIMEOUT_MS / 1000`, with additional time for connection and backend teardown.
The two budgets cover [webhook draining](/docs/server/webhooks#shutdown-delivery) and queue worker
shutdown. An early SIGKILL can lose buffered lifecycle events.
For example (illustrative), with the production values above:
```text
readiness detection = periodSeconds 10 × failureThreshold 3 = 30 s
SHUTDOWN_GRACE_PERIOD > 30 s + ~10 s endpoint propagation → 40 s
drain budget = 40 s + 2 × (QUEUE_SHUTDOWN_TIMEOUT_MS 30000 / 1000) = 100 s
terminationGracePeriodSeconds = 100 s + 20 s teardown margin → 120 s
```
The chart defaults (`shutdownGracePeriod: 10`, `terminationGracePeriodSeconds: 30`) are sized for
development; raise both for production.
`strategy` is passed through to the Deployment. Left unset, Kubernetes applies its default of
`maxUnavailable: 25%` / `maxSurge: 25%`, which takes a quarter of the connection capacity offline
during a rollout - exactly while the clients from the replaced pods are reconnecting. Surging
instead keeps capacity flat:
```yaml
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 25%
```
A PodDisruptionBudget governs the eviction API - node drains and upgrades - not Deployment
rollouts. Rollout safety comes from `strategy`.
During a rollout, watch connection count by pod. Kubernetes considers replica count, not how many
sockets each replica owns. One terminating pod may hold a disproportionate share after a long
uptime.
If strict connection drain is required, remove readiness first, wait for endpoint and load-balancer
propagation, then terminate. Test this behavior with the actual ingress controller; it is not
identical across controllers.
## Ingress and load balancers
The production example's `apiTrustHops: 1` and `wsTrustHops: 1` assume an ingress such as NGINX
that appends one client address to `X-Forwarded-For`. These values select the client's position
from the right, not a universal proxy count. A Google external Application Load Balancer directly
in front of Sockudo appends two addresses and needs `2`. Restrict pod access to trusted ingress
sources before enabling either value. The default `0` ignores both `X-Forwarded-For` and
`X-Real-IP`; see [client IP extraction](/docs/reference/configuration#rate-limiter-client-ip).
Confirm all layers:
- accept HTTP/1.1 WebSocket upgrades
- preserve `Upgrade` and `Connection` semantics
- have idle timeouts above the heartbeat interval with margin
- do not buffer WebSocket traffic
- expose a dependency-aware readiness path
- drain targets during pod termination
The ingress annotation example above is specific to ingress-nginx. Use the equivalent settings for
AWS Load Balancer Controller, Google Cloud Load Balancing, Azure Application Gateway, Traefik,
Envoy, HAProxy, or a service mesh. The chart also has a separate `wsIngress` block for controllers
that need different annotations for WebSocket traffic.
Sticky sessions are not required for basic pub/sub with a shared adapter. They can reduce
reconnect-to-new-node churn, but must not substitute for shared state.
When preserving client IPs with `externalTrafficPolicy: Local` or proxy-protocol features, retest
load distribution and health checks. Source-IP preservation can reduce the set of eligible nodes.
## Pod and node sysctls
First deploy with provider defaults and inspect counters. Managed Kubernetes node images already
ship with non-trivial tuning, and an unnecessary override can reduce stability.
Kubernetes classifies sysctls as safe or unsafe. Safe, namespaced values can be applied in the pod
security context when the cluster version and admission policy allow them:
```yaml
podSecurityContext:
runAsNonRoot: true
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
sysctls:
- name: net.ipv4.tcp_keepalive_time
value: "600"
- name: net.ipv4.tcp_keepalive_intvl
value: "60"
- name: net.ipv4.tcp_keepalive_probes
value: "5"
```
`net.core.somaxconn` is commonly treated as unsafe. A pod-level value requires both:
1. an exact kubelet `allowedUnsafeSysctls` entry on every eligible node
2. the matching `podSecurityContext.sysctls` value and an admission policy that permits it
See Kubernetes' [sysctl documentation](https://kubernetes.io/docs/tasks/administer-cluster/sysctl-cluster/)
and the provider-specific node-pool guide before enabling it. Do not grant a privileged container
or broad `net.*` permission to set one value.
Node-level sysctl configuration and pod network namespaces are distinct. Verify the effective value
inside the scheduled pod:
```bash
kubectl exec -n sockudo deploy/sockudo -- cat /proc/sys/net/core/somaxconn
kubectl exec -n sockudo deploy/sockudo -- sh -c 'ulimit -n; cat /proc/1/limits'
```
Kubernetes has no portable pod field for POSIX `rlimit` values. The effective `nofile` comes from
the container runtime and node image. If it is too low, change the runtime/node configuration or
use a provider-supported node image customization, then verify again inside the pod.
Provider examples are in [Cloud platforms](/docs/deployment/cloud-platforms).
## Network capacity
High socket counts can hit infrastructure before the pod:
- load-balancer connection and target limits
- node vNIC packets-per-second or bandwidth limits
- CNI IP allocation and conntrack tables
- NAT gateway state for outbound dependencies or load generators
- cross-zone traffic and broker latency
Inbound WebSockets do not consume a Sockudo pod's local ephemeral port range. Avoid “fixing”
`ip_local_port_range` unless evidence points to an outbound connection or NAT bottleneck.
## Verify the deployed state
1. Confirm the values Helm applied and where the pods landed:
```bash
helm get values sockudo -n sockudo
kubectl get pods -n sockudo -o wide
kubectl get pdb -n sockudo
```
Every pod is `Running` and `READY 1/1`, spread across zones, and the PDB shows
`MIN AVAILABLE 2`.
2. Check the file limit inside a pod:
```bash
kubectl exec -n sockudo deploy/sockudo -- sh -c 'ulimit -n'
```
The value must exceed `SOCKUDO_MAX_CONNECTIONS` plus reserve.
3. Check health through the Service:
```bash
kubectl port-forward -n sockudo svc/sockudo 6001:6001
curl -fsS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:6001/live
curl -fsS http://127.0.0.1:6001/up/production
```
`/live` prints `200` and `/up/production` prints `OK`.
4. Check metrics through the metrics Service:
```bash
kubectl port-forward -n sockudo svc/sockudo-metrics 9601:9601
curl -fsS http://127.0.0.1:9601/metrics | grep '^sockudo_connected'
```
The output contains `sockudo_connected` samples.
5. Run a node drain and a rolling restart under load, and watch reconnects and recovery. A manifest
that renders successfully is not yet a tested realtime deployment.
## Next steps
- [Cloud platforms](/docs/deployment/cloud-platforms): Node-pool tuning on EKS, GKE, and AKS, and managed container limits.
- [Capacity planning](/docs/deployment/capacity-planning): Measure per-pod limits before setting SOCKUDO_MAX_CONNECTIONS and HPA targets.
- [Scaling](/docs/server/scaling): Rolling deploy sequence, duplicate delivery, and recovery across nodes.
- [Observability](/docs/server/observability): Metrics, dashboards, and alerts for the deployed cluster.
---
# Cloud platforms (https://sockudo.io/docs/deployment/cloud-platforms)
> Deploy Sockudo on AWS, Google Cloud, and Azure with managed backends, WebSocket load balancing, and provider-supported node tuning.
This guide maps the Sockudo deployment model onto AWS, Google Cloud, and Azure: which compute
shape to pick, how to configure the provider's load balancer for WebSockets, and how to tune
managed Kubernetes nodes. Read [Kubernetes and Helm](/docs/deployment/kubernetes) or
[Linux VM and bare metal](/docs/deployment/linux) first; this page adds the provider-specific
parts.
Sockudo uses the same runtime model on every cloud: long-lived client connections at the edge,
shared fanout and coordination between instances, and optional durable stores. Provider services
change the failure modes and tuning controls, not those requirements.
```mermaid
flowchart TB
C[Clients] --> LB[Regional load balancer]
LB --> ZA[Sockudo zone A]
LB --> ZB[Sockudo zone B]
ZA <--> R[Managed Redis]
ZB <--> R
ZA --> DB[Managed database]
ZB --> DB
```
Whatever the provider, the same three questions decide the shape:
1. **How long can a connection stay open?** Managed container platforms often cap request
duration, which forces periodic reconnects.
2. **Do you control the node?** Kernel sysctls and process file limits are only tunable on VMs and
self-configured Kubernetes node pools.
3. **Where does shared state live?** Managed Redis and databases should sit in the same region as
Sockudo, on a private network.
## Platform map
| Need | AWS | Google Cloud | Azure |
| --- | --- | --- | --- |
| Direct VM control | EC2 Auto Scaling group | Compute Engine managed instance group | Virtual Machine Scale Sets |
| Managed Kubernetes | EKS | GKE Standard or Autopilot | AKS |
| Managed containers | ECS, including Fargate | Cloud Run | Azure Container Apps |
| WebSocket ingress | ALB or NLB | External Application Load Balancer | Application Gateway, Load Balancer, or platform ingress |
| Redis-compatible shared state | Managed Redis-compatible service | Memorystore for Redis | Managed Redis-compatible service |
| Relational app/history store | RDS or Aurora | Cloud SQL or AlloyDB | Azure Database for PostgreSQL/MySQL |
| Native fanout option | Redis, MSK, or self-managed NATS | Google Pub/Sub adapter | Redis, Kafka-compatible service, or self-managed NATS |
The table is a starting map, not a requirement to use every managed service. Keep latency-sensitive
dependencies in the same region as Sockudo and test cross-zone or cross-region traffic costs.
## AWS
### Recommended shapes
| Shape | Use when | Notes |
| --- | --- | --- |
| EC2 + ALB/NLB | You need full kernel, `rlimit`, NIC, and process control. | Best match for very high connection density and stable workloads. |
| EKS managed node group | You already operate Kubernetes and need pod-level rollout and placement. | Use a dedicated node pool when tuning or noisy neighbors matter. |
| ECS on EC2 | You want task scheduling with host control. | Set task `ulimits`; tune the EC2 host separately. |
| ECS Fargate | Moderate workloads and low node-operations overhead. | Validate supported `ulimits`, system controls, ENI, and per-task connection density. |
For a regional cluster, put Sockudo targets in at least two Availability Zones. Use one shared
adapter and cache across all replicas. Keep a minimum target count available during deployments.
### Application Load Balancer
Application Load Balancers support WebSockets natively. Their idle timeout defaults to 60 seconds,
which is too close to many heartbeat configurations. Set it deliberately and test through every
proxy layer:
1. Raise the idle timeout above Sockudo's 120-second default `activity_timeout`, with margin:
```bash
aws elbv2 modify-load-balancer-attributes \
--load-balancer-arn "$SOCKUDO_ALB_ARN" \
--attributes Key=idle_timeout.timeout_seconds,Value=180
```
2. Confirm the attribute:
```bash
aws elbv2 describe-load-balancer-attributes \
--load-balancer-arn "$SOCKUDO_ALB_ARN" \
--query "Attributes[?Key=='idle_timeout.timeout_seconds'].Value"
```
The output is `["180"]`.
3. Hold a quiet WebSocket open through the public endpoint for longer than 60 seconds and confirm it
is not closed by the load balancer.
AWS documents the default, valid range, and keepalive interaction in
[Application Load Balancer attributes](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/edit-load-balancer-attributes.html).
ALB access logs identify `ws` and `wss` requests and are useful for separating target disconnects
from client-side failures.
Configure:
- target health on `/up/` (or the capacity-aware `/ready`) and process monitoring on `/live`
- deregistration delay and service termination grace as one drain budget
- security groups that expose the Sockudo port only to the load balancer
- metrics on target connection errors, rejected connections, and 5xx responses
### EKS on Amazon Linux 2023
Amazon EKS managed node groups that use launch templates require MIME multipart user data for
Amazon Linux AMIs. Amazon Linux 2023 uses `nodeadm` and the `application/node.eks.aws` content type.
AWS documents the merge behavior and required self-managed fields in
[Customize managed nodes with launch templates](https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html).
This example applies a conservative node profile and allows a pod to request the otherwise unsafe
`net.core.somaxconn` sysctl:
```text
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="SOCKUDO"
--SOCKUDO
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
cat > /etc/sysctl.d/99-sockudo.conf <<'EOF'
net.core.somaxconn = 65535
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_slow_start_after_idle = 0
EOF
sysctl --system
--SOCKUDO
Content-Type: application/node.eks.aws
apiVersion: node.eks.aws/v1alpha1
kind: NodeConfig
spec:
kubelet:
config:
allowedUnsafeSysctls:
- "net.core.somaxconn"
--SOCKUDO--
```
For a managed node group, EKS merges its bootstrap data with the launch-template user data. For a
self-managed AL2023 group or a custom AMI workflow, include the required `spec.cluster.name`,
`apiServerEndpoint`, base64 certificate authority, and service CIDR described by AWS.
The two parts do different jobs:
- the shell script changes the node-wide baseline
- `allowedUnsafeSysctls` only permits a pod to request that exact sysctl in its own security context
If the pod does not request an unsafe sysctl, the allowlist alone changes nothing. Verify effective
values inside the Sockudo pod.
Do not add `/etc/security/limits.d` to EKS user data expecting it to raise a container's `nofile`.
PAM limits do not control the existing container process. Check `/proc/1/limits` inside the pod and
customize the node runtime only if the effective value is too low.
Apply this profile to a dedicated, canary node group first. A launch-template or node configuration
change replaces nodes and causes pod movement, and every moved pod disconnects its clients.
### AWS service choices
- a managed Redis-compatible service is the simplest shared adapter, cache, queue, and rate-limit
authority for most clusters
- RDS or Aurora PostgreSQL/MySQL can hold dynamic apps and supported durable state
- DynamoDB is available for supported app/history/version paths when the matching feature is built
- use private subnets and security groups; broker and database ports should not be public
- use workload identity or instance/task roles instead of long-lived AWS access keys
## Google Cloud
### Recommended shapes
| Shape | Use when | Notes |
| --- | --- | --- |
| Compute Engine managed instance group | Maximum connection density and host control. | Pair with an external load balancer and health checks. |
| GKE Standard | Kubernetes with node-pool and sysctl control. | Best GKE fit for dedicated realtime nodes. |
| GKE Autopilot | Lower cluster operations overhead. | Node configuration is more constrained; verify all required controls. |
| Cloud Run | Moderate connection count with forced reconnect tolerance. | WebSockets are requests with a finite timeout and per-instance concurrency. |
Google Pub/Sub is a supported horizontal adapter, but durable feature coordination still needs the
documented shared cache and stores. Choose it when managed GCP fanout fits the latency and request
pattern; do not assume it replaces every Redis responsibility.
### GKE node system configuration
GKE Standard supports a node system configuration file for kubelet and a defined set of Linux
sysctls. Google warns that changing it recreates nodes and recommends testing with a PodDisruption
Budget. See [Customizing node system configuration](https://cloud.google.com/kubernetes-engine/docs/how-to/node-system-config).
```yaml
kubeletConfig:
allowedUnsafeSysctls:
- "net.core.somaxconn"
linuxConfig:
sysctl:
net.core.somaxconn: "65535"
net.core.netdev_max_backlog: "16384"
```
Create a dedicated node pool with the file:
```bash
gcloud container node-pools create sockudo \
--cluster="$GKE_CLUSTER" \
--location="$GKE_LOCATION" \
--system-config-from-file=gke-sockudo-system.yaml
```
GKE accepts only its documented sysctl set and ranges. Check the current provider table before
adding a value. Use an exact unsafe allowlist rather than `net.*`, then set the matching pod sysctl
only when the node baseline is insufficient.
After pods are scheduled on the new pool, verify the effective value inside a pod:
```bash
kubectl exec -n sockudo deploy/sockudo -- cat /proc/sys/net/core/somaxconn
```
Pin Sockudo to the pool with a `nodeSelector` such as `cloud.google.com/gke-nodepool: sockudo` in
the chart values.
### Cloud Run
Cloud Run supports WebSockets, but treats them as long-running HTTP requests. Google currently
documents a request timeout of up to 60 minutes, best-effort session affinity, and up to 1000
concurrent connections per container. See
[Using WebSockets on Cloud Run](https://cloud.google.com/run/docs/triggering/websockets).
A WebSocket on Cloud Run ends when its request timeout expires, however healthy it is. With a
60-minute timeout, every client reconnects at least once an hour. Design for it: shared state across
instances and, where continuity matters, Protocol V2 recovery.
For Sockudo that means:
- configure the maximum request timeout and expect forced reconnects
- use a horizontal adapter and shared stores because a reconnect may reach another instance
- enable and test Protocol V2 recovery when continuity matters
- set minimum instances for latency and capacity rather than relying on a cold scale-out
- set maximum concurrency only after measuring memory per socket and burst CPU
- do not enable end-to-end HTTP/2 for the WebSocket service
- account for active WebSocket instances in the billing model
Cloud Run is useful when those constraints are acceptable. GKE Standard or Compute Engine is a
better fit when connections should remain open indefinitely or node-level tuning is required.
### Google Cloud service choices
- Memorystore for Redis can provide shared Redis state close to GKE or Compute Engine
- Cloud SQL or AlloyDB can provide supported relational app and durable stores
- workload identity avoids shipping service-account JSON in images
- place Sockudo, Redis, and database services in compatible regions and private networks
- report load-balancer and Cloud NAT limits separately from pod or VM limits
## Azure
### Recommended shapes
| Shape | Use when | Notes |
| --- | --- | --- |
| Virtual Machine Scale Sets | Full host control and dense, stable sockets. | Configure load balancing, health probes, and rolling upgrades explicitly. |
| AKS | Kubernetes rollout, node pools, identities, and managed control plane. | Use a dedicated Linux node pool for custom OS or kubelet settings. |
| Azure Container Apps | Managed container ingress for moderate workloads. | WebSocket ingress is supported; validate request/idle timeout and scaling behavior. |
Azure Application Gateway and Azure Load Balancer have different layers, health semantics, and
timeout controls. Document the selected path and test a quiet WebSocket through the public endpoint,
not only a busy local client.
### AKS custom node configuration
AKS accepts separate kubelet and Linux OS configuration files when creating a cluster or node pool.
Microsoft documents the supported ranges and notes that unsafe sysctls can affect node security and
stability in [Customize AKS node configuration](https://learn.microsoft.com/azure/aks/custom-node-configuration).
`aks-kubelet.json`:
```json
{
"allowedUnsafeSysctls": [
"net.core.somaxconn"
]
}
```
`aks-linux-os.json`:
```json
{
"sysctls": {
"netCoreSomaxconn": 65535,
"netCoreNetdevMaxBacklog": 16384,
"netIpv4TcpMaxSynBacklog": 65535,
"netIpv4TcpKeepaliveTime": 600,
"netIpv4TcpKeepaliveIntvl": 60,
"netIpv4TcpKeepaliveProbes": 5
}
}
```
Create a dedicated pool:
```bash
az aks nodepool add \
--resource-group "$AKS_RESOURCE_GROUP" \
--cluster-name "$AKS_CLUSTER" \
--name sockudo \
--kubelet-config ./aks-kubelet.json \
--linux-os-config ./aks-linux-os.json
```
AKS uses camelCase names in the JSON API even though the documentation tables show kernel names.
Provider defaults can already be high; for example, system-wide file handle ceilings on current
AKS Linux images may not be the limiting `nofile`. Always inspect the process limit inside the pod.
Verify the node pool and the effective values once pods are scheduled on it:
```bash
az aks nodepool show \
--resource-group "$AKS_RESOURCE_GROUP" \
--cluster-name "$AKS_CLUSTER" \
--name sockudo \
--query "linuxOsConfig.sysctls"
kubectl exec -n sockudo deploy/sockudo -- cat /proc/sys/net/core/somaxconn
```
### Azure Container Apps
Azure Container Apps ingress supports WebSockets. Environment ingress settings can include an idle
request timeout, and the platform owns the underlying nodes. See
[Container Apps ingress](https://learn.microsoft.com/azure/container-apps/ingress-overview) and
[environment ingress configuration](https://learn.microsoft.com/azure/container-apps/ingress-environment-configuration).
Use the same managed-container precautions as Cloud Run:
- shared fanout and coordination from the first multi-replica deployment
- minimum replicas sized for quiet socket capacity
- tested reconnect and recovery behavior
- no dependency on host sysctl or custom container-runtime limits
- explicit ingress idle-timeout validation in the selected environment tier
Use AKS or Virtual Machine Scale Sets when connection density requires host control.
### Azure service choices
- use a managed Redis-compatible service for the common shared adapter/cache path
- Azure Database for PostgreSQL or MySQL can provide supported relational storage
- use managed identities and Key Vault-backed secret delivery instead of static cloud credentials
- use private endpoints or VNet integration for Redis and databases
- watch SNAT/connection tracking for outbound dependencies and load generators
## Other providers
The same decision tree works on DigitalOcean, Hetzner, Oracle Cloud, Fly.io, Render, Railway, or an
on-premises Kubernetes platform:
1. Confirm native WebSocket support and the maximum idle or request duration.
2. Confirm per-instance concurrent connection, file descriptor, bandwidth, and packet limits.
3. Confirm whether you control node sysctls and process `rlimit`.
4. Confirm scale-down and connection-drain semantics.
5. Use a shared adapter before adding a second instance.
6. Keep stateful dependencies close enough for the measured latency budget.
7. Run a quiet-socket soak and reconnect storm, not only a short message-throughput test.
If the provider does not publish a socket or timeout limit, treat it as an unknown to test and
monitor, not as unlimited.
## Production checklist
On any provider, confirm before taking traffic:
- Sockudo runs in at least two zones behind one regional load balancer
- the load balancer idle timeout exceeds the client heartbeat interval, verified with a quiet socket
- target health uses `/up/` or `/ready`; process restarts use `/live`
- the Sockudo port accepts traffic only from the load balancer; metrics only from monitoring
- Redis and databases are private, in the same region, and reached through workload identity or
secrets, not long-lived keys in images
- `ADAPTER_FALLBACK_TO_LOCAL=false` on every replica
- node sysctls and the process `nofile` limit are verified inside a running instance or pod
- deregistration delay plus Sockudo's shutdown grace and drain budgets fit inside the platform's
termination window
## Next steps
- [Kubernetes and Helm](/docs/deployment/kubernetes): Chart values, probes, and drain timing for EKS, GKE, and AKS.
- [Capacity planning](/docs/deployment/capacity-planning): Size instances and run benchmarks from realistic client regions.
- [Scaling](/docs/server/scaling): Choose between Redis, NATS, Google Pub/Sub, and other adapters.
- [Security](/docs/server/security): Network boundaries, secrets, and origin policy.
---
# Capacity planning and benchmarks (https://sockudo.io/docs/deployment/capacity-planning)
> Size Sockudo from measured connection, fanout, memory, network, and failure budgets without benchmarking the load generator by accident.
This page shows how to turn a workload description into a safe per-node connection limit, and how
to benchmark Sockudo without measuring the load generator by accident. It is for anyone setting
`SOCKUDO_MAX_CONNECTIONS`, choosing instance sizes, or publishing benchmark numbers.
There is no single “connections per node” number. A quiet public-channel socket, a presence member,
a filtered Protocol V2 subscriber, and a slow client receiving large payloads have different memory,
CPU, and network costs. Capacity is the lowest limit across Sockudo, the process, the node, the load
balancer, shared backends, and the client generator.
## Name the workload
Always report at least these independent dimensions:
| Dimension | Example | Why it changes the result |
| --- | ---: | --- |
| Concurrent connected sockets | 100,000 | Drives file descriptors, memory, LB state, and heartbeat work. |
| New connections per second | 2,000/s | Drives TCP/TLS handshake, auth, and reconnect CPU. |
| Subscriptions per socket | 5 | Drives local channel state and adapter subscription behavior. |
| Subscribe/unsubscribe cycles | 500/s | Drives churn and presence/count coordination. |
| Backend publishes per second | 1,000/s | Measures ingest and adapter work. |
| Average subscribers per publish | 100 | Converts publishes into 100,000 client deliveries/s. |
| Payload size | 1 KiB | Drives serialization, buffers, broker traffic, and egress. |
| Private/presence percentage | 30% / 10% | Adds auth and membership work. |
| Protocol features | V2 recovery + deltas | Adds continuity, cache, compression, or storage work. |
| Client RTT and TLS | 180 ms, TLS 1.3 | Changes ramp, handshake, ack, and reconnect timing. |
Never publish “messages per second” without saying whether it means accepted publishes, adapter
messages, or delivered client frames.
## Capacity is the minimum budget
For one Sockudo node:
```text
safe connections = min(
Sockudo admission limit,
process file-descriptor budget,
measured memory budget,
measured CPU/latency budget,
node and vNIC limits,
load-balancer target limits,
shared-backend budget
) - operational headroom
```
Use 20–30% headroom after a stable test, more when traffic is bursty or failure of one node must be
absorbed by the others.
**Worked example (illustrative numbers).** Suppose measurements on one node give:
| Budget | Measured limit |
| --- | ---: |
| Process file-descriptor budget | 222,000 |
| Memory budget | 85,000 |
| CPU/latency budget (p99 still within target) | 70,000 |
| Node and vNIC limits | 150,000 |
| Load-balancer target limit | 100,000 |
| Shared-backend budget | 200,000 |
The minimum is the CPU/latency budget, 70,000. With 25% headroom the safe value is
`70,000 × 0.75 = 52,500`, so set `SOCKUDO_MAX_CONNECTIONS` to about 52,000. Raising the file limit
would not change this node's capacity; more CPU or a cheaper workload would.
For an `N`-node cluster that must survive one node loss, each node must be able to hold its share
of the peak with one node missing:
```text
per-node safe capacity >= cluster peak connections / (N - 1)
```
Equivalently, under normal operation each node should carry no more than
`(N - 1) / N` of its safe capacity.
**Worked example (illustrative numbers).** A peak of 150,000 connections on 4 nodes is 37,500 per
node in normal operation. After one node fails, the 3 survivors carry `150,000 / 3 = 50,000` each,
so every node's safe capacity must be at least 50,000. With the 52,500 limit from the previous
example, 4 nodes are enough; 3 nodes would need 75,000 each and are not.
That is an availability constraint, not only a scaling calculation. The remaining nodes also need
CPU, network, broker, and memory headroom for the reconnect burst.
## File descriptor budget
Estimate:
```text
nofile soft limit
- backend sockets
- listener, metrics, logging, and runtime descriptors
- 10–20% descriptor reserve
= maximum socket descriptor budget
```
**Worked example (illustrative numbers).** With a soft limit of 262,144, 200 backend sockets, 100
listener and runtime descriptors, and a 15% reserve (`262,144 × 0.15 ≈ 39,322`):
```text
262,144 - 200 - 100 - 39,322 = 222,522 socket descriptors
```
Then set `SOCKUDO_MAX_CONNECTIONS` below that result. Verify the actual process:
```bash
pid="$(pidof sockudo)"
cat "/proc/$pid/limits" | grep -i 'open files'
find "/proc/$pid/fd" -maxdepth 1 -type l | wc -l
```
In a container or pod, run the check inside that container. Host login limits do not prove the
container process limit.
## Measure memory per workload
Do not borrow a per-socket number from another server or another Sockudo feature set. Memory per
socket depends on subscriptions, presence membership, buffered writes, and which Protocol V2
features are enabled.
1. Start the exact release build, allocator, config, and container limit.
2. Hold the server idle and record steady resident memory, for example with
`ps -o rss= -p "$(pidof sockudo)"` (KiB) or the container's memory metric.
3. Ramp to a known socket and subscription count.
4. Wait for allocations, buffers, and metrics to settle.
5. Exercise the intended message, presence, filter, delta, and recovery workload.
6. Record both steady and peak resident memory.
7. Repeat at several counts; do not extrapolate from 100 sockets to 100,000.
Approximate the marginal result:
```text
marginal bytes per socket =
(steady RSS at high count - baseline RSS) / active sockets
```
**Worked example (illustrative numbers).** Baseline RSS is 80 MiB; at 50,000 active sockets steady
RSS is 1,280 MiB:
```text
(1,280 MiB - 80 MiB) / 50,000 = 1,258,291,200 B / 50,000 ≈ 25,166 B ≈ 24.6 KiB per socket
```
If peak RSS under the fanout workload reaches 1,600 MiB, size the memory limit from 1,600 MiB plus
margin, not from the steady figure.
Use peak RSS, not only marginal steady memory, when setting a cgroup or pod memory limit. Include
allocator fragmentation, slow-client buffers, adapter queues, recovery buffers, and rolling-deploy
overlap.
An OOM kill is a correlated disconnect event. Leave enough margin to avoid turning a transient
fanout burst into a reconnect storm.
## CPU and event throughput
Separate tests for:
- connection and TLS handshake rate
- public subscribe/unsubscribe churn
- private and presence authentication
- presence first-join and last-leave transitions
- backend publish admission
- fanout deliveries
- delta compression and tag filtering
- durable history and version writes
- replay and reconnect recovery
- webhook and push queue work
A workload at 1,000 publishes/s with one subscriber is not comparable to 1,000 publishes/s with
10,000 subscribers.
Approximate client delivery rate:
```text
deliveries/s = publishes/s × average matching subscribers per publish
```
**Worked example (illustrative numbers).** 1,000 publishes/s with an average of 100 matching
subscribers is `1,000 × 100 = 100,000` client deliveries/s. The same 1,000 publishes/s with 10,000
subscribers each is 10,000,000 deliveries/s, a completely different workload.
Adapter and serialization cost may scale with publishes while egress and per-client write cost scale
with deliveries.
## Network budget
Approximate payload egress before protocol, TLS, and TCP overhead:
```text
payload egress bytes/s =
publishes/s × matching subscribers × average delivered payload bytes
```
**Worked example (illustrative numbers).** 1,000 publishes/s × 100 subscribers × 1,024 bytes is
102,400,000 bytes/s, about 97.7 MiB/s or 819 Mbit/s of payload alone, before envelope, TLS, and
TCP overhead. On a VM with a 1 Gbit/s baseline that leaves little room, even though the CPU may be
mostly idle.
Also account for:
- WebSocket frame and Sockudo/Pusher envelope size
- TLS and TCP/IP overhead
- heartbeat traffic for every quiet socket
- retransmissions at the measured client RTT and loss
- cross-zone or cross-node adapter traffic
- metrics, webhook, history, and push traffic
Cloud VM “up to” bandwidth and packets-per-second limits can produce a sharp knee. Inspect vNIC and
load-balancer metrics while increasing load.
## The load generator is often the first limit
A remote benchmark can fail without Sockudo being saturated.
Check every generator for:
- CPU and scheduler saturation
- memory and garbage collection
- its own `nofile` limit
- local ephemeral port exhaustion
- `TIME_WAIT` accumulation between repeated runs
- source NAT or gateway connection tracking
- NIC bandwidth and packets per second
- DNS and TLS handshake rate
- dropped iterations or failure to maintain the requested arrival rate
One source IP connecting to one target IP and port has a finite TCP 4-tuple space, commonly much
less than a high-density Sockudo node can accept. To test beyond it, shard across generators, source
IPs, or target addresses. Do not “solve” a generator's ephemeral ports by changing
`ip_local_port_range` on the Sockudo server.
**Worked example (illustrative numbers).** A common Linux default range of `32768 60999` gives
28,232 ephemeral ports per source IP and destination. Holding 100,000 sockets to one target address
and port therefore needs at least `100,000 / 28,232 ≈ 3.5`, so 4, source IPs, generators, or target
addresses. Check the actual range with `sysctl net.ipv4.ip_local_port_range`.
Inspect a Linux generator:
```bash
ulimit -n
sysctl net.ipv4.ip_local_port_range
ss -s
nstat -az | grep -E 'TCPSynRetrans|TCPAbort|ListenDrop'
sar -n DEV,TCP,ETCP 1
```
If achieved request rate falls while Sockudo CPU, memory, network, queues, and latency remain flat,
the server is probably not the limiting system.
## Geography changes the benchmark
Record both generator and server region. “Clients in Taiwan” is not a reproducible topology:
document the cloud, city/region, ISP or cloud network, public or private path, RTT distribution, and
packet loss.
Long RTT changes:
- how quickly sockets can be established during a fixed ramp
- TLS handshake duration
- subscribe/auth acknowledgement latency
- retransmission recovery
- reconnect convergence after a failure
For geographically distributed users:
1. run generators in the actual client regions
2. synchronize their start time
3. report each region separately as well as the aggregate
4. keep server-side publish load independent from client-region socket generation
5. compare public internet and private cloud paths only when clearly labeled
A short test may end before a high-RTT generator reaches the intended connection count. Report
achieved active sockets over time, not only configured virtual users.
## Use distinct benchmark phases
### 1. Connect ramp
Measure connection success, TLS and WebSocket upgrade latency, admission rejections, and achieved
connections per second. Add jitter so the default test is not an unrealistic simultaneous SYN
flood, then run a separate reconnect-storm test intentionally.
### 2. Quiet-socket soak
Hold target connections long enough to observe memory stability, heartbeats, load-balancer idle
behavior, NAT state, and unexpected platform timeouts.
### 3. Subscription workload
Apply the real channel cardinality, subscriptions per socket, private/presence mix, and room churn.
### 4. Publish and fanout
Ramp accepted publishes and matching subscribers independently. Track delivery correctness and
end-to-end p50, p95, and p99 latency.
### 5. Failure and recovery
Remove a Sockudo node, interrupt the adapter, and slow a durable store. Measure reconnect time,
recovery success, duplicates, gaps, readiness changes, and backlog drain.
### 6. Cooldown
Allow connections and `TIME_WAIT` state to clear, or start from a fresh generator fleet. Back-to-back
runs without cooldown are not independent.
## Repository benchmarks
### Subscription churn
`benches/subscription-churn.js` is a k6 script that models room switching: each virtual user holds
one socket and moves to another channel every `ROOM_SWITCH_INTERVAL_MS`. Churn rate is
approximately:
```text
cycles/s = VUS / (ROOM_SWITCH_INTERVAL_MS / 1000)
```
For 10,000 sockets switching every 20 seconds, that is `10,000 / 20 = 500` unsubscribe/subscribe
cycles per second:
```bash
k6 run \
-e WS_HOSTS=wss://ws.example.com/app/app-key \
-e VUS=10000 \
-e CHANNEL_COUNT=4000 \
-e ROOM_SWITCH_INTERVAL_MS=20000 \
-e SOCKET_LIFETIME_MS=600000 \
-e DURATION=10m \
benches/subscription-churn.js
```
Shard `WS_HOSTS` or run several generators when one process or source IP approaches its limit.
### Horizontal adapter fanout
`benches/adapter-horizontal.js` records successful publishes, client deliveries, delivery ratio,
and latency. By default it targets three local nodes on ports 6001–6003; point it at your cluster
with comma-separated `WS_URLS` and `HTTP_URLS` lists of equal length, and the matching `APP_ID`,
`APP_KEY`, and `APP_SECRET`. `SUBSCRIBERS` must be divisible by `CHANNELS` times the node count.
```bash
SUBSCRIBERS=900 \
CHANNELS=100 \
MESSAGES_PER_SECOND=1000 \
WARMUP_SECONDS=30 \
DURATION_SECONDS=120 \
DRAIN_SECONDS=20 \
RESULT_PATH=/tmp/sockudo-adapter.json \
k6 run --quiet benches/adapter-horizontal.js
```
Each publish reaches `SUBSCRIBERS / CHANNELS` subscribers, so this run offers
`1,000 × (900 / 100) = 9,000` deliveries/s. Use the same image, nodes, subscriber distribution,
payload, warmup, and generator placement when comparing adapters.
## Diagnose the first knee
| Observation | Likely area | Next evidence |
| --- | --- | --- |
| Connect failures, Sockudo mostly idle | Generator, LB, firewall, NAT, SYN path | Generator ports/CPU, LB target errors, SYN retransmits, listen drops |
| `ListenOverflows` rises | Host listen/accept path | CPU, `somaxconn`, SYN backlog, connection ramp shape |
| High Sockudo CPU, low broker latency | Protocol, auth, compression, filtering, fanout | Flamegraph/profile and feature-isolated tests |
| Low Sockudo CPU, broker latency/backlog rises | Adapter, queue, cache, or database | Backend CPU, connections, command latency, network |
| Memory climbs after clients stabilize | Buffers, retained state, leak, slow clients | Heap/RSS profile, send queues, feature comparison |
| P99 rises, throughput still correct | Queueing before saturation | Per-stage latency and utilization; reduce offered load |
| Delivery ratio falls but publishes succeed | Fanout, slow clients, generator receive path | Per-node delivery counters, socket errors, client CPU |
| Reconnect test collapses only after node loss | Insufficient failure headroom | Remaining-node admission, LB distribution, backend burst |
Stop increasing the offered load at the first sustained latency or correctness failure. The
maximum technically accepted rate beyond that point is not the production capacity.
## Benchmark report template
Publish enough information for another operator to reproduce the result:
```text
Sockudo version and commit:
Build features and release profile:
Allocator and container image:
Server region, instance type, vCPU, RAM, NIC:
Replica count and per-node max_connections:
Load balancer and timeout:
Adapter, cache, queue, app manager, history:
Backend topology and region:
Protocol version and enabled features:
Generator version, count, regions, source IP count:
RTT p50/p95 and packet loss:
TLS termination path:
Connection ramp and achieved sockets:
Subscriptions, channels, presence/private mix, churn:
Publish rate, matching subscribers, delivery rate:
Payload and wire size:
Duration, warmup, drain, and cooldown:
Success/error/duplicate/gap counts:
Latency p50/p95/p99:
Sockudo, generator, LB, node, and backend utilization:
```
Without that context, benchmark numbers are anecdotes.
## Set the production limit
After the stable knee is known:
1. Choose a point below the first latency, error, or resource cliff.
2. Subtract failure and rollout headroom.
3. Set `SOCKUDO_MAX_CONNECTIONS` per node. Verify that connections beyond it are rejected and that
`/ready` returns `503 CAPACITY` at the high watermark (95% by default).
4. Configure load-balancer and autoscaling thresholds to add capacity before admission rejects.
5. Alert on connection utilization, rejection rate, memory, CPU throttling, network, adapter
latency, and recovery failures.
6. Rerun after changing the binary, features, instance type, kernel, runtime, CNI, load balancer, or
shared backend.
Capacity belongs to the complete deployment, not the Sockudo process in isolation.
## API pod topology
When HTTP publish traffic is large relative to WebSocket connection count, running
separate API pods (`server_role = "api"`) isolates publish latency and cuts memory
on the API tier. See [server role configuration](/docs/reference/configuration#server-role)
for setup and requirements.
API pods require a horizontal adapter: every publish they accept is sent through the adapter so
the WebSocket pods can deliver it. They do not accept WebSocket connections or answer cross-node
request/reply queries.
Expected operational behavior:
| Signal | API pod | WS pod |
| --- | --- | --- |
| `sockudo_connected` | 0 | > 0 |
| `sockudo_horizontal_adapter_received_requests` | 0 | > 0 when peers query it |
| WebSocket connections | 0 | > 0 |
| Memory (illustrative, 100K+ connections in the cluster) | ~50 MiB baseline | ~2–4 GiB |
API pods report `"server_role": "api"` in `/stats` so dashboards can distinguish
them from WS pods reporting zero connections. The field is absent on default-role pods.
## Next steps
- [Kubernetes and Helm](/docs/deployment/kubernetes): Apply the limit per pod and drive autoscaling from connection count.
- [Linux VM and bare metal](/docs/deployment/linux): Raise file limits and inspect kernel counters on the host.
- [Observability](/docs/server/observability): Metrics and alerts for connection utilization and fanout latency.
- [Scaling](/docs/server/scaling): Adapter choices and cross-node behavior that change capacity.
---
# Configuration (https://sockudo.io/docs/server/configuration)
> Configure apps, adapters, cache, queues, rate limits, history, recovery, webhooks, and push notifications.
This page walks through the runtime shape of a Sockudo deployment: where app credentials live, how
nodes fan out to each other, which shared stores carry state, and which protocol features are on.
Read it before your first production deploy, or whenever you add a node, a backend, or a V2 feature.
For TOML versus JSON, file discovery, `--config`, environment precedence, and secret placement, start
with [Static configuration](/docs/deployment/static-configuration). Every key is listed in the
[configuration reference](/docs/reference/configuration).
## How the pieces fit
Sockudo configuration should describe the runtime shape explicitly: app storage, fanout adapter,
cache, queue, protocol features, security controls, metrics, webhooks, and push notification
providers. Each top-level section owns one concern:
| Section | Owns | Single node | Multi-node |
| --- | --- | --- | --- |
| `[app_manager]` | App credentials and per-app policy | `memory` is fine | A shared database driver |
| `[adapter]` | Cross-node fanout and cluster-wide counts | `local` | A horizontal driver such as `redis` or `nats` |
| `[cache]` | Idempotency claims, token revocations, coordination | `memory` | `redis` or `redis-cluster` |
| `[queue]` | Webhook and background jobs | `memory` | A broker-backed driver |
| `[rate_limiter]` | HTTP API and WebSocket connect limits | `memory` | `redis` |
| `[connection_recovery]`, `[history]` | Reconnect replay and durable history | `memory` history backend | A shared history backend |
| `[push]` | Push storage, queue, and workers | Memory drivers in dev only | Durable storage and queue |
| `[database]` | Connection details the drivers above reuse | | |
The rule of thumb: anything a second node must see (a claimed idempotency key, a revoked token, a
replay position, a queued webhook) must live in a shared backend, not in process memory.
Built-in defaults are conservative: `[connection_recovery]`, `[history]`, and `[versioned_messages]`
are disabled, and `adapter.driver` is `local`. The shipped `config/config.toml` turns the V2
features on for local development. Set every feature you rely on explicitly in production config.
## Minimal local config
A single process with one static app. Good for development and tests.
```toml title="config.toml"
port = 6001
host = "0.0.0.0"
debug = true
[app_manager]
driver = "memory"
[app_manager.array]
[[app_manager.array.apps]]
id = "app-id"
key = "app-key"
secret = "app-secret"
enabled = true
[app_manager.array.apps.policy.limits]
max_connections = 1000
[app_manager.array.apps.policy.features]
enable_client_messages = false
```
## Production shape
Every node points at the same app database, adapter, cache, and queue. Metrics are served on a
separate port so you can keep it off the public listener.
```toml title="config.toml"
port = 6001
host = "0.0.0.0"
debug = false
[app_manager]
driver = "pgsql"
[adapter]
driver = "redis"
[cache]
driver = "redis"
[queue]
driver = "redis"
[database.redis]
host = "redis"
port = 6379
[metrics]
enabled = true
host = "0.0.0.0"
port = 9601
[metrics.tcp_exporter]
enabled = false
host = "127.0.0.1"
port = 5000
buffer_size = 1024
```
`[database.redis]` is the Redis connection reused by the Redis adapter, cache, queue, and rate
limiter. Sentinel, TLS, and Redis Cluster topology are covered in
[Redis connections, Sentinel, and TLS](/docs/reference/configuration#redis-connections-sentinel-and-tls).
## App manager
The app manager stores app credentials and app-level policy (limits, features, allowed origins,
channel namespaces, webhooks). Memory is useful for local development; persistent managers are
preferred for production because they let you add, rotate, and disable apps without a restart.
| Driver | Use when |
| --- | --- |
| `memory` | Credentials are static and local to one process. |
| `pgsql` (aliases `postgres`, `postgresql`) or `mysql` | You need relational app records and standard operational tooling. |
| `dynamodb` | You run on AWS and want managed key-value storage. |
| `scylladb` | You need wide-column scale. |
| `surrealdb` | You use SurrealDB 4 for app metadata. |
```toml title="config.toml"
[app_manager]
driver = "postgres"
# Close live connections of disabled/deleted apps (seconds, 0 = off).
availability_check_interval_seconds = 15
[app_manager.cache]
enabled = true
ttl = 300
```
The MySQL and PostgreSQL managers can encrypt app secrets at rest; see
[App secrets at rest](/docs/server/security#app-secrets-at-rest-mysql-and-postgresql). How disabling
an app reaches live connections is covered in [Disabling an app](/docs/server/security#disabling-an-app).
## Adapter
The adapter controls cross-node fanout. When a publish lands on node A, the adapter carries it to
every other node so each one can deliver to its own sockets. It also answers cluster-wide questions
such as "how many sockets are subscribed to this channel".
```toml title="config.toml"
[adapter]
driver = "redis"
enable_socket_counting = true
aggregate_counts = false
fast_presence_transitions = false
[adapter.redis]
prefix = "sockudo_adapter:"
requests_timeout = 5000
pubsub_ping_interval_ms = 10000
```
Use a shared adapter for every multi-node deployment. The `local` adapter is intentionally
process-local: a second node never sees the first node's publishes. Available drivers are `local`,
`redis`, `redis-cluster`, `nats`, `pulsar`, `rabbitmq`, `google-pubsub`, `kafka`, `iggy`, and `omq`;
see [Scaling](/docs/server/scaling#adapter-choices) for how to choose.
`pubsub_ping_interval_ms` (under `[adapter.redis]` and `[adapter.cluster]`, default `10000`) is how
often each node PINGs its Redis subscriber connection. A subscriber connection only reads, so when
Redis goes away without closing it (a network partition, or a managed instance resized or replaced
behind the same address) it would otherwise wait forever while publishing keeps working, and the
node silently stops receiving events from its peers. When a PING fails or gets no answer within one
interval, the node logs a warning, increments `sockudo_horizontal_transport_reconnections_total`,
and resubscribes. Set it to `0` to disable the probe.
With `pubsub_ping_interval_ms = 0`, a silently dropped subscriber connection leaves the node
publishing to peers but never hearing from them. Clients on that node see partial fanout with no
error.
### Redis Cluster broadcast bucketing
Without bucketing every broadcast is published to one channel, which hashes to a single slot and
therefore a single shard primary. That primary carries the whole fleet's broadcast egress.
`broadcast_buckets` (under `[adapter.cluster]`, 1–64, default `3`) splits each broadcast across that
many channel variants, each pinned to a different slot range with a hash tag, and every node
subscribes to exactly one bucket. Set it to the cluster's shard count. Bucketing requires
`use_sharded_pubsub = true` and a `prefix` without `{` or `}`; configure the same values on every
node.
`broadcast_mode` sequences a live migration, one rollout at a time:
| Mode | Publishes to | Subscribes to |
| --- | --- | --- |
| `legacy` (default) | the single `#broadcast` channel | `#broadcast` |
| `publish_both` | `#broadcast` and every bucket | `#broadcast` |
| `subscribe_bucket` | `#broadcast` and every bucket | this node's bucket |
| `bucketed` | every bucket | this node's bucket |
Any adjacent pair of modes delivers each broadcast exactly once to every node, so move
`legacy → publish_both → subscribe_bucket → bucketed` and let each rollout finish fleet-wide before
the next. To change `broadcast_buckets`, step back to `publish_both`, change the count, then move
forward again. Both knobs have environment overrides, `ADAPTER_CLUSTER_BROADCAST_MODE` and
`ADAPTER_CLUSTER_BROADCAST_BUCKETS`.
`enable_socket_counting` keeps the adapter's request/reply socket-count path enabled by default.
`aggregate_counts` is off by default; enable it when you want each node to maintain gossiped
cluster-wide channel counts for local count reads.
`fast_presence_transitions` is off by default. Enabling it uses the replicated presence registry
for first-join and last-leave checks, which avoids request/reply fanout but makes presence
webhook/history transition decisions eventually consistent.
For high-churn benchmark runs that should avoid distributed count fanout, use:
```bash
ADAPTER_ENABLE_SOCKET_COUNTING=false
ADAPTER_AGGREGATE_COUNTS=true
# Optional: enables eventual-consistency presence transition checks.
ADAPTER_FAST_PRESENCE_TRANSITIONS=true
```
## Cache and queue
The cache holds short-lived shared state such as idempotency claims and capability-token
revocations. The queue carries background work such as webhook delivery.
| Section | Drivers |
| --- | --- |
| `[cache]` | `memory`, `redis`, `redis-cluster`, `none` |
| `[queue]` | `memory`, `redis`, `redis-cluster`, `nats`, `pulsar`, `rabbitmq`, `google-pubsub`, `kafka`, `iggy`, `sqs`, `sns`, `none` |
Idempotency keys are claimed in the cache. With `cache.driver = "memory"` on a multi-node cluster,
a retry that lands on a different node is not recognized as a duplicate, and a token revoked on one
node stays valid on the others.
Retry, lease, and dead-letter behavior for queue jobs lives under `[queue.reliability]`; see
[Queue v2 reliability](/docs/reference/configuration#queue-v2-reliability).
## Recovery and history
Sockudo keeps two separate stores for past messages. They answer different questions, so size and
back them separately.
| | Hot replay buffer | Durable history |
| --- | --- | --- |
| Section | `[connection_recovery]` | `[history]` |
| Purpose | Fill the gap after a short disconnect | API reads, rewind, cold recovery, versioned messages, inspection |
| Lifetime | `buffer_ttl_seconds` (default 120) | `retention_window_seconds` (default 86400) |
| Size bound | `max_buffer_size` messages per channel (default 100) | `max_messages_per_channel`, `max_bytes_per_channel` |
| Storage | In process | `backend`: `postgres`, `mysql`, `dynamodb`, `surrealdb`, `scylladb`, or `memory` |
```toml title="config.toml"
[connection_recovery]
enabled = true
buffer_ttl_seconds = 120
max_buffer_size = 100
[history]
enabled = true
backend = "postgres"
retention_window_seconds = 86400
max_messages_per_channel = 10000
```
Recovery buffers are for reconnect continuity. Durable history is for API reads, rewind, versioned messages, and operational inspection. Keep those concerns separate.
`max_buffer_size` bounds both the per-channel hot buffer and a single durable fallback replay, so a
client that missed more than that many messages on one channel gets `position_expired` and must
backfill from history. The full recovery model is in [History and recovery](/docs/server/history-recovery).
## Push notifications
Push is a core Sockudo subsystem. Configure it with a queue, provider credentials, retention, admission limits, and metrics before sending production traffic.
```toml title="config.toml"
[push]
storage_driver = "postgres"
queue_driver = "redis"
publish_status_ttl_days = 30
analytics_retention_days = 30
scheduler_interval_secs = 5
cleanup_interval_secs = 300
cleanup_batch_size = 1000
cleanup_max_deleted_per_tick = 100000
# Runtime env for FCM monolith workers:
# PUSH_FCM_ENABLED=true
# PUSH_FCM_SERVICE_ACCOUNT_JSON_PATH=/var/run/secrets/fcm-service-account.json
# PUSH_FCM_PROJECT_ID is optional when the service account JSON has project_id.
# Runtime env for APNs monolith workers (token auth):
# PUSH_APNS_ENABLED=true
# PUSH_APNS_TOPIC=com.example.app
# PUSH_APNS_TEAM_ID=ABCDE12345
# PUSH_APNS_KEY_ID=KEY123ABCD
# PUSH_APNS_PRIVATE_KEY_PATH=/var/run/secrets/apns-auth-key.p8
```
Both `storage_driver` and `queue_driver` default to `memory`, which is node-local and loses push state
on restart. Production mode rejects memory push storage and queues unless
`allow_memory_drivers = true` is set explicitly.
Use the queue backend for push fanout. Direct synchronous provider delivery is only suitable for tests and can hide production latency.
APNs token auth needs a team ID, a key ID, and the private key (`PUSH_APNS_PRIVATE_KEY_PATH` or
`PUSH_APNS_PRIVATE_KEY`). Prefer the FCM service account JSON over `PUSH_FCM_PROVIDER_TOKEN`: a
static provider token cannot be refreshed and is only suitable for short-lived legacy dispatch.
Channel push rules are top-level entries. They trigger the existing push pipeline after matching realtime publishes:
```toml title="config.toml"
[[push_rules]]
enabled = true
channel_pattern = "notifications:*"
event_filter = ["agent-complete"]
rate_limit_per_second = 100
[push_rules.payload_mapping]
title_field = "title"
body_field = "body"
template_data_field = "data"
include_remaining_fields = true
```
`channel_pattern` accepts an exact name, `*`, or a single trailing wildcard such as
`notifications:*`, up to 256 bytes. The payload mapping values shown are the defaults. See
[Push notifications](/docs/server/push-notifications) for providers, templates, and status.
## Presence
```toml title="config.toml"
[presence]
max_members_per_channel = 100
max_member_size_in_kb = 2
update_rate_limit_per_member_per_second = 10
ungraceful_timeout_seconds = 0
v2_ungraceful_timeout_seconds = 15
```
`ungraceful_timeout_seconds = 0` preserves Protocol V1/Pusher's legacy immediate
`member_removed` behavior. Protocol V2 defaults
`v2_ungraceful_timeout_seconds` to `15`, retaining presence through short abnormal
disconnects without leave/enter flaps. Clean unsubscribe and clean WebSocket close
remove presence immediately for both protocol versions.
## Webhooks
Webhook endpoints are configured per app under `policy.webhooks`. The server-wide `[webhooks]`
section controls timeouts, batching, and retries for every app:
```toml title="config.toml"
[webhooks]
request_timeout_ms = 10000
[webhooks.batching]
enabled = true
duration = 50
size = 100
[webhooks.retry]
enabled = true
max_attempts = 5
max_elapsed_time_ms = 300000
initial_backoff_ms = 1000
max_backoff_ms = 60000
```
`batching.enabled` defaults to `true` when omitted (the shipped `config/config.toml` sets it to
`false`). `batching.duration` is the flush interval in milliseconds and `batching.size` the maximum
events per request. Webhooks are delivered through `[queue]`; with `queue.driver = "none"` they are disabled.
Webhook consumers must validate signatures with the raw request body. Configure retry behavior and dead-letter visibility before relying on webhooks for workflows. Endpoint setup and payloads are in [Webhooks](/docs/server/webhooks).
## Rate limits
The server-wide rate limiter has two buckets: signed HTTP API requests and new WebSocket connections.
Each limit allows `max_requests` per `window_seconds` per client IP.
```toml title="config.toml"
[rate_limiter]
enabled = true
driver = "redis"
[rate_limiter.api_rate_limit]
max_requests = 100
window_seconds = 60
trust_hops = 0
[rate_limiter.websocket_rate_limit]
max_requests = 20
window_seconds = 60
trust_hops = 0
```
The values shown are the built-in defaults. `trust_hops = 0` ignores forwarding headers and keys
limits on the socket peer; behind a trusted proxy, set it as described in
[Rate limiter client IP](/docs/reference/configuration#rate-limiter-client-ip). Per-app ceilings
such as `max_client_events_per_second` and `max_backend_events_per_second` live under
`[app_manager.array.apps.policy.limits]`.
Keep limits close to product intent. A collaboration app, trading dashboard, and push-heavy mobile app need different ceilings.
With `driver = "memory"`, each node counts independently, so the effective cluster limit is the
per-node limit multiplied by the node count.
## Environment variables
Use environment variables for secrets and deployment-specific settings:
```bash
SOCKUDO_DEFAULT_APP_ID=app-id
SOCKUDO_DEFAULT_APP_KEY=app-key
SOCKUDO_DEFAULT_APP_SECRET=app-secret
REDIS_URL=redis://redis:6379/0
PUSH_FCM_PROJECT_ID=project-id
PUSH_FCM_SERVICE_ACCOUNT_JSON_PATH=/run/secrets/fcm-service-account.json
```
Avoid putting app secrets, encryption master keys, webhook secrets, and push provider credentials in committed config files.
Every runtime environment override is listed in the [environment variable reference](/docs/reference/environment-variables).
## Next steps
- [Configuration reference](/docs/reference/configuration): Every section and key, with defaults.
- [Static configuration](/docs/deployment/static-configuration): File discovery, --config, and environment precedence.
- [Scaling](/docs/server/scaling): Choose an adapter and plan a multi-node cluster.
- [Security](/docs/server/security): Protect secrets, origins, auth endpoints, and push credentials.
---
# HTTP API (https://sockudo.io/docs/server/http-api)
> Publish events, query channels, read history, mutate messages, manage annotations, and operate push notifications.
Sockudo's HTTP API is the trusted server interface: your backend uses it to publish events, inspect
channels, read durable history, edit messages, and drive push notifications. Every request is
signed with an app's key and secret, so it must only be called from code you control. Use a
[server SDK](/docs/server-sdks) unless you are writing a custom integration; this page documents the
wire contract the SDKs implement.
## Request lifecycle
```mermaid
sequenceDiagram
participant B as Your backend
participant S as Sockudo
participant C as Subscribers
B->>S: POST /apps/app-id/events (signed)
Note over S: Check key, timestamp, body_md5, signature
S->>C: Fan out to subscribed sockets
S-->>B: 200 with ok or channel info
```
Requests are authenticated before the handler runs. A request that fails signing never reaches the
publish pipeline, consumes no idempotency key, and produces no fanout.
## Signing model
Signed requests include these query parameters:
| Parameter | Purpose |
| --- | --- |
| `auth_key` | App key. Must belong to the app in the `/apps/{app_id}` path. |
| `auth_timestamp` | Unix timestamp in seconds. Requests more than 600 seconds from server time are rejected. |
| `auth_version` | Signing protocol version (`1.0`). |
| `body_md5` | Hex MD5 of the raw JSON body. Required for a `POST` with a non-empty body; must be absent on `GET`, `DELETE`, and empty bodies. |
| `auth_signature` | Hex HMAC SHA-256 of the string to sign, keyed with the app secret. |
The string to sign is the uppercase method, the request path, and every query parameter except
`auth_signature`, sorted by key and joined as `key=value` pairs with `&`:
```text
POST
/apps/app-id/events
auth_key=app-key&auth_timestamp=1749377222&auth_version=1.0&body_md5=fc820aa38714282f8300c2ca039cd034
```
This is the Pusher signing scheme, so Pusher-compatible server libraries can sign Sockudo requests.
`body_md5` is computed over the raw body. Re-serializing JSON after signing (key reordering,
whitespace changes) produces a `401` with `body_md5 mismatch`. Clock skew beyond 600 seconds fails
with a `401` as well, so keep backend hosts on NTP.
## Error format
Every handler error returns JSON with a stable machine-readable `code`:
```json
{
"error": "Request must contain 'channels' (list) or 'channel' (string)",
"code": "missing_channel_info",
"status": 400
}
```
| Status | `code` values | Meaning |
| --- | --- | --- |
| `400` | `invalid_input`, `missing_channel_info`, `limit_exceeded` | Invalid request shape, bad query parameter, or a per-app limit such as event name length, channels per publish, or batch size. |
| `401` | `auth_failed` | Signature, timestamp, or `body_md5` failure, or the key does not match the path's app. |
| `403` | `forbidden` | The caller lacks a required capability, for example push admin. |
| `404` | `app_not_found`, `not_found` | Unknown app key, or the message, device, or resource does not exist. |
| `413` | `payload_too_large` | Event data exceeds `max_event_payload_in_kb`, or the body exceeds `http_api.request_limit_in_mb`. |
| `422` | `feature_disabled` | The feature (history, versioned messages, annotations) is disabled for this app or channel. |
| `429` | `rate_limited` | A rate limit was hit. Includes `Retry-After`. |
| `503` | `service_unavailable`, `backpressure`, `draining` | A shared backend is unavailable, an idempotent request is still in flight, or the node is shutting down. `backpressure` and `draining` include `Retry-After`. |
| `5xx` | `internal_error` and others | Server or backend dependency failure. |
AI Transport validation errors also include a numeric `ai_code`. The rate limiter middleware adds
`X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset` headers.
`503` and `429` responses are safe to retry with the same idempotency key; a load balancer may route
the retry to a healthy node. Other `4xx` responses will fail again until the request changes.
## Publish an event
`POST /apps/{app_id}/events` publishes one event to one or more channels. Auth: signed app request.
```http
POST /apps/{app_id}/events
Content-Type: application/json
```
```json
{
"name": "order.created",
"channel": "orders",
"data": {
"id": "ord_123"
},
"idempotency_key": "order-created-ord_123"
}
```
| Field | Description |
| --- | --- |
| `name` | Event name. Required. |
| `channel` or `channels` | One channel, or a list of channels (capped by `max_event_channels_at_once`). One of the two is required. |
| `data` | Event payload, as a string or JSON value. |
| `socket_id` | Exclude this connection from delivery. |
| `info` | Comma-separated attributes to return per channel: `subscription_count`, `user_count`. |
| `idempotency_key` | Deduplicate retries. The `X-Idempotency-Key` header is used when the body field is absent. |
| `message_id` | Idempotent create identity for V2 and AI Transport publishes. |
| `extras` | Protocol V2 extras envelope (headers, ephemeral, echo, push, AI Transport). Stripped for V1 clients. |
| `tags`, `delta` | Tag-filtering metadata and per-publish delta compression override. |
Use `socket_id` to suppress echo back to the originating connection.
A successful publish returns `200` with `{"ok": true}`. When `info` is set, it returns per-channel
attributes instead:
```json
{
"channels": {
"orders": { "subscription_count": 12 }
}
}
```
In the `api` server role, `subscription_count` and `user_count` are unavailable and return `400`.
### Idempotency
Idempotency keys are claimed atomically in the shared cache for `idempotency.ttl_seconds` (default
`120`) and must be non-empty and at most `idempotency.max_key_length` (default `128`) characters. A
retry with the same key returns the cached response without publishing again.
When `ai_transport` and versioned messages are enabled for the target channel,
AI events and publishes with `message_id` return serial acknowledgements under
the channel entry. Retries with the same `message_id` or `X-Idempotency-Key`
return the original serials without publishing a duplicate event.
If the publishing node exits after durable persistence but before recording the
idempotency acknowledgement, a retry reconstructs and atomically repairs the
receipt from native history/version storage. Raw idempotency keys are not stored
in message envelopes; recovery uses an app/channel-scoped hash plus the canonical
payload fingerprint.
```json
{
"channels": {
"ai:session-1": {
"message_serial": "00000000000000000001:test:00000000000000000001",
"history_serial": 1,
"delivery_serial": 1,
"version_serial": "00000000000000000001:test:00000000000000000001"
}
}
}
```
Idempotent publishes fail closed when their shared coordination backend is
unavailable: Sockudo returns `503 service_unavailable` instead of publishing
without a proven claim. Concurrent retries whose first request is still active also
return `503 backpressure` with `Retry-After: 1`. Retry either response with the same
idempotency key; a load balancer may route the retry to a healthy node.
**Errors:** `400` for a missing name or channel, a name or channel list over the app limit, or an
invalid idempotency key; `413` when `data` exceeds `max_event_payload_in_kb`; `503 draining` with
`Retry-After` on a shutting-down node without a horizontal adapter.
## Batch publish
`POST /apps/{app_id}/batch_events` publishes several events in one request. Auth: signed app request.
```http
POST /apps/{app_id}/batch_events
Content-Type: application/json
```
```json
{
"batch": [
{
"channel": "orders",
"name": "order.created",
"data": { "id": "ord_123" }
},
{
"channel": "orders",
"name": "order.paid",
"data": { "id": "ord_123" }
}
]
}
```
Each item accepts the same fields as a single publish. The response is `{}`, or
`{"batch": [ ... ]}` with one entry per item, in order, when any item sets `info`.
Batching reduces HTTP overhead, but do not use it to hide unbounded payloads. Keep per-event data compact.
`X-Idempotency-Key` identifies the whole batch request, while an
`idempotency_key` inside an item identifies only that logical event. The two key
domains do not collide. Every per-event key is validated with the same non-empty
and maximum-length rules as a single-event publish before any item is accepted,
then atomically claimed so concurrent batches cannot publish that event twice.
**Errors:** `400 limit_exceeded` when the batch is larger than `max_event_batch_size` (default `10`);
otherwise the same per-item validation as a single publish.
## Channel state
Read current occupancy. Auth: signed app request. These routes need local socket state, so they are
not registered in the `api` server role.
```http
GET /apps/{app_id}/channels?filter_by_prefix=presence-&info=user_count
GET /apps/{app_id}/channels/{channel_name}?info=subscription_count,user_count
GET /apps/{app_id}/channels/{channel_name}/users
```
| Endpoint | Response |
| --- | --- |
| List channels | `{"channels": {"presence-room": {"user_count": 3}}}`. `user_count` requires `filter_by_prefix=presence-`. |
| One channel | `{"occupied": true, "subscription_count": 12, "user_count": 3}`. Counts appear only when requested. |
| Presence users | `{"users": [{"id": "user-1"}]}`. Presence channels only; others return `400`. |
Use state APIs for dashboards and admin tooling. Do not poll them as a substitute for realtime subscription events.
If a cluster-wide count could not be collected from every node, the single-channel response adds
`"subscription_count_complete": false`.
For channels matched by `[ai_transport]`, `GET /apps/{app_id}/channels/{channel_name}`
also includes:
```json
{
"ai": {
"active_streams": 1,
"last_history_serial": 42,
"message_count": 7
}
}
```
## User connections
Disconnect or recycle every connection of an authenticated user (from user authentication).
Auth: signed app request. Not available in the `api` server role.
```http
POST /apps/{app_id}/users/{user_id}/terminate_connections
POST /apps/{app_id}/users/{user_id}/force_reconnect
```
Both return `{"ok": true}`. Use `terminate_connections` when a user is banned or signs out
everywhere; use `force_reconnect` when the user should come back immediately, for example after a
permission change.
## Capability token revocation
`POST /apps/{app_id}/revocations` revokes Protocol V2 capability tokens by `jti`, by `client_id`, or
both. Auth: signed app request.
```json
{
"jti": "tok_01J",
"client_id": "user-42",
"ttl_seconds": 3600,
"reason": "session revoked"
}
```
```json
{
"revoked_jti": true,
"revoked_client_id": true,
"closed_connections": 1
}
```
At least one of `jti` or `client_id` is required. The revocation lives for `ttl_seconds`, or until
`expires_at` (Unix seconds), or 24 hours when neither is set. The receiving node closes matching local
sockets at once; other nodes reject the token on next validation once the shared cache entry is
visible. See [Security](/docs/server/security#three-layer-auth-model).
## History
`GET /apps/{app_id}/channels/{channel_name}/history` reads durable channel history. Auth: signed app
request. Requires `[history]` enabled for the channel.
```http
GET /apps/{app_id}/channels/{channel_name}/history?limit=50&direction=newest_first
```
| Parameter | Description |
| --- | --- |
| `limit` | Page size. Defaults to and is capped at the channel's `max_page_size`. Must be greater than 0. |
| `direction` | `newest_first` (default, alias `backwards`) or `oldest_first` (alias `forwards`). |
| `cursor` | Opaque `next_cursor` from a previous page. |
| `start_serial`, `end_serial` | Serial bounds. |
| `start_time_ms`, `end_time_ms` | Time bounds in milliseconds. `start` and `end` are accepted aliases. |
```json
{
"items": [
{
"stream_id": "orders:main",
"serial": 42,
"published_at_ms": 1749377222000,
"message_id": "msg_01HX",
"event_name": "order.updated",
"payload_size_bytes": 128,
"message": { "event": "order.updated", "channel": "orders", "data": "..." }
}
],
"direction": "newest_first",
"limit": 50,
"has_more": true,
"next_cursor": "opaque-cursor",
"bounds": { "start_serial": null, "end_serial": null, "start_time_ms": null, "end_time_ms": null },
"continuity": {
"stream_id": "orders:main",
"oldest_available_serial": 1,
"newest_available_serial": 42,
"complete": true,
"truncated_by_retention": false
},
"stream_state": { "durable_state": "healthy", "recovery_allowed": true, "reset_required": false }
}
```
Items also carry `operation_kind`, and the `continuity` and `stream_state` objects carry more fields
than shown, including retained counts and bytes, transition timestamps, and the reporting node.
History responses use opaque cursors. Store and replay cursors as strings; do not parse them.
Use `direction=oldest_first`/`forwards` or `direction=newest_first`/`backwards`
with `start`, `end`, and the policy-capped `limit` to bound a stable page. A
cursor is opaque and must be replayed with the same query shape that produced
it. The Ably-compatible `/channels/{channel}/messages` projection additionally
returns credential-free `first` and optional `next` Link relations that preserve
that query shape.
When versioned messages are enabled, rows that point at a mutable message return its latest visible
version. Check `stream_state.durable_state` (`healthy`, `degraded`, or `reset_required`) and
`continuity.complete` before treating a page as authoritative.
**Errors:** `422 feature_disabled` when history is off for the channel; `400` for an invalid
`direction`, a zero `limit`, or a malformed cursor.
Operators can inspect and repair a stream with `GET .../history/state`,
`POST .../history/reset`, and `POST .../history/purge`. Reset and purge are destructive and require
`confirm_channel` (exactly the path channel), `confirm_operation` (`reset` or `purge`), and a
non-empty `reason`. Presence history has matching routes under `.../presence/history`. See the
[HTTP endpoint reference](/docs/reference/http-endpoints).
## Versioned messages
Protocol V2 mutable messages expose latest visible state and preserved versions. Auth: signed app
request. Requires `[versioned_messages]` enabled. Without `socket_id`, a mutation acts as the
privileged server and may change any message. With `socket_id`, it acts as that connected V2 socket:
the socket's verified user must match any `client_id` sent, and token-authenticated sockets need
`publish` on the channel plus the own/any mutation policy.
```http
GET /apps/{app_id}/channels/{channel_name}/messages/{message_serial}
GET /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/versions
POST /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/update
POST /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/delete
POST /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/append
```
`GET .../messages/{message_serial}` returns `{"channel": "...", "item": { ... }}` with the latest
version. `GET .../versions` accepts `limit`, `direction` (`newest_first` by default), and `cursor`,
and returns `channel`, `direction`, `limit`, `has_more`, `next_cursor`, and `items`.
The optional Ably compatibility route is
`PATCH /channels/{channel_name}/messages/{message_serial}`. Its encoded message
body selects `message.update`, `message.delete`, or `message.append` through the
`action` field and carries operation metadata in `version`. It is a projection
over the same native mutation service and `VersionStore`; it does not maintain
a compatibility-only message chain. Request and response formats are negotiated
independently as JSON or MsgPack.
An update body:
```json
{
"data": {
"body": "edited message"
},
"description": "user edit",
"op_id": "edit-msg-1-v2"
}
```
| Field | Applies to | Description |
| --- | --- | --- |
| `data` | all | New data (update, delete) or the string fragment to append (append; required, non-empty except for a terminal AI append). |
| `name` | update | New event name. |
| `extras` | all | Replaces the message's V2 extras when present. An append without `extras` keeps the existing ones. |
| `clear_fields` | update, delete | Fields to clear. Must not contain duplicates. |
| `client_id`, `socket_id` | all | Act as a connected V2 socket and its verified client instead of the privileged server. Not used for echo suppression: the mutation is delivered to every subscriber. |
| `description`, `metadata` | all | Operation metadata stored on the version. |
| `op_id` | all | Operation ID that makes retries idempotent. |
Mutation responses preserve the existing fields and include serials for SDK
ordering and retries:
```json
{
"channel": "ai:session-1",
"message_serial": "msg_123",
"action": "append",
"accepted": true,
"version_serial": "ver_456",
"history_serial": 12,
"delivery_serial": 34,
"status": "applied"
}
```
`status` is `applied` for a new version and `duplicate` when an `op_id` retry matched an existing one.
**Errors:** `404` for an unknown message serial; `422` when versioned messages or durable history are
disabled for the channel; `400`
for an invalid body or `op_id`; `401` when the `socket_id` actor is not connected, not V2, or does
not match `client_id`; `413` or `403` for AI Transport limits and mutable-not-permitted channels,
with an `ai_code`.
## Annotations
Annotations attach secondary state to a message without rewriting the message itself. Auth: signed
app request. Requires annotations enabled for the channel.
```http
POST /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/annotations
GET /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/annotations
DELETE /apps/{app_id}/channels/{channel_name}/messages/{message_serial}/annotations/{annotation_serial}
```
Annotation bodies and responses use camelCase field names:
```json
{
"type": "reactions:distinct.v1",
"name": "like",
"clientId": "user-1",
"count": 1
}
```
| Field | Description |
| --- | --- |
| `type` | Annotation type, such as `reactions:distinct.v1`. Required. |
| `name` | Annotation name within the type. |
| `clientId`, `socketId` | Acting client. With `socketId`, the actor is resolved from that connected socket. Not used for echo suppression. |
| `count` | Count for counting types. Must be greater than 0. |
| `data`, `encoding` | Optional payload. |
`POST` returns `{"annotationSerial": "..."}`. `DELETE` returns `annotationSerial` and
`deletedAnnotationSerial`. `GET` accepts `type`, `limit`, `from_serial`, and `socket_id`, and returns
`channel`, `messageSerial`, `limit`, `hasMore`, `nextCursor`, and `items`.
Use annotations for reactions, read receipts, moderation signals, and summary projections.
**Errors:** `404` when the target message does not exist; `422` when annotations are disabled;
`400` for `count: 0`.
## Push notifications
Push APIs are first-class HTTP APIs, not separate infrastructure. They cover device registration, channel subscriptions, credential management, publish admission, status inspection, scheduling, cancellation, and provider callbacks.
Auth: signed app request plus an optional `X-Sockudo-Push-Capability` header. Without the header the
request acts as `push-admin`. With `push-subscribe`, device-scoped routes also require
`X-Sockudo-Device-Identity-Token` matching the stored device. Push request and response bodies use
camelCase.
### Register a device
`POST /apps/{app_id}/push/deviceRegistrations` creates or updates a device. Admin or subscribe.
```http
POST /apps/{app_id}/push/deviceRegistrations
Content-Type: application/json
```
```json
{
"appId": "app-id",
"id": "ios-device-1",
"clientId": "user-42",
"formFactor": "phone",
"platform": "ios",
"deviceSecret": "",
"timezone": "Europe/Madrid",
"locale": "en-US",
"metadata": { "app_version": "4.5.0" },
"push": {
"recipient": { "transportType": "apns", "deviceToken": "provider-token" },
"state": "ACTIVE"
}
}
```
`appId` and `deviceSecret` are required by the decoder but replaced by the server: `appId` with the
path app, `deviceSecret` with a hash of a server-generated device identity token. `transportType` is
`apns`, `gcm` (FCM), or another provider recipient type.
The response is `201` with `change`, `tokenHash`, `device`, and, on first registration by an admin
(or with `X-Sockudo-Rotate-Device-Identity-Token: true`), `deviceIdentityToken`. Return that token
to the device; it is shown once.
### Subscribe a device to a channel
`POST /apps/{app_id}/push/channelSubscriptions` upserts a channel push subscription. Admin or
subscribe.
```http
POST /apps/{app_id}/push/channelSubscriptions
Content-Type: application/json
```
```json
{
"appId": "app-id",
"channel": "orders",
"deviceId": "ios-device-1",
"clientId": "user-42",
"provider": "apns",
"tokenHash": "token-hash-from-registration"
}
```
Returns `201` with the stored subscription. With `push-subscribe`, the device must exist and the
identity token must match, otherwise `404` or `403`.
### Publish push
`POST /apps/{app_id}/push/publish` admits a push publish. Admin only.
```http
POST /apps/{app_id}/push/publish
Content-Type: application/json
```
```json
{
"publishId": "push-order-ord_123-packing",
"recipients": [
{ "type": "channel", "channel": "orders" }
],
"payload": {
"title": "Order updated",
"body": "Order ord_123 moved to packing",
"templateData": { "order_id": "ord_123" }
}
}
```
Push returns `202 Accepted`:
```json
{
"publishId": "push-order-ord_123-packing",
"status": "accepted",
"expectedRecipients": 1,
"fanoutRegime": "fastPath",
"renderedPayloads": []
}
```
Send a stable `publishId` to make retries idempotent: a retry with the same `publishId` returns the
original publish and its current status instead of sending again. Without one, the server generates
a fresh ID per request. Admission is asynchronous: `sync` or `?mode=sync` still returns `202`, with
`X-Sockudo-Forced-Async: publish-api-is-async`. Use `notBeforeMs` to schedule and `expiresAtMs` to
bound delivery.
**Errors:** `403` without push admin; `400` for invalid recipients or payload; `503 backpressure`
when the admission rate or queue backpressure rejects the publish. A quota rejection is still
admitted with status `quota_exceeded`.
### Inspect publish status
```http
GET /apps/{app_id}/push/publish/{publish_id}/status
```
Admin only. Use this endpoint for operator tools, not user-facing busy loops.
## Ably compatibility routes
When Cargo feature and runtime option `ably-compat` are enabled, the root REST
router also exposes Ably-compatible batch operations:
```http
POST /messages
GET /presence?channels=channel-a,channel-b
POST /keys/{keyName}/revokeTokens
```
`POST /messages` accepts the raw object used by `Rest.request()` and the array of
batch specs used by `batchPublish()`. Results preserve spec and channel order.
The raw form returns `201` plus a flat channel result array on full success, or
`400`/`40020` plus `{ error, batchResponse }` when one or more channels fail.
The array form, batch presence, and token revocation return BatchResult envelopes
with exact `successCount`, `failureCount`, and ordered item results. Publish
success is reported only after the normal native publish pipeline commits; item
failures do not roll back unrelated successful items.
The compatibility batch edge runs no more than eight channel/target operations
at once and rejects requests exceeding 100 specs, 1,000 results, 10,000 publish
operations, or 10 MiB. Native event channel/message limits can lower those
ceilings. Bodies and responses negotiate JSON or MsgPack through the same REST
codec, and error responses include `X-Ably-ErrorCode` and
`X-Ably-ErrorMessage`.
With Cargo feature `ably-compat`, the compatibility router also exposes
`GET|POST /channels/{channel}/messages/{messageSerial}/annotations`. The POST
body is an Ably annotation array and supports both create and delete actions;
both operations call the same native annotation service shown above. Responses
negotiate JSON or MsgPack. List pages return credential-free relative Link
headers with an opaque app/channel/message-scoped cursor. Realtime action `21`
is delivered only to an attachment that negotiated `annotation_subscribe`,
while `message.summary` uses the original message serial for ordinary
subscribers. `annotation-publish`, `annotation-subscribe`,
`annotation-delete-own`, and `annotation-delete-any` are evaluated independently
from message mutation permissions.
When the server is built with `ably-compat`, the root Ably REST projection also
provides `/stats`, `/push/publish`, `/push/deviceRegistrations`,
`/push/channelSubscriptions`, and `/push/channels`. These routes authenticate
with Ably-compatible credentials but call a bounded typed compatibility stats
store plus the native push domain services; they do not call Sockudo HTTP
handlers internally. Stats pages use opaque cursors and stable Link query
propagation. Direct push and `extras.push` enter the native durable publish log,
queue, planner, provider-dispatch, feedback, retry, status, scheduler, and
retention pipeline. The realtime provider delivers `__ably_push__` through the
ordinary `MessageService` fanout path and records the actual outcome. External
providers continue through their configured native workers and fail truthfully
when the required feature or credentials are unavailable. Restricted Ably keys
must carry `push-admin` or channel-scoped `push-subscribe`; device-owned requests
also verify the stored hashed device identity token.
The same projection exposes `GET /channels/{channel}/presence` and
`GET /channels/{channel}/presence/history`. Current presence supports
`clientId`, `connectionId`, `limit`, and opaque cursor pagination. Presence
history additionally supports forward/backward direction and millisecond
`start`/`end` bounds. Both routes use the native presence service and return
credential-free `first`/`next` Link relations. History reads fail with Ably code
`50003` when durable continuity is degraded or reset-required.
## Complete endpoint reference
This page explains the common server workflows. The complete route matrix, including health probes, stats, metrics, all push credential/template/device/subscription endpoints, presence history state, durable history repair, and annotation filters, is in the [HTTP endpoint reference](/docs/reference/http-endpoints).
## Next steps
- [HTTP endpoint reference](/docs/reference/http-endpoints): Every route, method, and auth requirement.
- [Server SDKs](/docs/server-sdks): Signed clients for Node, Python, Go, PHP, Ruby, and more.
- [History and recovery](/docs/server/history-recovery): How durable history, cursors, and continuity work.
- [Push notifications](/docs/server/push-notifications): Providers, templates, scheduling, and delivery status.
---
# Realtime application patterns (https://sockudo.io/docs/server/realtime-application-patterns)
> Practical Sockudo patterns for public channels, private auth, presence, history, mutable messages, annotations, and scaling.
This page collects the patterns most Sockudo applications are built from: public feeds, private and
presence channels, recovery and rewind, mutable messages, annotations, and the shape of a scaled
deployment. Use it as a blueprint when you design a new realtime feature, and as a checklist when
you review one.
Sockudo starts Pusher-compatible and becomes Sockudo-native when you opt into Protocol V2 features.
The safest production pattern is to keep untrusted clients simple and put authority in your backend.
## The trust model
```mermaid
flowchart LR
B[Browser or mobile SDK] -->|WebSocket subscribe| S[Sockudo]
B -->|private/presence auth request| A[Your backend]
A -->|signed auth response| B
A -->|signed HTTP publish| S
S -->|fanout| B
S --> H[(History / version store)]
S --> P[(Push queue)]
```
Three actors, three roles:
- **Clients** hold only the app key. They connect, subscribe, and render. They can be inspected and
modified by their users, so nothing they send is trusted on its own.
- **Your backend** holds the app secret. It decides who may join which private or presence channel,
signs those decisions, and publishes business events over the signed HTTP API.
- **Sockudo** verifies signatures, fans messages out, and keeps history, versions, and the push queue.
Every pattern below follows from that split: authorization decisions are made in your backend, and
Sockudo enforces the signed result.
## Public feed
Public channels require no channel auth. Use them for data that is safe for anyone with the app key
to receive.
```ts
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
});
const channel = client.subscribe("market:btc");
channel.bind("price.updated", (event) => {
renderPrice(event.data);
});
```
Publish from a trusted backend:
```ts
await sockudo.trigger("market:btc", "price.updated", {
symbol: "BTC",
price: "104200.00",
});
```
Anyone can read the app key from your client bundle and subscribe to any public channel. If the data
is per-user or per-tenant, use a private channel.
## Private channel
Private channels prove the user may subscribe before Sockudo admits the subscription. Channels whose
name starts with `private-` trigger the flow automatically.
1. The client opens a WebSocket and receives a `socket_id`.
2. On `subscribe("private-...")`, the client SDK posts `socket_id` and `channel_name` to your
`channelAuthorization.endpoint`.
3. Your backend authenticates the user, checks the channel against its own policy, and signs the
pair with the app secret.
4. The client sends the signature to Sockudo, which verifies it and completes the subscription.
```ts {4-7}
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
channelAuthorization: {
endpoint: "/api/sockudo/channel-auth",
headers: { "x-csrf": csrfToken },
},
});
const account = client.subscribe("private-account:user-42");
account.bind("invoice.paid", updateInvoice);
```
Backend auth should derive the channel from verified user state, not from a trusted-looking request
body.
```ts {5-7}
app.post("/api/sockudo/channel-auth", async (req, res) => {
const user = requireUser(req);
const { socket_id, channel_name } = req.body;
if (channel_name !== `private-account:${user.id}`) {
return res.status(403).json({ error: "forbidden" });
}
res.json(sockudo.authorizeChannel(socket_id, channel_name));
});
```
Write the auth endpoint as an allowlist of channel shapes the user may join. Anything that does not
match returns `403`, including channel names you have not invented yet.
## Presence room
Presence channels add a current member list and join/leave events.
```ts
const room = client.subscribe("presence-room:incident-123");
room.bind("pusher:subscription_succeeded", (members) => {
renderMembers(members);
});
room.bind("pusher:member_added", (member) => {
addMember(member.id, member.info);
});
room.bind("pusher:member_removed", (member) => {
removeMember(member.id);
});
```
Presence auth includes user data:
```ts
res.json(sockudo.authorizeChannel(socketId, channelName, {
user_id: user.id,
user_info: {
name: user.name,
role: user.role,
},
}));
```
Members are keyed by `user_id`, so one user with several tabs appears once: `member_added` fires on
the first connection and `member_removed` after the last one leaves. Keep `user_info` small; it is
capped by `presence.max_member_size_in_kb` (default 2 KB) and sent to every member.
## Protocol V2 recovery and rewind
Protocol V2 clients can recover missed messages and ask for recent history during subscribe.
- **Recovery** fills the gap after a short disconnect, so a client that was already subscribed sees
every message exactly as if it had stayed connected.
- **Rewind** gives a client that is joining now the last few messages, so a new chat window or
dashboard starts populated.
```ts
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
protocolVersion: 2,
connectionRecovery: true,
});
const orders = client.subscribe("private-orders:user-42", {
rewind: { count: 50 },
});
orders.bind("sockudo:rewind_complete", ({ historical_count, live_count }) => {
console.log({ historical_count, live_count });
});
client.bind("sockudo:resume_success", ({ recovered, failed }) => {
console.log("resume", { recovered, failed });
});
```
```mermaid
sequenceDiagram
participant C as V2 client
participant S as Sockudo
participant H as Durable history
C->>S: subscribe with rewind
S->>H: read history until attach serial
H-->>S: historical page
S-->>C: historical messages
S-->>C: buffered live messages
S-->>C: sockudo:rewind_complete
C--xS: network interruption
C->>S: resume positions
S->>H: cold replay if hot buffer expired
S-->>C: sockudo:resume_success or resume_failed
```
Use `until_attach` for late joins: it prevents a history page from racing with live messages above
the attach serial.
When continuity cannot be proven, Sockudo reports `sockudo:resume_failed` instead of replaying a
partial gap. Resubscribe and backfill from history; never mark the channel as recovered on your own.
See [History and recovery](/docs/server/history-recovery).
## Mutable messages
Mutable messages let a logical message be updated, deleted, appended, and summarized while keeping a
version history. Every subscriber converges on the same latest version, and late joiners receive
that version rather than the original create payload.
```ts
const latest = await channel.getMessage("msg_01J");
const versions = await channel.getMessageVersions("msg_01J", {
limit: 20,
direction: "oldest_first",
});
```
Trusted backends mutate through signed HTTP:
```http
POST /apps/{app_id}/channels/private-doc:123/messages/msg_01J/update
Content-Type: application/json
```
```json
{
"data": { "title": "Final incident report" },
"description": "user edit",
"metadata": { "editor": "user-42" },
"op_id": "edit-msg_01J-title-v2"
}
```
Append is the right primitive for streams:
```json
{
"data": "next generated token ",
"extras": {
"ai": {
"transport": {
"run-id": "run_123",
"stream-id": "text",
"status": "streaming"
}
}
},
"op_id": "run_123:text:00042"
}
```
A stable `op_id` per operation makes retries safe: a repeated `op_id` returns `status: "duplicate"`
instead of appending twice. See [Mutable messages](/docs/server/mutable-messages).
## Annotations
Annotations attach secondary state to a message without rewriting the message itself. Use them for
reactions, read receipts, moderation notes, delivery receipts, and summaries.
```http
POST /apps/{app_id}/channels/chat/messages/msg_01J/annotations
Content-Type: application/json
```
```json
{
"type": "reactions:distinct.v1",
"name": "thumbs-up",
"clientId": "user-42",
"count": 1
}
```
Annotation permissions are separate from message mutation permissions, so a user can react to a
message they are not allowed to edit.
## Backend publishing checklist
| Concern | Recommendation |
| --- | --- |
| Secrets | Keep app secrets and provider credentials server-side. |
| Idempotency | Send `idempotency_key` for business events and `op_id` for mutations. |
| Echo | Use `socket_id` when the originating client already applied an optimistic update. |
| Payload size | Send stable identifiers and fetch large state from your app API. |
| Auth | Sign private, presence, encrypted, and user auth from your backend. |
| History | Store opaque cursors unchanged. Do not parse cursor internals. |
## Scaling shape
```mermaid
flowchart TB
LB[Load balancer] --> Cluster
subgraph Cluster [Sockudo cluster]
N1[Node A]
N2[Node B]
N3[Node C]
end
Cluster <-->|adapter fanout| Bus[(Adapter bus)]
Cluster --> Cache[(Shared cache)]
Cluster --> Store[(History + version store)]
Cluster --> Queue[(Webhook / push queue)]
```
The adapter bus is any horizontal adapter: Redis, Redis Cluster, NATS, Kafka, RabbitMQ, Pulsar,
Google Pub/Sub, Apache Iggy, or OMQ.
For horizontal deployments, avoid memory-only cache, queue, history, and version stores. Memory is
fine for local development and single-node demos; production clusters need shared stores for
idempotency, recovery, push dispatch, and durable history continuity.
## Normal Sockudo vs AI Transport
| Need | Normal Sockudo | AI Transport |
| --- | --- | --- |
| Broadcast app events | `trigger`, channels, private auth | Not needed |
| Presence/collaboration | Presence channels | Used for agent/user state too |
| Recover after reconnect | Protocol V2 recovery | Required for durable runs |
| Persist channel history | Durable history | Required for sessions |
| Edit or append messages | Mutable messages | Used for streamed output |
| Wake offline users | Push notifications | Used for background agents |
| Model/provider integration | Your app code | Provider adapters and run lifecycle |
Start with normal Sockudo when you are building realtime product events. Add AI Transport when model
runs need to outlive one HTTP request, one browser tab, or one device.
## Next steps
- [Authentication](/docs/getting-started/authentication): Private, presence, and user authentication end to end.
- [History and recovery](/docs/server/history-recovery): Hot replay, durable history, rewind, and failing closed.
- [Mutable messages](/docs/server/mutable-messages): Update, delete, and append with version history.
- [AI Transport](/docs/server/ai-transport-overview): Durable, resumable model runs on the same primitives.
---
# Scaling (https://sockudo.io/docs/server/scaling)
> Design multi-node Sockudo deployments with shared adapters, cache, queues, recovery, and push fanout.
This page explains how Sockudo runs as a cluster: how a publish on one node reaches sockets on
another, which dependencies every node must share, and how to deploy, drain, and load-test without
dropping messages. You need it as soon as you run more than one Sockudo process, including two
replicas behind a load balancer for availability.
Sockudo scales horizontally when every node shares the dependencies that carry cross-node state: adapter, cache, queue, app manager, and optional history store.
AI Transport has a stricter horizontal matrix because streaming recovery depends on durable history, versioned-message state, and shared orphan ownership. For horizontal adapters (`redis`, `redis-cluster`, `nats`, `pulsar`, `rabbitmq`, `google-pubsub`, `kafka`, `iggy`, or `omq`), configure shared non-memory history and version-store backends plus Redis or Redis Cluster cache. Startup rejects AI Transport with process-local memory history, version stores, or cache on horizontal adapters.
## Architecture
Each WebSocket lives on exactly one node, but a publish can arrive at any node: over HTTP through the
load balancer, or as a client event from a socket on another node. The adapter closes that gap. The
receiving node delivers to its own subscribers and publishes the message to the adapter bus; every
other node receives it and delivers to its own subscribers. Cluster-wide questions, such as channel
occupancy or presence membership, are answered by asking peers over the same bus or from gossiped
counts.
At minimum, a production cluster has:
- a load balancer with WebSocket upgrade support
- multiple Sockudo nodes
- a shared adapter for cross-node fanout
- shared cache for rate limits, idempotency, and coordination
- shared queue for webhooks and push delivery
- shared app manager for dynamic app credentials
- metrics and logs from every node
### What breaks if a dependency is node-local
| Dependency | If left on `memory` or `local` in a cluster |
| --- | --- |
| Adapter | Subscribers on other nodes never receive the publish. |
| Cache | Idempotent retries that land on another node publish twice; token revocations do not propagate. |
| Rate limiter | Each node enforces its own bucket, so the effective limit grows with node count. |
| Queue | Webhook and push jobs are lost when the node that holds them exits. |
| History and version store | Recovery and history reads on another node cannot prove continuity and fail closed. |
| App manager | App changes must be redeployed to every node. |
## Adapter choices
| Adapter | Driver | Strength |
| --- | --- | --- |
| Redis | `redis` | Simple, common, low-latency local and regional deployments. |
| Redis Cluster | `redis-cluster` | Higher Redis scale and shard-aware deployments. |
| NATS | `nats` | Lightweight pub/sub with strong operational ergonomics. |
| Kafka | `kafka` | Durable stream backbone and high-volume integration pipelines. |
| RabbitMQ | `rabbitmq` | Enterprise messaging and routing patterns. |
| Pulsar | `pulsar` | Multi-tenant stream workloads. |
| Google Pub/Sub | `google-pubsub` | Managed GCP fanout. |
| Apache Iggy | `iggy` | High-throughput persistent log workloads, with replicated VSR clusters. |
| OMQ | `omq` | Brokerless PUB/SUB between nodes; messages in flight are not retained. |
If you have no strong preference, start with Redis: it is also the natural choice for the shared
cache, queue, and rate limiter, so one dependency covers four roles.
### NATS notes
NATS is a good fit for lightweight cross-node fanout, but Sockudo clusters should avoid turning every client subscribe or unsubscribe into a distributed request/reply round trip. Keep room-switch churn on local channel state where possible, and reserve cluster-wide socket counts for post-ack meta events, webhooks, HTTP inspection, and operator views.
For subscription-churn or room-switch benchmarks, prefer gossiped aggregate counts so count reads stay local on each node:
```bash
ADAPTER_ENABLE_SOCKET_COUNTING=false
ADAPTER_AGGREGATE_COUNTS=true
```
Presence first-join and last-leave checks remain strict by default because they drive member
webhooks and presence history. For workloads that prefer maximum churn throughput over immediate
transition consistency, opt into the replicated presence-registry fast path:
```bash
ADAPTER_FAST_PRESENCE_TRANSITIONS=true
```
For Kubernetes NATS clusters, prefer explicit StatefulSet pod DNS entries over a single headless service URL when you want better initial client spread:
```json
{
"adapter": {
"driver": "nats",
"nats": {
"servers": [
"nats://sockudo-nats-0.sockudo-nats-headless:4222",
"nats://sockudo-nats-1.sockudo-nats-headless:4222",
"nats://sockudo-nats-2.sockudo-nats-headless:4222"
],
"request_timeout_ms": 5000,
"connection_timeout_ms": 5000,
"subscription_capacity": 131072,
"client_capacity": 131072,
"max_reconnects": 60,
"no_echo": true
}
}
}
```
If `nodes_number` is set, it means expected Sockudo nodes, not NATS server replicas. Use it only when the Sockudo replica count is fixed or injected from deployment automation; otherwise leave discovery enabled.
### Apache Iggy cluster notes
Sockudo supports Apache Iggy 0.9 VSR clusters for both the adapter and the
queue. Run at least three Iggy replicas: three tolerate one failed replica, five
tolerate two. A two-replica cluster cannot make progress after either fails.
Point Sockudo at one replica and list the others as seeds, so a node that is
down at startup does not block bootstrap:
```bash
ADAPTER_DRIVER=iggy
QUEUE_DRIVER=iggy
IGGY_CONNECTION_STRING=iggy://iggy:secret@iggy-1.internal:8090
IGGY_CLUSTER_SEEDS=iggy-2.internal:8090,iggy-3.internal:8090
QUEUE_IGGY_DURABILITY=persisted
```
After it connects, the Iggy SDK learns the full roster from cluster metadata,
reconnects to the leader, fails over to the surviving replicas when its node
disappears, and rejoins queue consumer groups.
Losing the Iggy leader pauses writes until the surviving replicas elect a new
one. A crashed or partitioned leader is noticed only after Iggy's
`cluster.heartbeat_timeout` (5 s by default, 2 s minimum), so expect roughly
6 to 9 seconds without cross-node fanout. Publishes wait through that window
for up to `failover_timeout_ms` (15 s by default) and then complete; local
subscribers on the publishing node are served immediately as always. `/up` on
every Sockudo node can return 503 for the same window, so keep readiness probes
tolerant of about 10 seconds of failures; the Kubernetes defaults are. Lowering
`heartbeat_timeout` shortens the pause at the cost of spurious elections on
loaded hosts.
Each replica must advertise an address Sockudo can reach: the Iggy roster
(`cluster.nodes[*].ip` or `advertised_address`) is what the SDK dials during
failover, not the seed hostnames. `IGGY_NODES_NUMBER` counts Sockudo nodes, not
Iggy replicas. See
[configuration](/docs/reference/configuration#apache-iggy-adapter-and-queue)
for durability and reconnection tuning, and `docker-compose.iggy-cluster.yml`
for a local three-replica stack.
## Load balancing
Sockudo does not require sticky sessions for basic pub/sub when the adapter is shared. Sticky sessions can still reduce reconnect churn and preserve local buffers during rolling updates.
Use:
- WebSocket upgrade headers
- idle timeouts longer than heartbeat intervals
- health and readiness checks
- draining before pod termination
- disruption budgets for production clusters
For Kubernetes probes, use `/live` for liveness and startup probes. `/up` checks configured apps and shared dependencies, so it is appropriate for readiness but too expensive for liveness under backend pressure. A slow app manager, cache, queue, or adapter should make the pod unready, not restart it.
```yaml
livenessProbe:
httpGet:
path: /live
port: http
startupProbe:
httpGet:
path: /live
port: http
readinessProbe:
httpGet:
path: /up/
port: http
```
If a shared dependency such as Redis slows down, every pod fails `/up` at once. As a liveness probe
that restarts the whole fleet together; as a readiness probe it only stops routing new traffic.
## Duplicate delivery
Distributed realtime systems must tolerate retries and duplicate delivery. Sockudo features that help:
- HTTP `idempotency_key`
- V2 `message_id`
- client-side message deduplication in native SDKs
- adapter-level duplicate suppression visibility
- recovery continuity checks
Consumers should treat application event IDs as stable and idempotent.
## Recovery across nodes
V2 recovery uses stream continuity. A reconnect to a different node can recover only if the required replay state is available through the configured shared backend or still present in a valid buffer.
The hot replay buffer lives in each node's memory and holds messages published through that node,
so a reconnect to a different node usually cannot prove continuity from it. Recovery then falls
through to durable history: with a shared history backend the gap is replayed; with `memory` history
it cannot be proven and the resume fails.
Fail closed if continuity cannot be proven. Do not display a recovered state unless the server returns a successful resume.
## Push fanout at scale
Push notification fanout is queue-oriented. A realtime publish and a push publish may target the same logical event, but they have different latency, retry, and delivery semantics.
Operational recommendations:
- keep push `sync` false in production
- use a stable `publishId` for publish retries
- partition high-volume channel pushes by tenant or campaign
- set publish status retention high enough for support workflows
- alert on provider error rates and queue backlog
- store provider credentials in secrets, not config maps
- use capacity planning before large campaigns
## Rolling deploys
On SIGTERM or SIGINT, Sockudo:
1. Reports not ready: `/up` and `/ready` return 503 `DRAINING` and new WebSocket upgrades are
rejected with 503. The listener stays open and HTTP API requests are still served, except
publishes (`/events`, `/batch_events`) on a node without a horizontal adapter: those return
503 `draining` with `Retry-After`, because no subscriber can receive them there any more.
Publishers can retry them against a healthy instance.
2. Closes existing connections with code 4200 so clients reconnect right away instead of
waiting for the grace period or a load-balancer cutoff. By default every connection closes
at once. `shutdown_close_delay_ms` waits that long first, so the load balancer can see
`/ready` return 503 before anyone reconnects. `shutdown_close_spread_ms` spreads the closes
evenly over that window, so a node's clients do not all reconnect onto the remaining nodes
in the same instant. A second signal closes any remaining connections at once.
3. Keeps the listener open until `shutdown_grace_period` seconds after the signal so load
balancers finish deregistration without connection-refused errors. The close delay and
spread run inside this window; together they must not exceed it.
4. Waits for connection cleanup, pending presence leases, and delayed lifecycle webhook
producers, then flushes the remaining webhook batch. This phase has a separate budget of
`queue.reliability.shutdown_timeout_ms` (30 seconds by default). Presence leases and
reconnect checks retain their normal timing; shutdown does not force an early departure.
5. Drains local memory-queue work before disconnecting the queue and other backends. Broker
queues retain their enqueued jobs and apply their normal worker shutdown budget.
Allow the process supervisor enough time for the listener grace, lifecycle drain, queue worker
shutdown, and connection/backend teardown. The two drain phases can each use
`queue.reliability.shutdown_timeout_ms`. An exhausted lifecycle budget logs a warning that
pending events may be lost; a forced process kill cannot flush in-memory events. See
[shutdown webhook delivery](/docs/server/webhooks#shutdown-delivery).
Set `shutdown_grace_period` to cover your load balancer's deregistration time (for a
readiness probe, roughly `periodSeconds × failureThreshold` plus endpoint propagation). Keep
it well below the load balancer's connection-draining timeout, otherwise the balancer cuts
idle WebSocket connections before Sockudo finishes deregistering.
For large nodes, spread the reconnect load. For example, with a load balancer that marks a node
down after about two seconds:
```toml title="config.toml"
shutdown_grace_period = 20
shutdown_close_delay_ms = 3000
shutdown_close_spread_ms = 15000
```
Clients can add their own spread too. For example, `@sockudo/client` has `retryJitterMs`.
Set `terminationGracePeriodSeconds` above `shutdown_grace_period` plus the lifecycle and queue drain
budgets, or the kubelet kills the process before in-memory webhook events are flushed.
Then:
1. Keep adapter and cache dependencies available.
2. Watch reconnect, resume, and missed-message metrics.
3. Roll nodes in small batches.
## Metrics to watch
Metric names below assume the default `sockudo_` prefix. Full tables and alert suggestions are in
[Observability](/docs/server/observability).
| Signal | Metrics |
| --- | --- |
| Active connections by node | `sockudo_connected`, by `instance` |
| Connection churn | `sockudo_new_connections_total`, `sockudo_new_disconnections_total` |
| Subscriptions by channel class | `sockudo_channel_subscriptions_total`, `sockudo_active_channels` (label `channel_type`) |
| Publish volume | `sockudo_http_calls_received_total`, `sockudo_ws_messages_sent_total` |
| Fanout latency | `sockudo_broadcast_latency_ms` |
| Cross-node requests and adapter health | `sockudo_horizontal_adapter_resolve_time`, `sockudo_horizontal_adapter_uncomplete_promises`, `sockudo_horizontal_transport_reconnections_total`, `sockudo_horizontal_transport_messages_dropped_total` |
| Recovery | `sockudo_history_recovery_success_total{source}`, `sockudo_history_recovery_failures_total{code}` |
| Durable state | `sockudo_history_degraded_channels`, `sockudo_history_reset_required_channels`, `sockudo_history_queue_depth` |
| AI stream orphans | `sockudo_ai_active_streams`, plus `ai_stream_orphaned` webhook delivery |
| Webhook and job queues | `sockudo_queue_depth`, `sockudo_queue_settled_total{outcome}` |
| Push | `sockudo_push_publish_accepted_total`, `sockudo_push_dispatched_total`, `sockudo_push_provider_failures_total`, `sockudo_push_dispatch_duration_seconds` |
## Capacity checklist
Before increasing traffic, follow [Capacity planning and benchmarks](/docs/deployment/capacity-planning)
and run a scenario that covers:
- peak connected clients
- high-frequency channel fanout
- private and presence auth throughput
- durable history writes
- recovery after node restart
- AI stream orphan closure after node death
- pause/unpause partition checks with `scripts/ai-transport-jepsen-lite.sh`
- push burst admission
- push provider throttling
- webhook retry behavior
For room-switch workloads, `benches/subscription-churn.js` models connected sockets that periodically unsubscribe and subscribe to a new room. Churn rate is approximately `VUS / (ROOM_SWITCH_INTERVAL_MS / 1000)`. For example, 10,000 users switching every 20 seconds produces about 500 unsubscribe+subscribe cycles per second:
```bash
k6 run \
-e WS_HOSTS=wss://example.com/app/app-key \
-e VUS=10000 \
-e CHANNEL_COUNT=4000 \
-e ROOM_SWITCH_INTERVAL_MS=20000 \
-e SOCKET_LIFETIME_MS=600000 \
-e DURATION=10m \
benches/subscription-churn.js
```
## Next steps
- [Kubernetes](/docs/deployment/kubernetes): Probes, disruption budgets, and Helm values for a cluster.
- [Capacity planning](/docs/deployment/capacity-planning): Benchmarks and sizing before a traffic increase.
- [Observability](/docs/server/observability): Metric tables and alerts for every subsystem.
- [History and recovery](/docs/server/history-recovery): Why cross-node recovery needs a shared history backend.
---
# Security (https://sockudo.io/docs/server/security)
> Secure Sockudo credentials, auth endpoints, origin policy, encrypted channels, webhooks, and push provider credentials.
This page maps each Sockudo security control to the attack it stops: leaked secrets, forged
subscriptions, cross-site socket hijacking, spoofed webhooks, abusive clients, and push misuse. Read
it before exposing a deployment to the internet, and revisit it whenever you add a channel type,
enable client events, or turn on push.
Sockudo security depends on clear trust boundaries. Client SDKs connect and subscribe; server SDKs hold secrets, sign auth responses, publish events, validate webhooks, and manage push.
## Threat model
| Threat | Control | Section |
| --- | --- | --- |
| An attacker obtains the app secret and publishes or signs subscriptions as your backend | Keep secrets server-side; encrypt them at rest | [Secrets](#secrets) |
| A user subscribes to another user's private channel | Your auth endpoint signs only channels the user owns | [Auth endpoints](#auth-endpoints) |
| A malicious web page opens a socket with your app key in a victim's browser | Origin allowlist | [Origin policy](#origin-policy) |
| A subscribed client impersonates the server on a shared channel | Client events off by default, `client-` prefix only | [Client messages](#client-messages) |
| A forged request hits your webhook endpoint | HMAC signature over the raw body | [Webhook validation](#webhook-validation) |
| A client floods connections, API calls, or events | Rate limits and per-app limits | [Rate limiting](#rate-limiting) |
| A stolen or stale client token keeps working | Short-lived capability tokens and revocation | [Three-layer auth model](#three-layer-auth-model) |
| Transport or infrastructure operators read message payloads | End-to-end encrypted channels | [Encrypted channels](#encrypted-channels) |
| Notifications sent to devices that did not opt in, or leaked provider keys | Push identity and credential controls | [Push security](#push-security) |
## Secrets
Anyone holding an app secret can publish to every channel, sign any private or presence
subscription, and read history for that app. Treat it like a database root password.
Keep these server-side only:
- app secret
- encryption master key
- webhook signing tokens
- database credentials
- Redis or adapter credentials
- push provider credentials
- APNs private key
- FCM service account JSON
- Web Push VAPID private key
Use secret managers or Kubernetes Secrets. Do not bake secrets into images or public environment bundles.
The app key ships in every client bundle and only identifies the app. The app secret must never
reach a browser, a mobile binary, or client-side logs. If it leaks, rotate it.
### App secrets at rest (MySQL and PostgreSQL)
By default the MySQL and PostgreSQL app managers store each app's `secret` as plaintext in the
`applications` table. To keep secrets out of the database, its backups, and dumps, give Sockudo
a 32-byte key, encoded as base64 or hex, through one of:
- `SOCKUDO_APP_SECRET_ENCRYPTION_KEY`: the key itself
- `SOCKUDO_APP_SECRET_ENCRYPTION_KEY_FILE`: a file that holds it, for example one mounted from a
secret store
With a key set, the app manager writes each secret to the `secret_enc` column, sealed with
AES-256-GCM (`v1:`), and writes the plaintext `secret` column empty. The app id is bound
into the ciphertext, so a `secret_enc` value copied onto another app's row fails to decrypt.
The column is added automatically on startup. Rows that only have a plaintext `secret` keep
working, so you can migrate one app at a time:
1. Generate a key and point every node at it.
```bash
openssl rand -base64 32 > /run/secrets/sockudo-app-secret-key
export SOCKUDO_APP_SECRET_ENCRYPTION_KEY_FILE=/run/secrets/sockudo-app-secret-key
```
2. Seal each app's secret. The secret is read from stdin, so it never appears in shell history or
the process list.
```bash
# Prints the secret_enc value for app "my-app"; the secret is read from stdin.
printf '%s' "$APP_SECRET" | sockudo --encrypt-app-secret my-app
```
3. Store the sealed value and clear the plaintext column.
```sql
UPDATE applications SET secret_enc = 'v1:...', secret = '' WHERE id = 'my-app';
```
Sockudo fails closed. A row with `secret_enc` but no configured key, or a key that does not
match, is rejected: lookups for that app return an error, and app listings skip it. An invalid
key setting stops startup. Every node needs the same key. To rotate the key, re-encrypt each
row with the new key, then switch nodes over.
## Disabling an app
Setting `enabled = false` on an app, or deleting it, rejects new connections once the app cache
expires. Each node also re-checks the apps of its live connections every
`app_manager.availability_check_interval_seconds` (default `15`). It closes sockets of disabled
apps with code `4003` and sockets of deleted apps with `4001`. Clients do not reconnect after
either code. The worst-case delay is the check interval plus `[app_manager.cache] ttl`. If an
app lookup fails, for example during a database outage, the check keeps the connections open.
This is your kill switch for a compromised or abusive tenant: disable the app in the app manager and
its traffic stops within that window, without a restart.
## Origin policy
**Protects against:** cross-site WebSocket hijacking. Browsers let any page open a WebSocket to any
host, and the app key is public, so without an allowlist a malicious site can connect with your key
from a visitor's browser.
Configure allowed origins for browser clients. Do not use wildcard origins for authenticated apps.
```toml title="config.toml"
[[app_manager.array.apps]]
id = "app-id"
key = "app-key"
secret = "app-secret"
[app_manager.array.apps.policy.channels]
allowed_origins = [
"https://app.example.com",
"https://admin.example.com"
]
```
When the list is non-empty, a connection whose `Origin` header is missing or not on the list is
rejected with error code `4009` and counted in `sockudo_connection_errors_total` with
`error_type="origin_not_allowed"`. Matching is case-insensitive, and `*.example.com` patterns match
subdomains. An empty list, `*`, or `any` allows every origin.
Browsers set `Origin` and scripts cannot change it, but a non-browser client can send any value.
The origin allowlist stops other websites from using your key; channel authorization is what stops
an attacker with their own client.
## Client messages
**Protects against:** a subscribed client injecting events that other subscribers trust as if your
server sent them.
Client events allow subscribed clients to publish events to private or presence channels. Disable them unless the product explicitly needs peer-to-peer client events.
```toml title="config.toml"
[app_manager.array.apps.policy.features]
enable_client_messages = false
```
When enabled, Sockudo accepts client events only on private and presence channels, only with a
`client-` name prefix, and never with reserved `pusher:`, `pusher_internal:`, `sockudo:`, or
`sockudo_internal:` prefixes. Rate limit them with `max_client_events_per_second` under
`[app_manager.array.apps.policy.limits]`.
When enabled, validate channel authorization carefully and use event names that are easy to audit.
Receivers must treat client events as untrusted user input.
## Auth endpoints
**Protects against:** horizontal privilege escalation, where a signed-in user subscribes to a channel
that belongs to someone else. Sockudo only checks that the signature is valid; your endpoint decides
whether the user should have it.
Auth endpoints must:
1. Authenticate the current user from your session or token, never from the request body.
2. Validate `socket_id` so a signature cannot be replayed onto another connection.
3. Validate `channel_name` against an allowlist of shapes the user may join.
4. Enforce tenant and resource policy, for example that `private-account:42` belongs to user 42.
5. Sign with the server SDK.
6. Avoid logging full signatures or secrets.
Because the signature covers `socket_id` and `channel_name`, a signature captured from one session
cannot be reused for another connection or another channel.
## Three-layer auth model
Sockudo separates client authority into three layers:
| Layer | Who uses it | What it controls |
| --- | --- | --- |
| Capability token | Protocol V2 WebSocket clients | Connection identity plus `publish`, `subscribe`, `history`, and `presence` permissions by channel pattern. |
| Customer auth endpoint | Your backend and client SDK | Private, presence, encrypted channel auth, and short-lived capability-token issuance. |
| SDK callbacks | Your app code | Product-specific actions such as cancel buttons, input validation, and UI-level workflow checks before sending a permitted operation. |
Capability tokens are additive. They do not replace Pusher-compatible HMAC channel auth, signed app HTTP APIs, or per-app mutation policies. A token-authenticated mutation must have `publish` for the channel and must also satisfy the existing own/any mutable-message policy.
Tokens are HS256-signed, at most 8 KiB, and live at most 24 hours, with 30 seconds of allowed clock
skew. Issue them with short lifetimes so a leaked token expires quickly on its own.
Revocation is stored through the configured Sockudo cache by `jti` or `client_id`. Use a shared cache such as Redis when multiple Sockudo nodes must observe revocations consistently. A node that receives `POST /apps/{appId}/revocations` closes matching local token-authenticated sockets immediately; other nodes reject the revoked token on the next validation or refresh once the cache entry is visible.
With `cache.driver = "memory"` on a cluster, a revoked token is rejected only on the node that
received the revocation. Every other node keeps accepting it until it expires.
## Encrypted channels
**Protects against:** anyone between your backend and the subscriber reading payloads, including
Sockudo operators, log pipelines, and history stores. Sockudo relays and stores only ciphertext.
Encrypted channels protect payload contents from the transport. They do not replace channel authorization.
Rules:
- channel names must start with `private-encrypted-`
- only one encrypted channel should be targeted per encrypted publish
- keep the encryption master key server-side
- rotate keys with a planned client migration window
## Webhook validation
**Protects against:** forged webhook requests. Your webhook URL is reachable from the internet, so an
attacker can POST fake `member_added` or `ai_run_ended` events unless you verify the sender.
Each delivery carries `X-Pusher-Key` and `X-Pusher-Signature`, a hex HMAC SHA-256 of the raw body
keyed with the app secret. Validate webhooks with the raw body:
```ts
const webhook = sockudo.webhook({ rawBody, headers });
if (!webhook.isValid()) {
return res.status(401).end();
}
```
Compute the signature over the exact bytes received. A framework that parses and re-serializes the
JSON first changes the bytes and makes every signature check fail, which tempts teams to disable it.
Support key rotation by accepting old and new webhook tokens during the migration window. Delivery
details are in [Webhooks](/docs/server/webhooks#verify-signatures).
## Push security
Push is a privileged subsystem. Treat it like an outbound messaging platform:
- device registration must be tied to an authenticated user or device identity flow
- provider tokens should be updateable and revocable
- channel push subscriptions must obey the same authorization model as realtime channels
- provider credentials must never be exposed to clients
- push publish APIs should be rate limited per tenant and workflow
- scheduled push should have cancellation and audit trails
Sockudo enforces part of this for you. A signed push request acts as `push-admin` by default.
Sending `X-Sockudo-Push-Capability: push-subscribe` narrows it: the request can only act on an
existing device and must present that device's `X-Sockudo-Device-Identity-Token`. The server stores
only a hash of that token. Publishing push always requires `push-admin`, so a device-scoped flow
cannot broadcast notifications.
## Rate limiting
**Protects against:** connection floods, credential stuffing on auth endpoints, noisy tenants, and
runaway clients exhausting shared capacity.
Rate limit:
- connection attempts by IP
- auth requests by user and IP
- client events by socket
- HTTP publish by app
- push registration and publish by tenant
- webhook retries
Sockudo enforces the server-side pieces directly:
```toml title="config.toml"
[rate_limiter]
enabled = true
driver = "redis"
[rate_limiter.api_rate_limit]
max_requests = 100
window_seconds = 60
trust_hops = 0
[rate_limiter.websocket_rate_limit]
max_requests = 20
window_seconds = 60
trust_hops = 0
```
Rejected HTTP requests return `429` with `Retry-After` and `X-RateLimit-*` headers, and every
decision is counted in `sockudo_rate_limit_checks_total` and `sockudo_rate_limit_triggered_total`.
Auth endpoint limits live in your backend, because Sockudo never sees those requests.
With `trust_hops = 0` (the default) Sockudo keys limits on the socket peer and ignores
`X-Forwarded-For`. A positive value reads the client IP from that header, which any client can set.
Use it only when Sockudo is reachable exclusively through proxies that rewrite the header; see
[Rate limiter client IP](/docs/reference/configuration#rate-limiter-client-ip).
Use shared Redis-backed limits in multi-node deployments.
## Operational hardening
- terminate TLS at a trusted proxy or enable TLS directly
- set proxy idle timeouts above heartbeat intervals
- use readiness checks during deploys
- emit audit logs for denied auth and push admin operations
- alert on authentication failures, signature failures, and provider credential errors
- keep the metrics port (`9601` by default) off the public network
## Next steps
- [Authentication](/docs/getting-started/authentication): Build private, presence, and user auth endpoints.
- [Webhooks](/docs/server/webhooks): Signature verification and idempotent handlers.
- [Push notifications](/docs/server/push-notifications): Device identity, credentials, and provider setup.
- [Configuration reference](/docs/reference/configuration): Rate limiter, capability token, and app policy keys.
---
# Observability (https://sockudo.io/docs/server/observability)
> Monitor Sockudo connections, subscriptions, publishes, fanout, history, recovery, webhooks, and push notifications.
This page lists the metrics Sockudo exports, what each one tells you, and which conditions are worth
paging on. Set it up before production traffic: realtime failures are often silent (a node that stops
hearing its peers, a history store that degrades, a webhook queue that backs up) and only metrics
make them visible.
Observability is part of the runtime contract. A realtime system should tell operators when it is connected, degraded, delayed, retrying, or dropping work.
## Metrics endpoint
```bash
curl http://127.0.0.1:9601/metrics
```
Prometheus metrics are served on the dedicated `[metrics]` listener (`9601` by default), not on the
WebSocket and HTTP API port, so you can keep them on a private network.
```toml title="config.toml"
[metrics]
enabled = true
driver = "prometheus"
host = "0.0.0.0"
port = 9601
[metrics.prometheus]
prefix = "sockudo_"
```
Scrape the endpoint with Prometheus and label metrics by environment, region, node, adapter, and app where possible. Sockudo emits metrics through the `metrics-rs` recorder and exposes them through the Prometheus exporter by default.
How to read the tables below:
- Names assume the default `sockudo_` prefix. `process_*` metrics are never prefixed.
- Most Sockudo metrics carry `app_id` and `port` labels; extra labels are listed per metric.
- A counter only goes up: graph it with `rate()` or `increase()`. A gauge is a current value. A
histogram exposes `_bucket`, `_sum`, and `_count` series: use `histogram_quantile()` for latency.
- Prometheus adds `instance` and `job` at scrape time; use `instance` to compare nodes.
For OTLP export of the same metrics plus traces and logs, see
[Observability configuration](/docs/reference/configuration#observability).
## TCP metrics exporter
For live debugging or sidecar consumers, Sockudo can also fan out metric events over TCP using `metrics-exporter-tcp`. This exporter streams protobuf-encoded metric events to connected clients; it is useful for local inspection and custom collectors, but Prometheus scraping should remain the primary production monitoring path.
```toml title="config.toml"
[metrics.tcp_exporter]
enabled = true
host = "127.0.0.1"
port = 5000
buffer_size = 1024
```
The TCP exporter has bounded buffering by default. When buffers fill, event samples can be dropped to avoid blocking the server, so do not use it as the only source for alerts or SLO dashboards.
## Core signals
| Area | Watch |
| --- | --- |
| Connections | active sockets, connection attempts, disconnects, connection errors by type |
| Subscriptions | subscribe and unsubscribe rates, active channels by type |
| Publish | HTTP calls, idempotency hits, rate-limit rejections |
| Fanout | broadcast latency, cross-node request latency, transport reconnects and drops |
| Recovery | resume successes by source, resume failures by code |
| History | writes, write failures, queue depth, degraded and reset-required channels |
| Webhooks | queue depth and settled outcomes for the `webhooks` queue |
| Push | accepted, dispatched, provider failures, queue age |
### Connections and traffic
| Metric | Type | Labels | What it tells you |
| --- | --- | --- | --- |
| `sockudo_connected` | gauge | | Sockets currently connected to this node. Compare across `instance` to spot imbalance. |
| `sockudo_new_connections_total` | counter | | Connection attempts. A spike after a deploy is normal reconnect churn. |
| `sockudo_new_disconnections_total` | counter | | Disconnects. Rising faster than connections means clients are leaving. |
| `sockudo_connection_errors_total` | counter | `error_type` | Failed connections, for example `origin_not_allowed`. |
| `sockudo_memory_pressure_rejections_total` | counter | `port` only | New sockets refused by memory-pressure admission. |
| `sockudo_memory_pressure_shedding` | gauge | none | `1` while the node is shedding new connections. |
| `sockudo_ws_messages_received_total`, `sockudo_ws_messages_sent_total` | counter | `app_id`, `port` | Messages received from connections, and application messages addressed to connections. Outbound control frames are counted separately in `sockudo_ws_control_frames_total`; see [WebSocket delivery metrics](#websocket-delivery-metrics). |
| `sockudo_socket_received_bytes`, `sockudo_socket_transmitted_bytes` | counter | | WebSocket bandwidth. |
| `sockudo_http_calls_received_total` | counter | | REST API calls, including publishes. |
| `sockudo_http_received_bytes`, `sockudo_http_transmitted_bytes` | counter | | REST API bandwidth. |
### Channels and fanout
| Metric | Type | Labels | What it tells you |
| --- | --- | --- | --- |
| `sockudo_channel_subscriptions_total`, `sockudo_channel_unsubscriptions_total` | counter | `channel_type` | Subscription churn by channel type. |
| `sockudo_active_channels` | gauge | `channel_type` | Channels with at least one local subscriber. |
| `sockudo_broadcast_latency_ms` | histogram | `channel_type`, `recipient_count_bucket` | End-to-end broadcast latency. Your main fanout SLI. |
| `sockudo_horizontal_adapter_resolve_time` | histogram | | Time for peers to answer cross-node requests such as counts and presence. |
| `sockudo_horizontal_adapter_sent_requests`, `_received_requests`, `_received_responses`, `_resolved_promises` | counter | | Cross-node request volume. |
| `sockudo_horizontal_adapter_uncomplete_promises` | counter | `request_type` | Cross-node requests not answered by every node. Counts may be partial. |
| `sockudo_horizontal_transport_reconnections_total` | counter | `driver` only | Adapter transport reconnects, including failed Redis subscriber PINGs. |
| `sockudo_horizontal_transport_messages_dropped_total` | counter | `driver` only | Cross-node messages dropped by the transport. Any increase is lost fanout. |
| `sockudo_horizontal_transport_queue_depth` | gauge | `driver` only | Transport backlog. |
| `sockudo_nats_events_total` | counter | `event` only | NATS client events such as `disconnected` and `slow_consumer`. |
### Publish controls
| Metric | Type | Labels | What it tells you |
| --- | --- | --- | --- |
| `sockudo_rate_limit_checks_total` | counter | `limiter_type`, `request_context` | Rate-limit decisions made. |
| `sockudo_rate_limit_triggered_total` | counter | `limiter_type`, `request_context` | Requests rejected by a rate limit. |
| `sockudo_idempotency_publish_total` | counter | | Publishes that carried an idempotency key. |
| `sockudo_idempotency_duplicates_total` | counter | | Retries answered from the idempotency cache. A high ratio points at aggressive client retries. |
| `sockudo_ephemeral_messages_total` | counter | | V2 ephemeral messages delivered. |
| `sockudo_event_filter_suppressed_total`, `sockudo_echo_suppressed_total` | counter | | V2 deliveries skipped by event filters or echo control. |
### History, recovery, and versioned messages
| Metric | Type | Labels | What it tells you |
| --- | --- | --- | --- |
| `sockudo_history_writes_total`, `sockudo_history_write_failures_total` | counter | | Durable history appends and failures. Failures put channels at risk of degrading. |
| `sockudo_history_write_latency_ms` | histogram | | Durable write latency. |
| `sockudo_history_queue_depth` | gauge | | History writer backlog. Growth means the backend is slower than publish rate. |
| `sockudo_history_retained_messages`, `sockudo_history_retained_bytes` | gauge | | Retained history volume. |
| `sockudo_history_evictions_total`, `sockudo_history_evicted_bytes_total` | counter | | Rows removed by retention. |
| `sockudo_history_degraded_channels` | gauge | | Channels whose durable stream cannot currently prove continuity. |
| `sockudo_history_reset_required_channels` | gauge | | Channels that need an operator reset before history and recovery resume. |
| `sockudo_history_recovery_success_total` | counter | `source` (`hot`, `cold`) | Successful resumes, from the hot buffer or durable history. |
| `sockudo_history_recovery_failures_total` | counter | `code` | Failed resumes: `position_expired`, `continuity_unverifiable`, `stream_reset`, `persistence_unavailable`. |
| `sockudo_versioned_message_mutations_total` | counter | `action`, `result` | Update, delete, and append attempts and outcomes. |
| `sockudo_versioned_message_retrieval_total` | counter | `surface`, `result` | Latest-version and version-list reads. |
| `sockudo_versioned_history_substitution_total` | counter | `result` | History rows replaced by their latest version. |
| `sockudo_presence_history_*` | mixed | | The same write, retention, queue, degraded, and reset-required signals for presence history. |
### Annotations and delta compression
| Metric | Type | Labels | What it tells you |
| --- | --- | --- | --- |
| `sockudo_annotations_published_total`, `sockudo_annotations_deleted_total` | counter | `channel`, `type` | Annotation create and delete volume. |
| `sockudo_annotation_summary_deliveries_total` | counter | `channel` | Summary messages delivered. |
| `sockudo_annotation_summary_clipped_total` | counter | `channel`, `type` | Summaries clipped by size limits. |
| `sockudo_annotation_projection_rebuild_total`, `_duration_seconds` | counter, histogram | `channel` | Full projection rebuilds and their cost. |
| `sockudo_delta_compression_bandwidth_saved_bytes`, `_original_bytes` | counter | `channel_name` | Bandwidth saved by delta compression. |
| `sockudo_delta_compression_full_messages_total`, `_delta_messages_total` | counter | `channel_name` | Full versus delta frames sent. |
| `sockudo_delta_cluster_coordination_backend_up` | gauge | `backend` only | Health of the delta coordination backend. |
Annotation and delta compression metrics are labeled by channel. With many short-lived channels they
create many series; drop or aggregate those labels in your scrape config if Prometheus memory grows.
### Queues, push, and runtime
| Metric | Type | Labels | What it tells you |
| --- | --- | --- | --- |
| `sockudo_queue_depth` | gauge | `backend`, `queue`, `state` | Jobs by state (`ready`, `active`, `delayed`, `dead-letter`, `completed`). Redis-family queue backends. |
| `sockudo_queue_settled_total` | counter | `backend`, `queue`, `outcome` | Job outcomes: `completed`, `retry`, `dead-letter`, `panic-or-lease-lost`. |
| `sockudo_queue_enqueue_total`, `sockudo_queue_claimed_total` | counter | `backend`, `queue` | Jobs added and picked up. |
| `sockudo_queue_processing_duration_seconds` | histogram | `backend`, `queue` | Job processing time. |
| `sockudo_push_publish_accepted_total` | counter | see push docs | Push publishes admitted. |
| `sockudo_push_dispatched_total`, `sockudo_push_provider_failures_total` | counter | see push docs | Provider sends and failures by provider and failure class. |
| `sockudo_push_queue_oldest_age_seconds` | gauge | see push docs | Age of the oldest push work item; the best push backlog signal. |
| `sockudo_tokio_active_tasks`, `sockudo_tokio_worker_busy_ratio` | gauge | `worker` on busy ratio | Runtime saturation. Busy ratio near 1.0 on every worker means the node is CPU bound. |
| `process_resident_memory_bytes`, `process_open_fds` | gauge | none | Memory and file descriptors. Each socket uses a descriptor; compare with `process_max_fds`. |
The full push metric catalog, with labels and push-specific alerts, is in
[Push operations](/docs/server/push-operations).
### WebSocket delivery metrics
| Metric | Type | Labels | Meaning |
| --- | --- | --- | --- |
| `sockudo_ws_messages_sent_total` | counter | `app_id`, `port` | Application messages addressed to connections. Each node counts only its own local recipients, so `sum` across nodes is the cluster delivery total. Control frames are excluded. |
| `sockudo_ws_control_frames_total` | counter | `app_id`, `port`, `frame_type` | Control frames sent through the connection handler. `frame_type` is one of `pong`, `error`, `connection_established`, `ack`, `auth`, `cache_miss`, `delta_negotiation`, `history`, `other`. |
| `sockudo_ws_message_bytes` | histogram | `app_id`, `port` | Serialized size of each published application message, buckets 64 B to 1 MiB. |
| `sockudo_socket_transmitted_bytes` | counter | `app_id`, `port` | Bytes addressed to connections (message size × recipients). |
How these are counted:
- Deliveries are counted at address time, before the socket write. Delta conflation and tag
filtering can still skip individual writes, so the counter can slightly exceed actual writes.
This matches Pusher's addressed-delivery semantics.
- `sockudo_ws_message_bytes` is observed once per published message on the origin node, never on
receiving nodes or per recipient. It is a per-message size distribution; use
`sockudo_socket_transmitted_bytes` for total bytes. A Prometheus histogram cannot report `max`.
- Presence `member_added`, `member_removed` and `presence_update` events are counted on remote
nodes but not on the node that handled the join or leave.
- Server-initiated `pusher:ping` frames are not counted in `sockudo_ws_control_frames_total`.
- A watch on `frame_type="other"` flags new protocol events that need a named `frame_type`.
When `ably-compat` is enabled, two lock-free operator snapshots supplement Prometheus:
- `/operator/stats/ably-runtime` reports queued messages/bytes, overflow and continuity loss,
total encoded frames, shared data-projection encodes (`data_encoded`), fanout counts, replay
source and recovery backend calls, duplicate suppression, backend/degraded/reset state, expiry,
and derived-filter cache pressure. `data_encoded` excludes ACK, heartbeat, connection, and
channel-control frames, so it is the counter used to verify one encode per active data format.
- `/operator/stats/aggregation` reports the bounded stats queue capacity, backlog, accepted,
flushed, dropped, and failed observations.
These endpoints expose counters only. They never return credentials, device tokens, channel
payloads, or raw encrypted values.
## AI Transport
AI Transport observability is domain-blind. Sockudo reads only the well-known `extras.ai.transport` headers and never labels metrics by channel, run ID, message ID, or client ID.
Headers are validated once and then passed as a typed borrowed view to metric and webhook
classification. Empty `run-client-id` and `step-client-id` unknown-owner sentinels remain
unchanged on the wire but are treated as absent when deriving an identity.
Metrics exposed at `/metrics` include:
- `sockudo_ai_runs_started_total`
- `sockudo_ai_runs_ended_total{reason}`
- `sockudo_ai_cancel_signals_total`
- `sockudo_ai_active_streams`
- `sockudo_ai_stream_duration_seconds`
- `sockudo_ai_stream_bytes_total`
- `sockudo_ai_messages_rejected_total{code}`
- `sockudo_ai_messages_unparseable_total`
- `sockudo_appends_received_total`
- `sockudo_appends_delivered_total`
- `sockudo_rollup_ratio`
- `sockudo_flush_latency`
- `sockudo_history_recovery_success_total{source}`
- `sockudo_history_recovery_failures_total{code}`
- `sockudo_versioned_message_mutations_total{action,result}`
- `sockudo_versioned_message_retrieval_total{surface,result}`
The four production signals are run outcomes, stream rate, rejects by code, and rollup efficiency. Use `docs/public/grafana/ai-transport-observability.json` as the starting Grafana dashboard.
`sockudo_rollup_ratio` is the number of input appends folded into one delivered frame, so higher
means rollup is saving more fanout. `sockudo_flush_latency` is in milliseconds and should stay near
the configured rollup window.
Active-stream gauges use exact atomic insertion/removal deltas. Tracker entries that never receive
a terminal event expire at the configured AI orphan TTL and decrement the same gauge exactly.
AI lifecycle webhooks are awaited into the configured bounded webhook queue, which supplies
backpressure and retry handling; Sockudo does not spawn an unbounded Tokio task per event or retain
AI events in the optional in-process batching vector.
Dashboard starter panels:
| Panel | Metric |
| --- | --- |
| Active AI streams | `sockudo_ai_active_streams` |
| Run outcomes | `sockudo_ai_runs_started_total`, `sockudo_ai_runs_ended_total` |
| Rejects by code | `sockudo_ai_messages_rejected_total` |
| Rollup efficiency | `sockudo_appends_received_total` vs `sockudo_appends_delivered_total`, `sockudo_rollup_ratio` |
| Rollup latency | `sockudo_flush_latency` |
| Recovery health | `sockudo_history_recovery_success_total`, `sockudo_history_recovery_failures_total` |
| Durable state | `sockudo_history_degraded_channels`, `sockudo_history_reset_required_channels` |
| Push backlog | push queue/status/provider metrics from the push dashboard |
For support escalation, capture the channel, wall-clock time window, verified `clientId`, and first error code before collecting logs. Do not ask customers for provider payloads unless the codec layer explicitly requires them.
## Persist completed AI runs
For Ably-style production persistence, persist completed runs from your backend instead of making Sockudo a domain store:
1. Enable the `ai_run_ended` webhook for the app.
2. On `reason: "complete"`, query the history endpoint for the channel around the webhook time.
3. Select messages with the same `extras.ai.transport.run-id`.
4. Store the reduced transcript in your application database.
```ts {13-15}
app.post("/sockudo/webhooks", async (req, res) => {
for (const event of req.body.events) {
if (event.name !== "ai_run_ended" || event.reason !== "complete") continue;
const history = await sockudo.channelHistory(event.channel, {
limit: 1000,
direction: "newest_first",
});
await storeCompletedRun({
runId: event.run_id,
channel: event.channel,
items: history.items.filter((item) => {
return item.message?.extras?.ai?.transport?.["run-id"] === event.run_id;
}),
});
}
res.sendStatus(200);
});
```
Each history item wraps the delivered message in its `message` field, so AI Transport headers are at
`item.message.extras.ai.transport`. The page `limit` is capped by the channel's history
`max_page_size`; follow `next_cursor` for long runs. Verify the webhook signature before trusting the
event, as described in [Webhooks](/docs/server/webhooks#verify-signatures).
## Logs
Use structured logs for events that operators need to investigate:
```json
{
"level": "warn",
"target": "sockudo_push",
"app_id": "app-id",
"publish_id": "pub_123",
"provider": "apns",
"error": "BadDeviceToken"
}
```
Avoid logging secrets, raw auth signatures, provider tokens, or encrypted payloads. Log filters,
JSON output, and the data-safety policy are in the [logging reference](/docs/reference/logging).
## Grafana dashboards
Recommended panels:
- active connections by node
- connection churn
- publish rate by app
- fanout latency histogram
- subscription auth failure rate
- recovery success ratio
- replay buffer pressure
- queue depth for webhooks and push
- push provider failure rate by provider
- APNs, FCM, Web Push latency by outcome
- AI run outcomes by reason
- AI stream duration and active streams
- AI reject codes and unparseable headers
- AI append rollup efficiency
## Alerts
Alert on symptoms operators can act on:
- readiness failures
- adapter connection loss
- rising publish failures
- high auth rejection rate after deploy
- recovery success ratio dropping
- history write failures
- webhook retry backlog
- push queue backlog
- push provider credential failures
- push delivery status callback failures
Starting rules, to tune against your own baseline:
| Alert | Expression | Why it matters |
| --- | --- | --- |
| Cross-node fanout loss | `increase(sockudo_horizontal_transport_messages_dropped_total[5m]) > 0` | Subscribers on some nodes missed messages. |
| Adapter flapping | `increase(sockudo_horizontal_transport_reconnections_total[10m]) > 3` | The adapter link is unstable; expect gaps until it settles. |
| Durable history degraded | `max(sockudo_history_degraded_channels) > 0` for 5m | Recovery and history reads on those channels fail closed. |
| Reset required | `max(sockudo_history_reset_required_channels) > 0` | An operator must reset the stream; it will not heal on its own. |
| History write failures | `increase(sockudo_history_write_failures_total[5m]) > 0` | The history backend is rejecting writes. |
| Recovery failure ratio | `sum(rate(sockudo_history_recovery_failures_total[10m])) / (sum(rate(sockudo_history_recovery_success_total[10m])) + sum(rate(sockudo_history_recovery_failures_total[10m]))) > 0.05` | Reconnecting clients are falling back to full backfill. Break down by `code`. |
| Fanout latency | `histogram_quantile(0.99, sum by (le) (rate(sockudo_broadcast_latency_ms_bucket[5m]))) > 250` | p99 delivery is slower than your SLO. |
| Memory shedding | `max(sockudo_memory_pressure_shedding) == 1` for 2m | The node is refusing new connections. |
| Webhook backlog | `sockudo_queue_depth{queue="webhooks",state="ready"}` above baseline for 10m | Webhook consumers are falling behind or failing. |
| Webhook dead letters | `increase(sockudo_queue_settled_total{queue="webhooks",outcome="dead-letter"}[15m]) > 0` | Events were given up on after all retries. |
| Rate limiting | `sum(rate(sockudo_rate_limit_triggered_total[5m])) by (app_id)` well above baseline | A client or tenant is being throttled. |
Raw counters scale with traffic. Ratios such as recovery failures over attempts stay meaningful as
your user base grows and do not page you on a busy day.
## Push status workflow
Push publishes are asynchronous. Store the `publishId` returned by the API when a business workflow needs support visibility.
```ts
const response = await sockudo.publishPush({
recipients: [{ type: "channel", channel: "orders" }],
payload: { title: "Order updated", body: "Packed" },
});
console.log(response.publishId);
```
Then inspect status with a signed request, for example through the server SDK:
```ts
const status = await sockudo.getPublishStatus(response.publishId);
```
This calls `GET /apps/{app_id}/push/publish/{publish_id}/status`, which requires push admin.
Status records should be retained long enough for customer support and incident review. Retention is
`push.publish_status_ttl_days` (default `30`).
## Next steps
- [Push operations](/docs/server/push-operations): Push metric catalog, alerts, and provider runbooks.
- [Logging reference](/docs/reference/logging): Filters, JSON output, and what is never logged.
- [Scaling](/docs/server/scaling): Which signals to watch during rolling deploys.
- [AI Transport production ops](/docs/server/ai-transport-production-ops): Runbooks for AI streams, rollup, and orphans.
---
# History and recovery (https://sockudo.io/docs/server/history-recovery)
> Use Protocol V2 stream continuity, replay buffers, channel history, presence history, rewind, and mutable messages.
This page explains how Sockudo gives clients a gapless view of a channel: recovering messages missed
during a disconnect, sending recent history to late joiners, and serving durable history to your
backend. Read it when you enable Protocol V2 recovery or history, or when you need to know exactly
what a client can trust after a reconnect.
Sockudo separates connection recovery from durable history. Recovery keeps a connection continuous after a short interruption. History is an application feature for reading and reconstructing past state.
## Mental model
Every V2 broadcast on a channel gets a position: a `stream_id` naming the stream's generation and a
`serial` that increases with each message. A client that remembers its last position can ask
"send me everything after serial 42 on stream `orders:main`", and Sockudo can check whether it still
holds every message in that range.
Two stores can answer that question:
| | Hot replay buffer | Durable history |
| --- | --- | --- |
| Where | In each node's memory | Shared backend (`postgres`, `mysql`, `dynamodb`, `surrealdb`, `scylladb`, or `memory`) |
| Holds | The last `connection_recovery.max_buffer_size` messages per channel, for `buffer_ttl_seconds` | Messages within `history.retention_window_seconds` and the per-channel caps |
| Used for | Fast resume after a short disconnect | Cold resume, rewind, history reads, versioned messages |
| Survives | Nothing: a restart or a reconnect to another node misses it | Node restarts and cross-node reconnects |
**Fail closed** is the governing rule. If Sockudo cannot prove that a replay contains every message
in the gap, in order, it refuses the resume and says why, instead of sending a partial replay that
the client would mistake for complete. The client then resubscribes and backfills explicitly.
```mermaid
flowchart TB
A[Client sends resume] --> B{Hot buffer covers gap?}
B -- yes --> H[Replay from buffer]
B -- no --> C{History proves continuity?}
C -- yes --> D[Replay from history]
C -- no --> F[resume_failed with code]
H --> S[resume_success]
D --> S
```
## Recovery
Protocol V2 broadcasts carry stream metadata:
```json
{
"event": "order.updated",
"channel": "orders",
"data": { "id": "ord_123" },
"message_id": "msg_01HX",
"stream_id": "orders:main",
"serial": 42
}
```
Clients store recovery positions and provide them during reconnect. Sockudo replays missed messages if continuity can be proven.
For V2 clients, the subscription acknowledgement carries the first recovery position when recovery is enabled. Keep that position even before the channel receives any application event. A reconnect can then resume from the subscribed-but-idle checkpoint and replay messages published while the client was disconnected.
```ts
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
protocolVersion: 2,
connectionRecovery: true,
});
client.bind("sockudo:resume_success", (payload) => {
console.log(payload.recovered, payload.failed);
});
```
Each entry in `recovered` names the `channel`, the `source` (`hot` or `cold`), and the terminal
`position` the client should store. Each entry in `failed` names the `channel`, a `code`, and a
`reason`.
### Resume failure codes
| `code` | Meaning | What the client should do |
| --- | --- | --- |
| `position_expired` | The gap is older than retention, or larger than `max_buffer_size` (`reason: "recovery_window_exceeded"`). | Resubscribe, then backfill with `until_attach: true`. |
| `continuity_unverifiable` | The position is ahead, non-contiguous, duplicate, or non-progressing, or the stream head is unknown. | Resubscribe and backfill. Do not advance the stored cursor. |
| `stream_reset` | The stream ID changed, for example after an operator reset. | Discard local state for the channel, resubscribe, and reload. |
| `persistence_unavailable` | The durable tier could not be read. | Resubscribe and backfill once history is reachable again. |
If a channel emits `sockudo:resume_failed` with `code: "position_expired"`, the hot replay buffer and durable recovery window could not prove continuity. Resubscribe, then backfill with V2 channel history using `until_attach: true` so the history page is bounded at the subscription's attach serial.
`code: "continuity_unverifiable"` means Sockudo rejected an ahead, non-contiguous, duplicate, or non-progressing recovery position. Treat it the same way: resubscribe and backfill instead of advancing the client cursor.
After `resume_failed`, live messages continue but the gap is not filled. UI that shows the channel as
up to date without a backfill will silently miss messages.
### How the two tiers work
Recovery is two-tier:
1. Hot replay uses the bounded per-channel replay buffer.
2. Durable recovery uses `HistoryStore` when `stream_id`, serial bounds, and retention prove
continuity.
Both tiers enforce `connection_recovery.max_buffer_size` for each recovered
channel. A larger gap fails with `position_expired` and
`reason: "recovery_window_exceeded"` before Sockudo builds or partially sends an
unbounded replay. When the SDK has already reattached a subscription, a bounded
continuity gate holds concurrent live messages until replay has been enqueued;
the aggregate `resume_success` is enqueued before those live messages are
released. Gate overflow closes that connection instead of delivering frames out
of order.
Both tiers replay the Protocol V2 wire message, not Sockudo's durable storage
wrapper. Durable rows are decoded through the shared history projection first,
so recovery and ordinary history reads preserve the same IDs, publisher fields,
version projection, stream ID, and serial.
Versioned mutations keep the delivery position reserved by the native version
service. The hot replay buffer adopts that position instead of allocating a
second serial. Recovery validates every serial in the returned range; a missing
position or mixed stream generation falls through to durable recovery and then
fails closed if the durable tier cannot prove the same continuity.
The native V2 attach and resume gates register the subscriber before capturing
the durable high-water mark. Concurrent publishes are held in a count-and-byte-
bounded gate, shared across subscribers with immutable message references, then
drained in ordered batches without a lock held across delivery. Overflow closes
the affected connection because the live-plus-replay sequence can no longer be
proven.
If the durable stream is degraded or reset-required, Sockudo fails closed instead of pretending the
stream is continuous.
A stream is **degraded** when durable writes or reads failed and continuity is in doubt, and
**reset-required** when an operator must explicitly reset it before history and recovery resume.
Both states are exported as `sockudo_history_degraded_channels` and
`sockudo_history_reset_required_channels`; see [Observability](/docs/server/observability).
## Configuration
Both tiers are off in the built-in defaults; the shipped `config/config.toml` enables them.
```toml title="config.toml"
[connection_recovery]
enabled = true
buffer_ttl_seconds = 120
# Bounds both the per-channel hot buffer and one durable fallback replay.
max_buffer_size = 100
[history]
enabled = true
rewind_enabled = true
backend = "postgres"
retention_window_seconds = 86400
max_page_size = 100
max_messages_per_channel = 1000
[presence_history]
enabled = true
retention_window_seconds = 86400
max_page_size = 100
max_events_per_channel = 1000
```
Use a shared history backend whenever more than one node serves the same channels: `memory` history
is local to one process, so a client that reconnects to another node cannot be cold-recovered.
Raising `max_buffer_size` lets clients survive longer gaps, at the cost of memory per active channel
and larger replays on reconnect. Every key is in the
[configuration reference](/docs/reference/configuration#protocol-features).
## Rewind
Subscribe-time rewind asks Sockudo to send recent history when a client joins:
```ts
const channel = client.subscribe("orders", {
rewind: { seconds: 30 },
});
channel.bind("sockudo:rewind_complete", (payload) => {
console.log(payload.historical_count, payload.complete);
});
```
Rewind accepts a message `count` or a time window in `seconds`. `sockudo:rewind_complete` reports
`historical_count`, `live_count`, `complete`, and whether the page was truncated by retention or by
the limit.
When a V2 subscription has a compound predicate, rewind scans durable history in bounded pages until it fills the requested matching count or reaches the scan/retention boundary. Suppressed rows still advance the channel's canonical continuity position. Hot and durable recovery apply the same predicate after the client has reattached that subscription; a legacy resume sent before reattachment retains the unfiltered recovery contract. Each recovered-channel entry in `sockudo:resume_success` includes the terminal `position` so clients advance continuity even when the predicate suppresses the tail.
## Durable channel history
Server SDKs expose app-key channel history helpers. Use signed app-key HTTP history for backend jobs, administrative reads, and server SDKs.
```python
page = sockudo.get_channel_history(
"orders",
HistoryParams(limit=50, direction="newest_first"),
)
```
Opaque cursors make storage implementation details private. Store the cursor as a token and pass it back unchanged.
The HTTP endpoint, its query parameters, and the full response shape are documented in
[HTTP API: History](/docs/server/http-api#history).
Protocol V2 clients use WebSocket `channel_history` frames instead of direct HTTP history. Token-authenticated clients need the `history` capability for the requested channel. The existing app-key HTTP endpoint remains server-only; there is no client-token HTTP history surface in v1.
```json
{
"event": "sockudo:channel_history",
"data": {
"channel": "orders",
"limit": 100,
"direction": "backwards",
"cursor": null,
"start_serial": 1,
"end_serial": 100,
"until_attach": true
}
}
```
The response uses the same history payload shape as app-key HTTP history: `items`, `direction`, `limit`, `has_more`, `next_cursor`, `bounds`, `continuity`, and `stream_state`. Limits are capped at `1000` and may be lowered by channel policy. `direction` defaults to backwards/newest-first. `until_attach: true` bounds results to `history_serial <= attach_serial`, while live delivery after subscribe continues above that attach serial. Mutable messages are returned as the latest aggregated V2 message, not raw operation log rows.
### Late joins with until_attach
`until_attach` is the late-join gaplessness rule: the history page stops at the serial captured when
the subscription succeeded, and live fanout continues above that serial. Clients should reduce
history first, then live messages. In a cluster, route this request to the node that owns the live
attachment. If Sockudo cannot find that authoritative attachment position, it rejects the read
instead of silently removing the upper bound and returning a page that could overlap live delivery.
```mermaid
sequenceDiagram
participant C as Client
participant S as Sockudo
C->>S: subscribe orders
S-->>C: subscription_succeeded at serial 120
S-->>C: live message serial 121
C->>S: channel_history until_attach
S-->>C: history serials up to 120
Note over C: Apply history, then buffered live 121
```
1. Subscribe and keep the attach position from the subscription acknowledgement.
2. Buffer live messages that arrive while history loads.
3. Request `channel_history` with `until_attach: true`, paging with `next_cursor` as needed.
4. Apply history oldest to newest, then the buffered live messages, skipping any serial already seen.
## Presence history
Presence history records joins, leaves, causes, and continuity metadata. It is different from the current presence member list.
Protocol projections must write transitions through the native presence
service. Current membership remains available if the durable tier fails, but
the tracking store marks that channel degraded. Continuity-sensitive history
reads then fail closed until the durable stream is healthy or explicitly reset;
they must not return a partial page as complete.
```ts
const channel = client.subscribe("presence-lobby");
const page = await channel.history({
limit: 50,
direction: "newest_first",
});
const snapshot = await channel.snapshot({ atSerial: 4 });
```
Client helpers call your backend proxy. They do not sign Sockudo REST requests directly.
## Mutable messages
Protocol V2 mutable messages use action events:
- `sockudo:message.update`
- `sockudo:message.delete`
- `sockudo:message.append`
Clients reduce those events into local state. Server SDKs can fetch the latest visible state or list preserved versions.
When a V2 subscription uses rewind, historical rows that point at mutable messages are delivered as the latest visible version, so late joiners see accumulated appended content rather than only the original create payload.
```ts
const latest = await channel.getMessage("42");
const versions = await channel.getMessageVersions("42", {
limit: 20,
direction: "oldest_first",
});
```
## Push and history
Push notifications should include stable identifiers that let the app fetch authoritative state after the user opens the notification.
```json
{
"title": "Order updated",
"body": "Order ord_123 is now packed",
"data": {
"channel": "orders",
"message_serial": "42",
"order_id": "ord_123"
}
}
```
Do not encode the entire durable state into the push payload. Use push to wake the app, then read the latest message or application API state.
Push delivery is best effort and can arrive late or out of order. Treat the notification as a hint
and read the channel history or message version on open.
## Next steps
- [Protocol V2](/docs/clients/protocol-v2): Client-side recovery, rewind, and position handling.
- [Presence history](/docs/clients/presence-history): Read join and leave history from clients.
- [Mutable messages](/docs/server/mutable-messages): Versions, serials, and mutation authorization.
- [HTTP API](/docs/server/http-api): Signed history, state, reset, and purge endpoints.
---
# Indestructibility simulator (https://sockudo.io/docs/server/indestructibility-simulator)
> Run Sockudo's deterministic disaster simulator for durable realtime and push invariants.
The Sockudo simulator (`sockudo-sim`) is a seed-replayable disaster lab for durable Protocol V2
state: history, versioned messages, presence history, recovery, and push. Run it when you change
any of those subsystems, when you want a fast deterministic reproduction of a durability bug, or as
a pull-request and nightly gate. Every run is a pure function of its seed and flags, so any failure
can be replayed exactly.
## Mental model
The simulator runs inside one process. It drives real `sockudo-core` memory stores, calls the real
`sockudo-push` memory store and accept pipeline for durable push boundaries, injects node, network,
IO, queue, and fake-provider faults, and checks a shadow model continuously.
```mermaid
flowchart TB
S[Seed and flags] --> W[Workload generator]
W --> F[Fault scheduler]
F --> R[Real memory stores]
R -->|accepted| M[Shadow model]
R --> O[Oracles]
M --> O
O -->|violation| C[Replay command]
```
Each tick picks one weighted action, routes it through seeded faults, and applies it to the real
store first. The shadow model advances only when the real store accepts the operation. Oracles then
compare what the stores return against the shadow; any disagreement stops the run with the seed,
tick, and a replay command.
It proves that Sockudo-side durability contracts survive dropped live fanout, duplicated
delivery, node crashes, pauses, partitions, stream resets, retention purges, reconnect recovery,
storage-level dropped writes, torn multi-record writes, stale or corrupted reads, delayed commit
visibility, push queue loss, lost write responses, retryable provider failures, invalid tokens, and
repair.
The simulator deliberately does **not** claim that APNs, FCM, WebPush, HMS, WNS, browsers, mobile
OSes, radios, or devices will deliver every notification. External providers are modeled as
fallible systems. "Indestructible" means Sockudo does not lose, corrupt, double-apply logically, or
make unrecoverable an accepted Sockudo-side operation.
## Run it
1. Run one fixed profile. `--seed` makes the run replayable; without it a random seed is printed at
startup. `--ticks` defaults to `5000`.
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- --seed 42 --ticks 10000
```
or through the Makefile wrapper (defaults `SIM_SEED=12648430`, `SIM_TICKS=5000`; extra flags go
in `SIM_ARGS`):
```bash
make simulator SIM_SEED=42 SIM_TICKS=10000
```
2. Turn up the pressure with the disaster-heavy JSON profile, which raises node, queue, storage,
and provider fault rates across 9 nodes:
```bash
make simulator-disaster SIM_SEED=12648430 SIM_TICKS=50000
```
3. Hunt for new combinations with VOPR-style randomized distributions and liveness checks:
```bash
make simulator-swarm SIM_SEED=3735928559 SIM_TICKS=50000
make simulator-liveness SIM_SEED=3735928559 SIM_TICKS=50000
```
4. On failure, copy the printed replay command and follow [Reading failures](#reading-failures).
A passing run prints a one-line summary starting with `sockudo-sim: OK seed=...` with operation,
oracle, commit, recovery, fault, and upgrade counters. Add `--json` for the full report.
Every failure prints a replay command with the seed. Keep the seed, tick count, and fault arguments
unchanged when reducing a failure, or use the [shrinker](#vopr-style-maturity) to reduce it safely.
## What It Exercises
The simulator covers these high-value durability surfaces:
| Surface | Oracle |
| --- | --- |
| Protocol delivery | V1 renderings are serialized through `sockudo-protocol` and must not contain V2-only fields; V2 renderings must preserve continuity fields and strip internal idempotency keys. |
| Durable history | Reserved serials, retained rows, page cursors, stream inspection, retention purges, and reset behavior match the shadow model. |
| Connection recovery | Simulated V2 clients recover dropped live fanout from durable history unless retention has legitimately truncated the gap; stale and corrupt storage reads must fail closed or return a valid prefix, and recovery cursors must remain contiguous. |
| Versioned messages | Delivery serial replay is contiguous, version chains page in both directions, cursors round-trip through JSON, latest reads match the shadow, version/history/delivery serials stay monotonic, and `latest_by_history` preserves original history ordering. |
| Presence history | First-join/last-leave edge decisions, retained events, cursor paging, stream inspection, and reconstructed snapshots match the shadow model under deterministic reconnect churn. |
| Push workflow | Device registration, channel subscription, scheduled-job storage, publish acceptance, initial status, publish logs, and publish-id idempotency go through real `sockudo-push` memory store/pipeline APIs. The simulator then faults modeled worker queues, status transitions, and provider outcomes and checks they converge. |
| Crash/restart recovery | Restarted nodes immediately re-read durable history, versions, presence history, and push status/log/idempotency state through the same safety oracles. |
| Rolling upgrades | Opt-in upgrade runs restart nodes one by one with mixed legacy/target feature gates, schema activation, V1/V2 wire checks, before/during/after durable data counters, and push status oracles. |
It does not replace live multi-node integration or Jepsen-style external testing. It is the fast,
deterministic inner loop for the durable primitives those tests rely on.
## Outside-In Binary Chaos Harness
Sockudo also has a separate **outside-in chaos** runner for manual local experiments against actual
`sockudo` server binaries and real client traffic. It is inspired by TigerBeetle Vortex in spirit,
but it is not deterministic, not a CI gate, and not part of the simulator's seed-replayable model.
```bash
make binary-chaos CHAOS_SEED=42 CHAOS_DURATION_MS=12000
```
The Make target builds `target/debug/sockudo`, starts it as a child process with a generated local
memory-backed config, connects clients over WebSocket, publishes signed HTTP API events, and injects
bounded outside-in faults:
| Fault surface | Current behavior |
| --- | --- |
| Process restart/kill | Sends `SIGKILL`, restarts the binary, waits for `/up/app-id`, and reconnects clients. |
| Network delay/drop/duplication | By default runs clients and HTTP publishes through a local TCP proxy that applies seeded connection/chunk delay and stream drops. Duplicate publish probes remain application-level and reuse the same idempotency key. |
| Config changes/reloads | Writes a second config and applies it by process restart because Sockudo loads config at startup. |
| Push-provider fake outcomes | Optionally starts seeded `scripts/push-mock-provider.mjs`, points FCM worker env vars at it when the binary is built with push/monolith features, and can require the mock provider to produce a seeded fake outcome. |
| Recovery/reconnect | Enables Protocol V2 connection recovery, drops client connections, reconnects, and records resume success/failure counters. |
Every run writes an artifact directory under `target/outside-in-chaos/-seed-/`
containing:
| File | Purpose |
| --- | --- |
| `artifact.json` | Seed, command, replay command with effective flags, generated config paths, fault timeline, counters, publish results, provider metrics, and recovery observations. |
| `sockudo-chaos.toml` | Initial generated server config. |
| `sockudo-chaos-restart.toml` | Restart config used for the config-change fault. |
| `sockudo.log` | Captured server stdout/stderr. |
| `push-provider.log` | Captured mock provider stdout/stderr when enabled. |
Push-provider exercise is opt-in because it requires a compatible feature build. The wrapper
builds with `v2,monolith,push-fcm` and enables the flaky mock provider profile:
```bash
make binary-chaos-push CHAOS_SEED=42 CHAOS_DURATION_MS=12000
```
For network experiments, the default `--network-fault-mode proxy` is unprivileged and local-only.
Use `--network-fault-mode publisher` to limit faults to HTTP publisher delay/drop behavior, or
`--network-fault-mode off` to run only process/client/config/push chaos. The replay command in
`artifact.json` reuses the same seed, ports, timings, probabilities, and push-provider settings;
outside-in wall-clock scheduling can still vary between runs.
Sockudo's provider dispatch layer rejects private/local provider URLs by design. If that guard
prevents the server process from calling the local mock provider, the artifact records the guarded
dispatch attempt and the harness performs a direct mock-provider probe so the fake provider outcome
is still captured with the same seed.
Keep this harness local/manual only. Do not add CI, scheduled jobs, or GitHub Actions for it.
## Real-Code Boundary
The simulator is intentionally not a full server-in-a-process. Its current real-code boundary is:
| Path | Boundary |
| --- | --- |
| History | Real `MemoryHistoryStore` reservations, appends, reads, cursors, stream inspection, purges, and resets. |
| Versioned messages | Real `MemoryVersionStore` delivery reservations, version appends, replay, latest reads, paged chain reads, and `latest_by_history`. |
| Presence history | Real `MemoryPresenceHistoryStore` transition recording, dedupe, reads, cursors, stream inspection, snapshot reconstruction, and resets; the simulator separately models active connection counts for first-join/last-leave edge decisions. |
| Push durable state | Real `MemoryPushStore` device, subscription, schedule, status, publish-log, and idempotency APIs, with publish acceptance through `PushPipeline::accept_publish`. |
Still modeled: live socket fanout, horizontal transport timing, replay-buffer hot recovery,
operator routing, push worker queues after accept, provider dispatch, provider feedback, and repair
timing. Those modeled paths exist to make deterministic faults cheap and replayable; they are
checked against the durable stores above rather than replacing them.
## Deterministic IO Model
The simulator keeps deterministic IO under `crates/sockudo-simulator/src/io.rs`. The production
server does not use these wrappers; they are simulator-local boundaries around modeled IO and real
memory-store calls.
| Wrapper | Controls |
| --- | --- |
| `DeterministicClock` | The logical tick and durable timestamps. Timer advancement during quiesce jumps through due fanout and push work instead of sleeping wall-clock time. |
| `DeterministicFaultScheduler` | Seeded random choices, operation trace entries, and named fault rolls such as `fanout_drop`, `queue.ack_lost`, and `storage.push_publish.write_after_commit`. |
| Schedule harness | A separate seed domain for simulator-executed task ordering: top-level tick tasks, due fanout deliveries, due push schedules, and ready push queue items. |
| `DeterministicNetwork` | Modeled live fanout delivery order, delivery tick, dropped messages, duplicates, and delayed delivery. |
| `DeterministicQueue` | Modeled push worker queue order, delayed delivery, redelivery, lease timeout duplication, and repair requeueing. |
| `DeterministicStorage` | Storage read/write behavior around real memory-store calls: dropped writes before reservation, torn multi-record writes, fail-after-commit, lost responses, stale reads, corrupted reads, delayed commit visibility, and backend outages. |
The final JSON report includes a bounded `io_trace` with the most recent operation, fault,
scheduling, and logical-timer decisions. That trace is intentionally compact: it is for replay
orientation, not a full event log. Use the seed, tick count, mode, fault flags, and workload
weights as the canonical replay input.
To inspect scheduler and timer decisions for a seed:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--seed 42 \
--ticks 10000 \
--json | sed '1d' | jq -r '.io_trace[]'
```
To replay a saved failure capsule and print the same trace window:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file /tmp/sockudo-sim-failure.json \
--json | jq -r '.[0].io_trace[]'
```
Current limitations:
| Limitation | Why |
| --- | --- |
| No real sockets or multi-process transport | Fanout is a deterministic in-process queue so drops, duplicates, and delays are exactly replayable. |
| No wall-clock timer waits | Logical ticks make timer behavior reproducible and keep shrink/replay fast. |
| No real provider calls | Provider responses are seeded outcomes so retry, invalid-token, and lost-response paths can be covered without external services. |
| No real production storage backend faults | The simulator wraps real memory stores and injects storage behavior at the simulator boundary; Redis/Postgres/disk failures still need integration or chaos tests. |
| Bounded trace memory | `recent_trace`, `push.recent_trace`, and `io_trace` keep recent context only; failure capsules and replay commands should preserve the full seed/config. |
## VOPR-Style Maturity
The simulator has three layers:
| Layer | Command | Purpose |
| --- | --- | --- |
| Fixed safety | `make simulator` | Replays one fixed profile. Use this for direct bug reproduction. |
| Swarm safety | `make simulator-swarm` | Randomizes topology, workload weights, page sizes, retention, and fault distributions from the seed before checking safety invariants. |
| Swarm liveness | `make simulator-liveness` | Runs the same deterministic swarm profile, then asserts bounded convergence after workload generation stops. |
Swarm profiles are deterministic: the seed generates both the simulator run and the profile
distribution. A failing swarm seed can be replayed without recording a separate config.
Safety mode answers: did accepted Sockudo-side work remain correct? Liveness mode adds: after
faults stop, did durable work become available and drain within a bounded number of ticks?
Use failure capsules when saving CI failures:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--seed 42 \
--ticks 50000 \
--swarm \
--failure-artifact /tmp/sockudo-sim-failure.json
```
Use shrink mode to reduce the replay before debugging. The shrinker first proves the original
config still fails, then deterministically tries smaller tick, operation, and fault prefixes,
topology reductions, and workload/fault profile simplifications. It accepts only candidates that
still fail and writes a smaller failure capsule:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file /tmp/sockudo-sim-failure.json \
--shrink-failure \
--shrink-output /tmp/sockudo-sim-failure.shrunk.json
```
The shrink command prints the exact replay command, and the artifact stores the same command in
`replayCommand`:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file /tmp/sockudo-sim-failure.shrunk.json
```
When shrinking a `--swarm` run directly from seed flags, the artifact also includes
`seedDerivedProfile`. That preserves the original seed-generated topology, workload, and fault
profile before any shrink simplifications, while replay continues to use the explicit shrunk
`config` from the capsule. Do not pass `--swarm` when replaying a failure capsule.
## Rolling Upgrade Replay
The simulator has an opt-in upgrade risk profile for production-style rolling changes. It keeps the
run deterministic while modeling:
| Upgrade surface | Model |
| --- | --- |
| Rolling restarts | Nodes are restarted one at a time into the target generation, with at least one live node left serving traffic. Durable recovery oracles run after each upgraded node comes back. |
| Mixed config/features | Legacy nodes reject feature-gated versioned-message writes while target nodes accept them after the schema gate opens. Ordinary history, recovery, and push traffic continue through the mixed fleet. |
| V1/V2 compatibility | Mixed V1/V2 simulated clients keep receiving live and recovered deliveries. V1 projections must strip V2-only fields, and V2 projections must preserve continuity and upgrade metadata. |
| Durable before/after data | History, versioned messages, and push publishes are counted by `before_feature_change`, `during_rolling_change`, and `after_feature_change` phases in the final JSON. |
| Schema/version gates | Versioned writes are rejected before schema activation. The upgrade oracle fails if any version record appears in the pre-activation phase. |
| Push status semantics | Push publish status/log/idempotency oracles continue to run across all upgrade phases, and target nodes must use the target push-status semantics. |
Run the manual local wrapper:
```bash
make simulator-upgrade SIM_SEED=48879 SIM_TICKS=1200
```
Or replay the same profile directly and inspect upgrade coverage:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--seed 48879 \
--ticks 1200 \
--upgrade-risk-profile \
--upgrade-require-coverage \
--json | sed '1d' | jq '.upgrade, .protocol_oracles, .push'
```
Use a compressed schedule when reducing a failure or forcing the rollout to happen early in a short
manual run:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--seed 43981 \
--ticks 320 \
--nodes 4 \
--upgrade-risk-profile \
--upgrade-require-coverage \
--upgrade-schema-prepare-tick 8 \
--upgrade-start-tick 12 \
--upgrade-schema-activate-tick 18 \
--upgrade-restart-duration-ticks 2 \
--upgrade-interval-ticks 8 \
--json
```
To preserve a failing upgrade run for exact replay:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--seed 48879 \
--ticks 1200 \
--upgrade-risk-profile \
--upgrade-require-coverage \
--failure-artifact /tmp/sockudo-upgrade-failure.json
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file /tmp/sockudo-upgrade-failure.json \
--json
```
The replay command printed on invariant failure includes the upgrade timing flags. The full failure
artifact remains the canonical replay input when workload weights, fault probabilities, topology, or
retention settings were changed in addition to the upgrade flags.
## Fault Model
The default workload is a weighted deterministic generator, not an ad hoc random branch in the
runner. Each tick samples one action from these weights:
| Flag | Default | Action |
| --- | ---: | --- |
| `--weight-publish` | `35` | Append a durable history message and fan it out to clients. |
| `--weight-create-versioned` | `15` | Create a new mutable Protocol V2 message. |
| `--weight-mutate-versioned` | `20` | Update, append to, or delete an existing mutable message. |
| `--weight-presence` | `20` | Record a presence join/leave transition. |
| `--weight-recovery` | `7` | Probe durable recovery for one simulated client/channel. |
| `--weight-purge` | `2` | Purge an older retained history prefix. |
| `--weight-push-register` | `8` | Register or update a push device. |
| `--weight-push-delete` | `3` | Delete a push device and its subscriptions. |
| `--weight-push-subscribe` | `8` | Subscribe a device to a channel. |
| `--weight-push-unsubscribe` | `5` | Unsubscribe a device from a channel. |
| `--weight-push-publish` | `12` | Accept a push publish and drive durable fanout. |
| `--weight-push-scheduled` | `4` | Store and release a scheduled push publish. |
| `--weight-push-feedback` | `2` | Replay duplicate provider feedback. |
| `--weight-push-repair` | `2` | Run durable push queue repair. |
| `--weight-oracle` | `1` | Force a full oracle sweep outside the periodic cadence. |
`--swarm` replaces these fixed defaults with seed-derived distributions. Use fixed weights when
reproducing a concrete bug and swarm weights when hunting for new combinations.
Set a weight to `0` to disable that action family. At least one action weight must be non-zero.
The final JSON report includes the weights and selected action counts so coverage is visible for a
seed.
The default fault model injects:
| Flag | Default | Meaning |
| --- | ---: | --- |
| `--drop-prob` | `0.08` | Probability that a live fanout message is dropped before reaching a simulated client. |
| `--duplicate-prob` | `0.03` | Probability that a fanout message is duplicated. |
| `--max-delay-ticks` | `12` | Maximum deterministic fanout delay. |
| `--crash-prob` | `0.002` | Per-tick probability that one live node crashes. |
| `--restart-prob` | `0.020` | Per-tick probability that one crashed node restarts. |
| `--pause-prob` | `0.002` | Per-tick probability that one live node pauses without losing durable state. |
| `--resume-prob` | `0.030` | Per-tick probability that one paused node resumes. |
| `--partition-prob` | `0.003` | Per-tick probability that one live node is isolated from traffic. |
| `--heal-prob` | `0.020` | Per-tick probability that one partitioned node heals. |
| `--slow-prob` | `0.003` | Per-tick probability that one node adds deterministic fanout delay. |
| `--stale-prob` | `0.002` | Per-tick probability that one node rejects traffic as stale until it catches up. |
| `--stream-reset-prob` | `0.0005` | Per-tick probability of an operator history/presence stream reset. |
| `--storage-drop-write-prob` | `0.004` | Probability a core durable write is dropped before taking a reservation or changing the store. |
| `--storage-torn-write-prob` | `0.002` | Probability a multi-record versioned-message operation commits history but tears before the version record. |
| `--storage-stale-read-prob` | `0.006` | Probability a recovery read sees an older visible history prefix. |
| `--storage-corrupt-read-prob` | `0.002` | Probability a recovery read is treated as corrupted and ignored fail-closed. |
| `--storage-delayed-commit-prob` | `0.006` | Probability a committed core durable write is hidden from recovery reads until a later deterministic tick. |
| `--storage-max-commit-delay-ticks` | `8` | Maximum delayed commit visibility window for core durable writes. |
| `--queue-produce-lost-prob` | `0.015` | Probability that a push queue produce disappears after durable state was written. |
| `--queue-ack-lost-prob` | `0.010` | Probability that a consumed push queue item is redelivered because ack was lost. |
| `--queue-lease-timeout-prob` | `0.010` | Probability that a queue lease times out and duplicates work. |
| `--write-fail-before-commit-prob` | `0.006` | Probability a push/device/subscription write fails before commit. |
| `--write-fail-after-commit-prob` | `0.004` | Probability a write commits durably but reports failure. |
| `--response-lost-prob` | `0.006` | Probability a committed write loses its response. |
| `--read-stale-prob` | `0.010` | Probability a push worker sees a stale/empty read and fails closed. |
| `--provider-retryable-prob` | `0.100` | Probability a fake provider returns quota/timeout/transient failure. |
| `--provider-reject-prob` | `0.040` | Probability a fake provider permanently rejects a delivery. |
| `--provider-invalid-token-prob` | `0.025` | Probability a fake provider reports an invalid or expired token. |
| `--provider-lost-response-prob` | `0.020` | Probability a fake provider accepted externally but Sockudo lost the response. |
Rejected operations are part of the model: a workload request may route to a crashed or partitioned
node and fail before it reaches durable storage. The shadow only advances after durable Sockudo
stores accept the operation.
Storage faults are injected at simulator-local boundaries. Dropped core writes are rejected before
serial reservations so the model does not create artificial gaps. Torn writes are limited to
multi-record operations where a safe prefix can exist, such as a versioned-message history row
committing before its version row. Delayed commits are tracked as per-channel visible prefixes; a
recovery read may lag, but it must never expose a gap or invalid cursor. Corrupted reads fail closed
and are retried by later recovery probes.
## Push Oracles
Push disasters are checked against Sockudo-side invariants:
| Oracle | Meaning |
| --- | --- |
| Durable acceptance | Every accepted push publish has a durable status row and durable publish-log event. |
| Idempotency | Duplicate publish keys map back to the same logical publish and do not create a second effect. |
| Queue repair | Lost publish-log and delivery queue messages are recreated from durable state by idempotent repair scans. |
| Provider convergence | Each planned eligible target eventually becomes accepted, retryable, rejected, expired, cancelled, or dead-lettered. |
| Status transitions | Modeled publish lifecycle transitions never move backward, never leave a terminal state, and agree with terminal counters. |
| Retry bounds | Retryable outcomes schedule bounded retries and then converge. |
| Token invalidation | Invalid/unregistered/expired tokens are marked terminal and removed from future eligible sends. |
| Counters | Status counters never exceed planned logical deliveries and terminal statuses match terminal counters. |
| Safety after removal | No provider send is allowed after the model-visible state says the device is unsubscribed, deleted, or invalid. |
## CI Profile
A good pull-request smoke profile is:
```bash
cargo test -p sockudo-simulator
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--seed 12648430 \
--ticks 5000 \
--json
```
To replay the checked-in seed corpus:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file crates/sockudo-simulator/corpus/disaster-seeds.json \
--ticks 5000 \
--json
```
To replay the storage-fault corpus:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file crates/sockudo-simulator/corpus/storage-faults.json \
--json
```
For nightly burn-in, run multiple fixed seeds and keep any new failing seed as a regression test:
```bash
for seed in 1 2 3 4 5 12648430 3735928559; do
cargo run -p sockudo-simulator --bin sockudo-sim -- --seed "$seed" --ticks 50000
done
```
For nightly swarm burn-in:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file crates/sockudo-simulator/corpus/swarm-seeds.json \
--ticks 50000 \
--swarm
```
For liveness-focused nightly burn-in:
```bash
cargo run -p sockudo-simulator --bin sockudo-sim -- \
--corpus-file crates/sockudo-simulator/corpus/swarm-seeds.json \
--ticks 50000 \
--mode liveness \
--swarm
```
## Reading Failures
Invariant failures include the seed and simulator tick:
```text
sockudo-sim: FAILED - reproduce with --seed 42
simulator invariant failed at tick 812 (seed=42): history serial gap on sim-channel-1: 41 then 43
replay command:
cargo run -p sockudo-simulator --bin sockudo-sim -- --seed 42 --ticks 5000 --mode safety
config_json:
{"seed":42,"ticks":5000,...}
recent trace:
tick=807 push queue produce lost for PublishLog { publish_id: "push-00000000000000000042" }
tick=811 node 2 restarted
```
Failure artifacts written with `--failure-artifact` include the full config and can be replayed
directly with `--corpus-file`. Shrunk artifacts add a `shrink` block with the original and reduced
tick, operation, and fault counts plus each accepted simplification step.
Treat these as product bugs unless the invariant is deliberately stronger than the subsystem's
documented contract. If the contract changes, update the simulator shadow and this page in the same
change.
## JSON Shape
The final JSON includes top-level core counters, a `protocol_oracles` object with V1/V2 delivery,
cursor, identity, monotonicity, rewind, and presence-edge check counts, storage fault counters,
restart recovery check counts, a nested `push` object with durable status/log counts, status
transition counts, provider results, queue losses, repair counts, phase counts for upgrade runs, a
nested `upgrade` object with rolling restart/schema gate/oracle coverage counters, recent push trace
entries, and top-level `io_trace` entries for recent deterministic operation/fault decisions. Keep sample
outputs with the exact seed and tick count that produced them so they remain replayable. A compact
example generated with `--seed 12648430 --ticks 16 --json` lives at
`crates/sockudo-simulator/examples/disaster-report.example.json`.
## Extending It
Add new workloads by following the existing pattern:
1. Generate the operation through the deterministic scheduler or workload generator.
2. Apply it to the real Sockudo store or subsystem first.
3. Advance the shadow only after the real operation succeeds.
4. Add a cheap per-run oracle and a quiesce oracle.
5. Add or pin a seed that fails before the fix and passes after it.
Advancing the shadow before the real store accepts an operation hides exactly the bugs the
simulator exists to find: a dropped or rejected write would look like a store bug instead of a
legitimate rejection.
## Next steps
- [History and recovery](/docs/server/history-recovery): The durable history and recovery contracts the oracles enforce.
- [Mutable messages](/docs/server/mutable-messages): Serial continuity and version chains checked by the versioned-message oracles.
- [Push operations](/docs/server/push-operations): Status, retries, dead letters, and repair in production.
- [Scaling](/docs/server/scaling): Live multi-node behavior the simulator models but does not replace.
---
# Mutable messages (https://sockudo.io/docs/server/mutable-messages)
> Versioned message actions, aggregation, endpoints, authorization, limits, and error behavior.
Mutable messages let one logical message change after it is published. You can edit it, delete
it, or stream text into it fragment by fragment, and Sockudo keeps every change as an ordered
version. Use them for edited chat messages, moderation, and token-by-token AI responses: history,
rewind, and recovery always return the current state of a message, and you can still audit how it
got there.
This is a Protocol V2 feature. V1 (Pusher-compatible) clients keep receiving plain events and never
see mutations.
## Mental model
A mutable message is a **version chain**. Publishing creates the first version (`create`); every
later `update`, `delete`, or `append` adds a version on top of it. Nothing is overwritten in place:
- **Latest reads** (`GET .../messages/{messageSerial}`, history, rewind) return the *aggregate*,
which is the visible state after every version has been applied.
- **Version reads** (`GET .../versions`) return the individual operations in order.
- **Realtime** subscribers receive each new version as its own `sockudo:message.*` event and reduce
those events into local state.
```mermaid
sequenceDiagram
participant B as Your backend
participant S as Sockudo
participant V as VersionStore
participant C as V2 subscribers
B->>S: POST /events
S->>V: commit create version
S-->>B: message_serial
B->>S: POST .../append
S->>V: compare-and-apply
S->>C: sockudo:message.append
S-->>B: status applied
```
### The four serials
Each version carries four identities. They answer different questions, so do not mix them up:
| Serial | Type | Identifies | Changes per version? |
| --- | --- | --- | --- |
| `message_serial` | string | The logical message. Use it in every mutation and read URL. | No |
| `version_serial` | string | One specific version of that message. Strictly increases along the chain. | Yes |
| `history_serial` | integer | The message's original position in channel history (assigned at create). | No |
| `delivery_serial` | integer | This version's position in the channel's delivery stream. It is the `serial` field on the wire and drives replay and recovery. | Yes |
Server-generated `message_serial` and `version_serial` values have the shape
`<20-digit ms timestamp>::<20-digit counter>`, for example
`00000001730000000000:node-a:00000000000000000007`. The create version's `version_serial` equals the
message's `message_serial`. Serials are opaque: `1..=128` bytes with no whitespace. Treat them as
identifiers, not as something to parse.
An edited message keeps its place in history (`history_serial`), but each edit is a new delivery
(`delivery_serial`) so that reconnecting clients can replay exactly the versions they missed.
Replay requires contiguous delivery serials. If continuity cannot be proven, recovery fails closed
instead of skipping versions.
## Actions
| Action | Realtime event | Meaning |
| --- | --- | --- |
| `create` | `sockudo:message.create` | Create the logical message and reserve its original history identity. |
| `update` | `sockudo:message.update` | Shallow patch `name`, `data`, `extras`, or clear fields. |
| `delete` | `sockudo:message.delete` | Mark/delete visible fields while preserving the version chain. |
| `append` | `sockudo:message.append` | Append a string fragment to existing string data. |
| `summary` | internal `message` event with `data.action: "message.summary"` | Annotation summary projection for a message. Emitted by the annotations subsystem, not by a mutation endpoint; the JavaScript client surfaces it as `message.summary`. |
Wire constants live in `sockudo-protocol/src/versioned_messages.rs`; store semantics live in
`sockudo-core/src/versioned_messages.rs` and `sockudo-core/src/version_store.rs`.
### How each action changes the aggregate
`update` and `delete` use tri-state field semantics. For each of `name`, `data`, and `extras` the
operation either **keeps** the current value (field omitted), **replaces** it (field present), or
**clears** it (field listed in `clear_fields`):
| Operation | `name` | `data` | `extras` | Notes |
| --- | --- | --- | --- | --- |
| `update` | keep / replace / clear | keep / replace / clear | keep / replace / clear | Must include at least one patch field or operation metadata. `extras` is replaced whole, not merged. |
| `delete` | keep / clear | keep / replace / clear | keep / replace / clear | `name` cannot be replaced. An empty body records a delete version and keeps the content. |
| `append` | unchanged | current string + fragment | replaced if sent, otherwise kept | `data` must be a non-empty string. The current data must be a string or absent. |
A field cannot be both replaced and cleared in the same request, and `clear_fields` must not
contain duplicates.
A delete only changes the fields you name. To remove the visible payload, send
`"clear_fields": ["data"]` (and `"extras"` if needed), or replace `data` with a tombstone value.
Earlier versions stay in the version chain until retention removes them.
At commit time Sockudo also records a protocol-neutral message envelope in the
version record and new history payloads. It preserves the client message ID,
publisher identity and timestamp, typed stream/history/delivery positions,
complete encoding chain, user-visible extras, and operation metadata. Older
stored payloads remain readable through the envelope migration fallback.
## Enable mutable messages
1. Enable durable history. Versioned messages build on it: a channel whose resolved history policy
is disabled cannot create or mutate versioned messages.
```toml title="config.toml"
[history]
enabled = true
backend = "postgres"
```
2. Enable versioned messages and choose a version store. `driver` accepts `memory` (the default),
`postgres`, `mysql`, `dynamodb`, `scylladb`, or `surrealdb`.
```toml title="config.toml"
[versioned_messages]
enabled = true
driver = "postgres"
max_page_size = 100
retention_window_seconds = 0
```
The same keys are available as environment variables: `VERSIONED_MESSAGES_ENABLED`,
`VERSIONED_MESSAGES_DRIVER`, `VERSIONED_MESSAGES_MAX_PAGE_SIZE`,
`VERSIONED_MESSAGES_RETENTION_WINDOW_SECONDS`, `VERSIONED_MESSAGES_PURGE_INTERVAL_SECONDS`,
`VERSIONED_MESSAGES_PURGE_BATCH_SIZE`, and `VERSIONED_MESSAGES_MAX_PURGE_PER_TICK`.
3. Connect subscribers with Protocol V2. Only V2 connections receive `update`, `delete`, and `append`
events. Choose how appends arrive with the `append_mode` connection parameter (see
[Realtime delivery](#realtime-delivery)).
4. Publish as usual. While versioned messages are enabled, every non-ephemeral publish to a channel
with durable history becomes a `create` version.
The `memory` driver keeps versions in process memory, so they are lost on restart and are not
shared between nodes. Use a durable driver for anything you deploy.
## Walkthrough
The examples use the signed app HTTP API. Signed requests without `socket_id` act as a privileged
server and may mutate any message (see [Authorization](#authorization)). Server SDKs sign requests
for you; the Node SDK exposes `getMessage`, `getMessageVersions`, `updateMessage`, `deleteMessage`,
and `appendMessage`.
1. **Create the message and capture its serial.** Publish through `/events`. To get the serials
back, include `message_id` (or `info`) in the body; otherwise the response is `{"ok": true}`.
```http
POST /apps/{appId}/events
```
```json
{
"name": "chat-message",
"channel": "private-chat-room-1",
"data": "Sockudo keeps",
"message_id": "reply-7f3a"
}
```
```json
{
"channels": {
"private-chat-room-1": {
"acknowledgement_id": "00000001730000000000:node-a:00000000000000000007",
"message_serial": "00000001730000000000:node-a:00000000000000000007",
"history_serial": 41,
"delivery_serial": 57,
"version_serial": "00000001730000000000:node-a:00000000000000000007"
}
}
}
```
2. **Append a fragment.** The fragment is concatenated to the current string data.
```http
POST /apps/{appId}/channels/private-chat-room-1/messages/{messageSerial}/append
```
```json
{ "data": " every version.", "op_id": "reply-7f3a-append-1" }
```
```json
{
"channel": "private-chat-room-1",
"message_serial": "00000001730000000000:node-a:00000000000000000007",
"action": "append",
"accepted": true,
"version_serial": "00000001730000000150:node-a:00000000000000000008",
"history_serial": 41,
"delivery_serial": 58,
"status": "applied"
}
```
3. **Update it.** Replace `data` and attach operation metadata to the new version.
```json
{
"data": "Sockudo keeps every version of a message.",
"description": "rewrite",
"metadata": { "editor": "support" },
"op_id": "reply-7f3a-edit-1"
}
```
The response has the same shape with `"action": "update"`, a new `version_serial`, and the next
`delivery_serial`. `history_serial` stays `41`.
4. **Delete it.** Clear the payload so the aggregate no longer shows it.
```json
{ "clear_fields": ["data"], "description": "removed by moderator" }
```
5. **Read the result.** Fetch the latest aggregate, or page through the individual versions.
```http
GET /apps/{appId}/channels/private-chat-room-1/messages/{messageSerial}
GET /apps/{appId}/channels/private-chat-room-1/messages/{messageSerial}/versions?direction=oldest_first&limit=2
```
The same flow with the Node server SDK:
```ts title="mutate.ts"
const sockudo = new Sockudo({ appId, key, secret, host: "127.0.0.1", port: 6001 });
const channel = "private-chat-room-1";
await sockudo.appendMessage(channel, messageSerial, { data: " every version." });
await sockudo.updateMessage(channel, messageSerial, {
data: "Sockudo keeps every version of a message.",
description: "rewrite",
op_id: "reply-7f3a-edit-1",
});
await sockudo.deleteMessage(channel, messageSerial, { clear_fields: ["data"] });
const latest = await sockudo.getMessage(channel, messageSerial);
const versions = await sockudo.getMessageVersions(channel, messageSerial, {
direction: "oldest_first",
limit: 20,
});
```
## Realtime delivery
Every applied mutation is fanned out to the channel's V2 subscribers as a `sockudo:message.*` event.
The version identities travel in `extras.headers`, and `serial` is the version's `delivery_serial`.
An abridged update frame:
```json
{
"event": "sockudo:message.update",
"channel": "private-chat-room-1",
"data": "Sockudo keeps every version of a message.",
"name": "chat-message",
"serial": 59,
"extras": {
"headers": {
"sockudo_action": "message.update",
"sockudo_message_serial": "00000001730000000000:node-a:00000000000000000007",
"sockudo_version_serial": "00000001730000000300:node-a:00000000000000000009",
"sockudo_version_timestamp_ms": 1730000000300,
"sockudo_history_serial": 41
}
}
}
```
- **Appends** follow the subscriber's `append_mode` connection parameter. `delta` (the V2 default)
delivers only the new fragment in `data`; `full` delivers the accumulated string on every append.
An unknown value rejects the WebSocket upgrade with `400`.
- **V1 connections** receive the original `create` as a plain event with `extras`, `serial`, and
other V2 fields stripped. They never receive `update`, `delete`, or `append` events.
- **Late joiners** that rewind or read history get the latest visible version of each message
(`latest_by_history` returns one latest version per logical message, sorted by original history
serial), so they see accumulated appended content rather than only the create payload.
- **Webhooks:** each committed version emits a `message_version_created` webhook carrying
`message_serial`, `version_serial`, and `action`. See [Webhooks](/docs/server/webhooks).
## HTTP reference
Signed app HTTP auth can create mutable messages through `/events`. Mutation and read surfaces are:
| Method | Path | Returns |
| --- | --- | --- |
| `POST` | `/apps/{appId}/channels/{channel}/messages/{messageSerial}/update` | Mutation acknowledgement |
| `POST` | `/apps/{appId}/channels/{channel}/messages/{messageSerial}/delete` | Mutation acknowledgement |
| `POST` | `/apps/{appId}/channels/{channel}/messages/{messageSerial}/append` | Mutation acknowledgement |
| `GET` | `/apps/{appId}/channels/{channel}/messages/{messageSerial}` | Latest aggregate |
| `GET` | `/apps/{appId}/channels/{channel}/messages/{messageSerial}/versions` | Page of versions |
### Mutation request body
| Field | Applies to | Description |
| --- | --- | --- |
| `data` | all | Replacement data (update, delete). For append: the string fragment, required and non-empty unless the append carries a terminal AI status (`complete` or `cancelled`). |
| `name` | update | Replacement event name. |
| `extras` | all | Replacement V2 extras. |
| `clear_fields` | update, delete | Any of `"name"` (update only), `"data"`, `"extras"`. No duplicates. |
| `socket_id` | all | Binds the operation to a connected V2 actor. See [Authorization](#authorization). |
| `client_id` | all | Must match the signed-in user of `socket_id` when both are sent. Without `socket_id` it is recorded as attribution only. |
| `description`, `metadata` | all | Operation metadata stored on the new version. |
| `op_id` | all | `1..=64` bytes. Makes retries idempotent. |
### Mutation response
```json
{
"channel": "private-chat-room-1",
"message_serial": "00000001730000000000:node-a:00000000000000000007",
"action": "update",
"accepted": true,
"version_serial": "00000001730000000300:node-a:00000000000000000009",
"history_serial": 41,
"delivery_serial": 59,
"status": "applied"
}
```
`status` is `applied` for a new version and `duplicate` when an `op_id` retry matched an existing
one.
### Read the latest aggregate
```json
{
"channel": "private-chat-room-1",
"item": {
"event": "sockudo:message.update",
"channel": "private-chat-room-1",
"data": "Sockudo keeps every version of a message.",
"name": "chat-message",
"serial": 59,
"action": "update",
"message_serial": "00000001730000000000:node-a:00000000000000000007",
"history_serial": 41,
"delivery_serial": 59,
"version": {
"serial": "00000001730000000300:node-a:00000000000000000009",
"timestamp_ms": 1730000000300,
"description": "rewrite",
"metadata": { "editor": "support" }
}
}
}
```
`item.action` is the action of the newest version; `data` is the accumulated, visible state. The
example is abridged: `extras` is included when the message has extras.
### List versions
Query parameters: `limit` (greater than `0`, capped at `versioned_messages.max_page_size`),
`direction` (`newest_first` by default, or `oldest_first`), and `cursor` (the previous page's
`next_cursor`).
```json
{
"channel": "private-chat-room-1",
"direction": "oldest_first",
"limit": 2,
"has_more": true,
"next_cursor": "00000001730000000150:node-a:00000000000000000008",
"items": [
{ "event": "sockudo:message.create", "action": "create", "data": "Sockudo keeps", "...": "..." },
{ "event": "sockudo:message.append", "action": "append", "data": " every version.", "...": "..." }
]
}
```
In a version page, an `append` item's `data` is the fragment that operation added, so the page can
be replayed operation by operation. The latest-aggregate read returns the concatenated string.
### Ably compatibility facade
The optional Ably compatibility facade projects the pinned SDK shape over the
same service:
```http
PATCH /channels/{channel}/messages/{messageSerial}
Content-Type: application/json
```
```json
{
"serial": "message-serial",
"action": "message.append",
"data": " fragment",
"version": {
"serial": "optional-version-serial",
"clientId": "verified-or-server-attributed-client",
"timestamp": 1710000000000,
"description": "stream fragment",
"metadata": { "source": "agent" }
}
}
```
`message.update` and `message.delete` use the same envelope. JSON and MsgPack
requests, successes, and errors all use the negotiated compatibility encoder.
The path and body serial must identify the same logical message. The facade is a projection over
the same mutation service and `VersionStore`; it does not keep a separate message chain.
## Authorization
Every mutation is evaluated against an **actor**. Which actor you get depends on whether the
request carries `socket_id`:
| Request | Actor | May mutate |
| --- | --- | --- |
| Signed app HTTP, no `socket_id` | Privileged server | Any message on the app. |
| Signed app HTTP with `socket_id` | The V2 connection with that socket ID, on the node that receives the request | Only what that connection's capabilities allow. |
When `socket_id` is sent, Sockudo checks that the socket is connected to the receiving node and
uses Protocol V2. If `client_id` is also sent, it must equal the connection's signed-in user.
Connections authenticated with a capability token also need `publish` on the channel.
In `server_role = "api"` mode, `socket_id` cannot be verified (no local WebSocket connections)
and the request is rejected with HTTP 400. Omit `socket_id` or send the request to a WebSocket node.
See [Server role](/docs/reference/configuration#server-role).
### Capabilities
There are six mutation capabilities: `message_update_own`, `message_update_any`,
`message_delete_own`, `message_delete_any`, `message_append_own`, and `message_append_any`.
- `*_any` grants are checked first and allow mutating any message on matching channels.
- `*_own` grants require an identified actor, an identified original creator, and a constant-time
match between the actor's `client_id` and the original creator's `client_id`.
User sign-in data maps each capability to channel patterns:
```json title="user_data.capabilities"
{
"message_append_own": ["private-ai:user-42:*"],
"message_update_own": ["private-ai:user-42:*"],
"message_delete_own": ["private-ai:user-42:*"],
"history": ["private-ai:user-42:*"]
}
```
Capability tokens use the inverse shape (channel pattern to operations) and accept both the
underscore and hyphenated spellings, for example `message-update-own`. See
[AI Transport authentication](/docs/server/ai-transport-authentication).
For token-authenticated compatibility clients, an operation-supplied `clientId`
is metadata only after the authenticated actor and capability check succeeds;
it is never identity proof.
The original creator is the signed-in user of the connection that published the message: a client
publish, or an HTTP publish that passed that connection's `socket_id`. A message published by your
backend without `socket_id` has no identified creator, so `*_own` grants can never match it. Only
`*_any` grants or privileged server requests can mutate such messages.
## Idempotency and concurrency
`op_id` is scoped by app, channel, message, and action. Replaying the same ID
with the same canonical operation returns the original acknowledgement with `status: "duplicate"`
and does not fan out again. Reusing it with a different operation is an idempotency conflict
(`400`). Memory, PostgreSQL, MySQL, DynamoDB, ScyllaDB, and SurrealDB implement the same
compare-and-apply contract.
Every create or mutation commits through one `VersionStore` transaction. The
transaction validates the exact predecessor, reserves the next delivery
position, writes the version, and stores any idempotency receipt together. A
failed transaction exposes neither a version nor a delivery gap. Concurrent
appends retry from the newest aggregate, so each accepted fragment appears once
and in commit order. Accumulated-byte, append-count, open-stream, and terminal
state checks are part of that transaction.
Streaming workers retry on timeouts. With a stable `op_id` per fragment, a retried append that
already committed returns `duplicate` instead of appending the text twice.
## Limits
| Limit | Default | Applies to |
| --- | --- | --- |
| `versioned_messages.max_page_size` | `100` | Version-history pages |
| `versioned_messages.retention_window_seconds` | `0`, no expiry | Version entries |
| `versioned_messages.purge_interval_seconds` | `300` (minimum 10) | Purge worker, backends without native TTL |
| `versioned_messages.purge_batch_size` | `1000` | Rows per purge query |
| `versioned_messages.max_purge_per_tick` | `100000` | Rows per purge run |
| `ai_transport.max_accumulated_message_bytes` | `1048576` | AI Transport channels |
| `ai_transport.max_appends_per_message` | `4096` | AI Transport channels |
| `ai_transport.max_open_streaming_messages_per_channel` | `1024` | AI Transport channels |
| `op_id` length | `1..=64` bytes | All mutations |
| serial length | `1..=128` bytes, no whitespace | All serials |
ScyllaDB and DynamoDB apply `retention_window_seconds` as native row TTL. PostgreSQL, MySQL,
SurrealDB, and memory use the background purge worker.
The `ai_transport.*` limits apply only to channels that match an `[[ai_transport.channels]]` entry
while `[ai_transport]` is enabled. On those channels an append to a message that already reached a
terminal state is rejected.
## Errors
Mutation errors use the same HTTP envelope as other app APIs. AI Transport-coded errors add
`ai_code`:
```json
{
"error": "mutations not permitted on this channel",
"code": "mutable_not_permitted",
"ai_code": 93002,
"status": 403
}
```
| Status | `code` | When |
| --- | --- | --- |
| `400` | `invalid_input` | Malformed body, empty append, conflicting or duplicate `clear_fields`, invalid `op_id` or serial, `op_id` reused with a different operation, append to non-string data or to a terminal message, `socket_id` in the `api` server role. |
| `401` | `auth_failed` | `socket_id` is not connected to the receiving node or is not V2, or `client_id` does not match the connection's signed-in user. |
| `403` | `mutable_not_permitted` (`ai_code` `93002`) | The actor lacks the required capability, a capability token lacks `publish`, or versioned messages are disabled on an AI Transport channel. |
| `404` | `not_found` | Unknown message serial in that channel. |
| `413` | `payload_too_large` (`ai_code` `40009`) | An AI Transport accumulated-bytes, append-count, or open-stream limit was exceeded. |
| `422` | `feature_disabled` | Versioned messages are disabled, or durable history is disabled for the channel. |
Feature-disabled, not-found, malformed-input, payload-too-large, and authorization failures are
kept distinct so SDKs can preserve behavior across languages.
## Storage internals
This section is for operators running a durable version store.
The memory version store maintains
latest-version, version-identity, idempotency-receipt and retained append-count
indexes under the same lock as the version chain and replay log. Latest selection
uses the greatest version serial, including for out-of-order imported records.
Retention removes corresponding index entries; it does not reset the stream ID
or its delivery allocator. Full-chain validation remains available for import
and repair tooling.
### Append-count table (PostgreSQL and MySQL)
PostgreSQL and MySQL maintain the same retained append count in a small companion
table (`_ac`). Insert/delete triggers update it within the
entry write transaction, including imports, duplicate insert handling, rollback,
and retention deletes. Updates and deletes do not query the count; capped appends
read one indexed counter. Delivery allocation and mutation transaction boundaries
are unchanged.
The first startup after upgrading installs the counter table and triggers and
backfills existing entries while temporarily locking version-entry writes. Plan
this initialization for a maintenance window on large stores. The database role
needs permission to create tables and triggers, PostgreSQL trigger functions, and
MySQL table locks. Concurrent installers serialize; a completion marker
(`_c1`) is written only after backfill succeeds. Interrupted
installation retries on startup. Subsequent startups skip the backfill.
Include the companion table, marker and triggers in consistent database backups.
Older server versions can continue inserting and deleting entries because the
database triggers maintain the count. For manual repair that changes entry actions
or truncates entries, stop writers, validate the repaired version chains, remove
the completion marker row, and restart to rebuild counts before serving mutations.
Do not independently edit the counter table during normal operation. Rebuild it with the
marker-removal procedure above.
### Append storage
Each original append remains its own version entry, including its fragment,
metadata, extras, envelope facts, all four serial identities and any `op_id`
receipt. Public reads (latest, version pages in either direction, replay, recovery, rewind,
history projection and duplicate receipts) return the same results whichever storage format is
active. The storage representation does not change the wire API.
Durable stores read self-contained legacy rows, format-1 run snapshots and
format-2 chunked runs. Format 2 stores the aggregate in bounded byte chunks:
1 KiB on PostgreSQL/MySQL and 4 KiB on DynamoDB/ScyllaDB/SurrealDB. Each append
rewrites only its tail chunk and adds chunks for the new fragment, together
with the original operation and fixed run metadata. Latest-state payloads on
DynamoDB, ScyllaDB and SurrealDB are compact too. Memory retains its in-place
append-run string.
Chunks can split a multibyte character. Readers join the required prefix and
check its length, UTF-8 boundary and terminating operation fragment before
returning a record. Missing or malformed storage fails closed. Every chunked
run has a unique generation. The bounded 8 MiB per-store cache on DynamoDB,
ScyllaDB and SurrealDB uses that generation plus tenant/channel/message identity;
a cached prefix never supplies a longer version written by another node.
PostgreSQL/MySQL fetch page rows and their required chunks in one statement.
DynamoDB follows service pagination for version history, replay and latest-state
enumeration, including pages containing only expired records. Large append
fragments that exceed one transaction's chunk capacity use fenced staging:
bounded batches write an unpublished suffix, then a final transaction publishes
the operation, receipt and new head together. Concurrent writers, marker changes
and abandoned staging cannot expose a partial append. The original operation
and receipt must still fit DynamoDB's individual item size limit.
ScyllaDB reuses prepared statements through a bounded 64-entry cache. SurrealDB
initialization defines immutable, versioned mutation functions alongside its
tables; database credentials must permit defining those functions. Mutations
still run inside one explicit transaction, including the format fence, serial
checks, original entry, chunks and receipt. Older releases ignore the functions,
and rollback materialization still restores the legacy full-row representation.
SurrealDB also enforces a unique delivery position per app/channel. Initialization
rejects existing duplicate positions instead of silently accepting a forked
history. Inspect and repair such data before restarting writers. Concurrent
mutation conflicts retry the complete operation with fresh state, while retaining
the original precondition and operation receipt identity.
The SurrealDB version store requires a stable server release **3.3.0 or later**.
Earlier in-memory engines can lose concurrent updates even inside guarded
transactions, and older servers also have a FlatBuffers limit affecting large
legacy byte-array records. Startup rejects older versions because the remote
connection cannot reliably identify which storage engine the server uses.
Upgrade the database before starting writers with this release. See the
[upstream concurrency defect](https://github.com/surrealdb/surrealdb/issues/7473)
and [3.3 release notes](https://surrealdb.com/releases/3.3).
#### Rolling upgrade and activation
The persisted store marker starts **off**. New nodes continue writing full rows
while old nodes remain in service. There is no format setting in the server
configuration. If a store already contains C2 format-1 rows, releases older than
C2 cannot read those rows; materialize them before introducing such readers.
1. Roll out a release that supports format 2 to every writer and reader. Remove older C2 writers;
they ignore the marker.
2. Stop writers.
3. Activate compact append storage:
```bash
sockudo --config --enable-append-storage
```
4. Restart writers.
Activation seeds chunks for existing latest messages before enabling compact
writes. This moves conversion of large existing aggregates into maintenance,
so their first append can extend a seed. New creates, updates and imports keep
full records and prepare a seed while chunked storage is enabled. Include the
run/chunk tables and marker in consistent backups. Startup needs permission to
create the companion tables and add nullable pointer columns.
Marker changes use transactional fences on PostgreSQL, MySQL, DynamoDB and
SurrealDB. ScyllaDB's channel-partition conditional batches cannot fence a
separate store-wide row; the stopped-writer maintenance requirement is essential
there.
#### Rollback and retention
1. Stop every writer.
2. Check the backend's legacy representation constraints:
```bash
sockudo --config --check-append-storage-rollback
```
3. Disable compact writes and materialize in one command:
```bash
sockudo --config --rollback-append-storage
```
4. Start the older release only after the command completes.
The check does not change the format marker or rewrite version records. On
DynamoDB it scans compact entries, receipts and latest states and checks their
materialized item sizes, including attribute names. A successful check is valid
only while writers remain stopped; it is not a cluster-wide admission fence.
Numeric attributes use a conservative size bound, so an item extremely close
to the service limit may be rejected even if DynamoDB could encode it smaller.
The rollback command performs the size preflight before changing the marker. If a later
database operation fails, keep writers stopped and rerun the command; earlier
rewrites are safe to repeat. The separate `--disable-append-storage` and
`--materialize-append-storage` commands remain available. DynamoDB materialization also performs a
preflight before its first rewrite.
Materialization supports both compact formats, including latest states and
receipts, and is idempotent. It restores full per-version storage and its write
cost. Disabling the marker alone does not make existing compact rows readable by
older releases; those releases reject unknown formats rather than return
partial data.
DynamoDB's 400 KiB item limit cannot be raised by this command.
If any required legacy row exceeds it, the preflight rejects rollback before
changing the marker or rewriting entries. Continue using a release that supports
chunked storage; do not truncate messages or delete receipts to force rollback.
The limit applies to the encoded full item, which may contain multiple copies
of message data, not just to the aggregate string length.
Chunks are removed only when no retained entry or receipt needs them.
PostgreSQL/MySQL remove chunks with unreferenced runs; SurrealDB also protects
latest-state references and receipt-pinned runs. DynamoDB chunks do not receive
an independent TTL that could expire before later entries: reference-checked
cleanup reclaims inactive runs, and receipt references pin them. ScyllaDB keeps
chunks with its non-expiring commit rows. Unpublished staged DynamoDB seeds
become eligible for bounded cleanup after five minutes. A persisted deletion
fence prevents late staging or publication from racing cleanup. Never delete snapshots or
chunks independently of their entries.
## Next steps
- [History and recovery](/docs/server/history-recovery): How rewind, replay, and durable history serve the latest version of each message.
- [Webhooks](/docs/server/webhooks): React to every committed version with message_version_created.
- [HTTP API](/docs/server/http-api): Signing, publish acknowledgements, and the full endpoint reference.
- [Token streaming rollup](/docs/server/token-streaming-rollup): Coalesce high-rate appends for AI token streams.
- [AI Transport overview](/docs/server/ai-transport-overview): Streamed AI responses built on mutable messages.
---
# Webhooks (https://sockudo.io/docs/server/webhooks)
> Deliver channel lifecycle, client event, presence, versioned-message, annotation, and AI Transport events to your backend.
Webhooks let Sockudo tell your backend when something happens on the realtime side: a channel
gains its first subscriber, a presence member leaves, a client sends an event, a message gets a new
version, or an AI run ends. Use them to keep your own database, analytics, or workflows in sync
with realtime state without polling the HTTP API. Delivery is asynchronous and can repeat, so every
handler must be idempotent.
## How delivery works
Webhooks are configured per app and delivered by a background worker that consumes the server's
`[queue]`. When an event fires, Sockudo checks whether any of the app's webhooks subscribes to that
event type. If none does, nothing is enqueued. Otherwise the event becomes a queue job, and a worker
filters it per endpoint, signs the body with the app secret, and POSTs it.
```mermaid
flowchart TB
A[Realtime event fires] --> B{App subscribes type?}
B -- no --> X[Dropped]
B -- yes --> C[Optional batch buffer]
C --> D[Webhook queue]
D --> E[Worker filters endpoint]
E --> F[Sign with secret]
F --> G[POST to url]
G -- retryable failure --> F
```
Three things follow from this design:
- **Webhooks need a queue.** With `[queue] driver = "none"` (or a queue that fails to initialize)
webhooks are disabled. The `memory` driver works for single-node development but loses pending
jobs when the process exits; use a broker-backed driver such as Redis in production.
- **Delivery is at least once, not exactly once.** A timeout, retry, or worker restart can deliver
the same event again.
- **Order is not guaranteed.** Each node sends up to 20 deliveries concurrently and retries run in
the background. Use `time_ms` and the identifiers inside each event, not arrival order.
## Configure an endpoint
Webhook endpoints belong to an app. In the TOML array app manager, add one
`[[app_manager.array.apps.policy.webhooks]]` table per endpoint:
```toml title="config.toml"
[[app_manager.array.apps]]
id = "app-id"
key = "app-key"
secret = "app-secret"
enabled = true
[[app_manager.array.apps.policy.webhooks]]
url = "https://api.example.com/sockudo/webhooks"
event_types = ["channel_occupied", "channel_vacated", "member_added", "member_removed"]
request_timeout_ms = 5000
[app_manager.array.apps.policy.webhooks.filter]
channel_prefix = "presence-"
[app_manager.array.apps.policy.webhooks.headers]
X-Webhook-Source = "sockudo"
```
Database and JSON app managers use the same fields in a top-level `webhooks` array on the app
record:
```json
{
"id": "app-id",
"key": "app-key",
"secret": "app-secret",
"webhooks": [
{
"url": "https://api.example.com/sockudo/webhooks",
"event_types": ["channel_vacated"],
"headers": { "X-Webhook-Source": "sockudo" },
"filter": { "channel_prefix": "private-" }
}
]
}
```
| Field | Description |
| --- | --- |
| `url` | HTTPS endpoint that receives a signed `POST`. |
| `lambda` | `{ function_name, region }` to invoke an AWS Lambda function instead of HTTP. Requires the `lambda` Cargo feature (included in `full`). |
| `lambda_function` | Legacy Lambda function name. Uses region `us-east-1`; prefer `lambda`. |
| `event_types` | Event names this endpoint receives. Required; an event not listed here is never sent to this endpoint. |
| `filter` | Optional channel filter. See [Filter by channel](#filter-by-channel). |
| `headers` | Extra HTTP headers added to every request to this endpoint. |
| `retry` | Per-endpoint override of any `[webhooks.retry]` field. Unset fields fall back to the server value. |
| `request_timeout_ms` | Per-endpoint override of `[webhooks] request_timeout_ms`. |
Lambda endpoints are invoked asynchronously (`Event` invocation type) with the same JSON body as
HTTP deliveries. They do not carry `X-Pusher-Signature`, and Sockudo does not apply
`[webhooks.retry]` to them; AWS handles asynchronous invocation retries.
### Filter by channel
A filter narrows an endpoint to a subset of channels. Every field you set must match; fields you
omit always pass.
| Filter field | Matches when |
| --- | --- |
| `channel_prefix` | The channel name starts with this string. |
| `channel_suffix` | The channel name ends with this string. |
| `channel_pattern` | The channel name matches this regular expression. |
| `channel_namespace` | The channel's namespace equals this value. |
| `channel_namespaces` | The channel's namespace is one of these values. |
The namespace is the part before the first `:` after Sockudo strips the channel type prefix
(`private-`, `presence-`, `private-encrypted-`, and the `cache-` variants). For example,
`presence-chat:room-1` has the namespace `chat`.
If `channel_pattern` is not a valid regular expression, the endpoint matches no channels and
receives nothing. Sockudo logs `invalid webhook channel_pattern regex` at warn level.
## Delivery settings
Server-wide delivery behavior lives under `[webhooks]`. These settings apply to every app; the
`retry` and `request_timeout_ms` values can be overridden per endpoint.
```toml title="config.toml"
[webhooks]
request_timeout_ms = 10000
[webhooks.batching]
enabled = false
duration = 50
size = 100
[webhooks.retry]
enabled = true
max_attempts = 5
max_elapsed_time_ms = 300000
initial_backoff_ms = 1000
max_backoff_ms = 60000
```
| Key | Built-in default | Description |
| --- | --- | --- |
| `request_timeout_ms` | `10000` | Timeout for a single HTTP attempt. |
| `batching.enabled` | `true` | Buffer events in memory and enqueue them as multi-event jobs. The shipped `config/config.toml` sets `false`. |
| `batching.duration` | `50` | Flush interval for the batch buffer, in milliseconds. |
| `batching.size` | `100` | Maximum events per delivered request when batching. |
| `retry.enabled` | `true` | Retry retryable failures. |
| `retry.max_attempts` | unset | Total attempts, including the first. Unset means only `max_elapsed_time_ms` bounds retries. The shipped config sets `5`. |
| `retry.max_elapsed_time_ms` | `300000` | Retry budget measured from the first attempt. |
| `retry.initial_backoff_ms` | `1000` | Delay before the first retry. |
| `retry.max_backoff_ms` | `60000` | Upper bound for the doubling backoff. |
Batching can also be set with `WEBHOOK_BATCHING_ENABLED`, `WEBHOOK_BATCHING_DURATION`, and
`WEBHOOK_BATCHING_SIZE`. See [environment variables](/docs/reference/environment-variables#webhooks).
### Batching
With batching off, every event becomes its own queue job and its own HTTP request. With batching
on, events accumulate in an in-process buffer that flushes every `batching.duration` milliseconds.
Consecutive events for the same app are merged into jobs of at most `batching.size` events, so one
request can carry many events. The envelope's `time_ms` is then the earliest event time in the batch.
AI Transport lifecycle events (`ai_run_started`, `ai_run_ended`, `ai_cancel_requested`,
`ai_stream_orphaned`, and the legacy `ai_turn_*` events) always bypass the batch buffer and go
straight to the queue, so queue backpressure applies to them directly.
Write handlers that loop over `events`, even with batching off. The envelope shape is the same in
both modes, so turning batching on later needs no receiver change.
## The request
Each HTTP delivery is a `POST` with a JSON body:
```http
POST /sockudo/webhooks HTTP/1.1
Content-Type: application/json
X-Pusher-Key: app-key
X-Pusher-Signature: 3b1f6c0a5e...e9d2
{"time_ms":1767225600000,"events":[{"name":"channel_occupied","channel":"presence-chat:room-1"}]}
```
| Part | Meaning |
| --- | --- |
| `X-Pusher-Key` | The app key. Use it to pick the secret when one receiver serves several apps. |
| `X-Pusher-Signature` | Lowercase hex HMAC-SHA256 of the raw request body, keyed with the app secret. |
| `time_ms` | Unix time in milliseconds when the event (or earliest batched event) was created. |
| `events` | Array of event objects. Each has a `name` and, for every current event type, a `channel`. |
Custom `headers` from the endpoint config are added to every request. Builds with OpenTelemetry
also add trace-context propagation headers.
The signature covers the exact body sent to that endpoint. Because endpoints receive only the
events they subscribe to, two endpoints can receive different bodies and different signatures for
the same underlying events.
## Verify signatures
1. Capture the raw request body before any JSON parsing. Re-serializing parsed JSON changes the
bytes and breaks the signature.
2. Look up the secret for the `X-Pusher-Key` header.
3. Compute HMAC-SHA256 of the raw body with that secret and compare it to `X-Pusher-Signature`
in constant time.
4. Reject the request with `401` if it does not match. Only then parse and process `events`.
With the Node server SDK:
```ts title="webhooks.ts" {4-7}
app.post("/sockudo/webhooks", express.raw({ type: "application/json" }), (req, res) => {
const webhook = sockudo.webhook({
rawBody: req.body.toString("utf8"),
headers: req.headers,
});
if (!webhook.isValid()) {
return res.status(401).send("invalid signature");
}
for (const event of webhook.getEvents()) {
enqueueWebhookEvent(event);
}
res.sendStatus(200);
});
```
The SDK also checks that `Content-Type` is exactly `application/json`. During a secret rotation,
pass the previous credentials so both are accepted:
```ts
webhook.isValid([{ key: "app-key", secret: process.env.SOCKUDO_PREVIOUS_SECRET! }]);
```
Without an SDK, verification is a few lines:
```ts title="verify.ts"
export function verifySockudoWebhook(rawBody: string, signature: string, secret: string): boolean {
const expected = createHmac("sha256", secret).update(rawBody).digest("hex");
const a = Buffer.from(expected);
const b = Buffer.from(signature ?? "");
return a.length === b.length && timingSafeEqual(a, b);
}
```
## Event reference
Subscribe to an event by adding its name to `event_types`. Every example below shows one element
of the `events` array.
| Event | Fires when |
| --- | --- |
| `channel_occupied` | A subscribe brings a channel's subscriber count to 1. |
| `channel_vacated` | A channel's subscriber count reaches 0 and is still 0 three seconds later. |
| `subscription_count` | The subscriber count of a non-presence channel changes. |
| `member_added` | A user's first connection joins a presence channel. |
| `member_removed` | A user's last connection leaves a presence channel. |
| `member_updated` | A Protocol V2 client updates its presence data. |
| `client_event` | A client sends a client event (or an AI event frame). |
| `cache_miss` | A client subscribes to a cache channel that has no cached event. |
| `message_version_created` | A version of a versioned message is committed. |
| `annotation_created` | An annotation is published. |
| `annotation_deleted` | An annotation is deleted. |
| `ai_run_started` | An AI Transport run starts. |
| `ai_run_ended` | An AI Transport run ends. |
| `ai_cancel_requested` | A cancel signal is published for an AI run. |
| `ai_stream_orphaned` | The orphan janitor closes a stale streaming message. |
| `ai_turn_started`, `ai_turn_ended` | Legacy aliases of `ai_run_started` and `ai_run_ended`. |
### Channel lifecycle
`channel_occupied` and `channel_vacated` describe whether a channel has any subscribers across the
cluster. `channel_vacated` is delayed by three seconds and re-checked, so a quick reconnect does not
produce a vacated/occupied pair.
```json
{ "name": "channel_occupied", "channel": "private-orders" }
```
```json
{ "name": "channel_vacated", "channel": "private-orders" }
```
`subscription_count` fires on every subscribe and unsubscribe for non-presence channels. Presence
channels use the member events instead.
```json
{ "name": "subscription_count", "channel": "private-orders", "subscription_count": 42 }
```
### Presence
Presence lifecycle events are:
- `member_added`
- `member_updated`
- `member_removed`
Presence lifecycle events use the `user_id` authorized for that channel in
`channel_data`. One socket can have different member IDs in different presence
channels, independently of its sign-in identity. On unsubscribe or disconnect,
`member_removed` identifies the member that joined that channel, and is emitted
only after that member's last connection leaves. The client presence event uses
the same channel-specific identity.
```json
{ "name": "member_added", "channel": "presence-chat:room-1", "user_id": "user-123" }
```
```json
{ "name": "member_removed", "channel": "presence-chat:room-1", "user_id": "user-123" }
```
`member_removed` is sent three seconds after the last connection leaves, and only if the user has
not reconnected to that channel in the meantime.
`member_updated` is emitted for Protocol V2 `sockudo:presence_update` frames. Its payload includes
`channel`, `user_id`, and `user_info` with the latest member data.
```json
{
"name": "member_updated",
"channel": "presence-chat:room-1",
"user_id": "user-123",
"user_info": { "name": "Ada", "status": "typing" }
}
```
### Client events
`client_event` carries the event name, the data the client sent, and the sender's socket ID.
`data` is passed through as sent: a string for Pusher-style clients, or a JSON value when the client
sent structured data.
```json
{
"name": "client_event",
"channel": "private-chat:room-1",
"event": "client-typing",
"data": "{\"typing\":true}",
"socket_id": "123.456"
}
```
AI event frames published by clients also produce a `client_event` with `event` set to `ai-event`
and `data` set to `null`.
Pusher includes `user_id` on `client_event` webhooks from presence channels. Sockudo currently
omits it. If you need the sender's identity, resolve it from `socket_id` in your own records.
Messages marked with `extras.ephemeral` never produce `client_event` webhooks.
### Cache channels
`cache_miss` fires when a client subscribes to a `cache-`, `private-cache-`,
`private-encrypted-cache-`, or `presence-cache-` channel that has no cached event. Use it to
publish an initial value.
```json
{ "name": "cache_miss", "channel": "cache-prices", "data": "{}" }
```
### Versioned messages
`message_version_created` fires once for every committed version: the original create and each
update, delete, or append. `action` is `message.create`, `message.update`, `message.delete`, or
`message.append`. See [mutable messages](/docs/server/mutable-messages) for how the serials relate.
```json
{
"name": "message_version_created",
"channel": "chat:room-1",
"message_serial": "00000000000000000001:node:00000000000000000001",
"version_serial": "00000000000000000001:node:00000000000000000002",
"action": "message.update"
}
```
The pair `message_serial` + `version_serial` identifies one version and makes a natural
deduplication key.
### Annotations
Annotation webhooks include `channel`, `message_serial`, `annotation_serial`, and
`annotation_type`; delete events also include `deleted_annotation_serial`.
```json
{
"name": "annotation_created",
"channel": "chat:room-1",
"message_serial": "00000000000000000001:node:00000000000000000001",
"annotation_serial": "00000000000000000001:node:00000000000000000007",
"annotation_type": "reaction:distinct.v1"
}
```
```json
{
"name": "annotation_deleted",
"channel": "chat:room-1",
"message_serial": "00000000000000000001:node:00000000000000000001",
"annotation_serial": "00000000000000000001:node:00000000000000000009",
"deleted_annotation_serial": "00000000000000000001:node:00000000000000000007",
"annotation_type": "reaction:distinct.v1"
}
```
### AI Transport
AI Transport event types are:
- `ai_run_started`
- `ai_run_ended`
- `ai_cancel_requested`
- `ai_stream_orphaned`
Legacy `ai_turn_started` and `ai_turn_ended` event types are still accepted for existing webhook
consumers, but new integrations should subscribe to `ai_run_started` and `ai_run_ended`. The legacy
events carry `turn_id` instead of `run_id`.
```json
{
"name": "ai_run_started",
"channel": "private-ai-chat",
"run_id": "run-42",
"client_id": "user-123"
}
```
`ai_run_ended` includes `reason` and optional `error_code`. `reason` is one of `complete`,
`cancelled`, `error`, or `unknown`. A suspended run does not emit `ai_run_ended`.
```json
{
"name": "ai_run_ended",
"channel": "private-ai-chat",
"run_id": "run-42",
"reason": "error",
"error_code": "provider_timeout"
}
```
`ai_cancel_requested` includes both `run_id` and the legacy `turn_id` alias:
```json
{
"name": "ai_cancel_requested",
"channel": "private-ai-chat",
"run_id": "run-42",
"turn_id": "run-42",
"client_id": "user-123"
}
```
`ai_stream_orphaned` is emitted when the distributed orphan janitor closes a stale streaming
message after `ai_transport.rollup.orphan_ttl_ms`:
```json
{
"name": "ai_stream_orphaned",
"channel": "private-ai-chat",
"message_serial": "00000000000000000001:node:00000000000000000001",
"reason": "orphan_timeout"
}
```
`run_id`, `client_id`, and `error_code` are `null` when the run did not carry them. To persist
completed runs from these events, see [persist completed AI runs](/docs/server/observability#persist-completed-ai-runs).
## Retries and response codes
Your response status tells Sockudo whether to try again:
| Your response | Sockudo does |
| --- | --- |
| Any `2xx` | Marks the delivery done. |
| `408`, `409`, `425`, `429`, or any `5xx` | Retries with backoff. |
| Timeout, connection error, or other transport failure | Retries with backoff. |
| Any other status (for example `400`, `401`, `404`) | Gives up immediately. No retry. |
Retries start after `initial_backoff_ms` and double each time, capped at `max_backoff_ms`. They stop
when `max_attempts` is reached, or when waiting for the next retry would exceed
`max_elapsed_time_ms` measured from the first attempt.
If in-process retries are exhausted, the queue job itself is retried by the queue backend and
dead-lettered after its attempt limit. On SQS that limit is the queue's `maxReceiveCount`, or
`queue.reliability.max_attempts` when no redrive policy exists. Delivery is at-least-once: when one
endpoint fails, a retried job re-POSTs the same batch to endpoints that already succeeded, so make
receivers idempotent.
Return `2xx` for events that are valid but not useful to you, and record them as ignored. Return
`503` or `429` when you are temporarily unable to process. A `400` or `401` stops delivery of that
request permanently.
## Handle events idempotently
Events carry no delivery ID, so derive a deduplication key from the event itself:
| Event | Suggested key |
| --- | --- |
| `message_version_created` | `message_serial` + `version_serial` |
| `annotation_created`, `annotation_deleted` | `annotation_serial` |
| `ai_run_started`, `ai_run_ended`, `ai_cancel_requested` | `name` + `run_id` |
| `ai_stream_orphaned` | `message_serial` |
| Channel and presence events | Treat as state, not as a log: apply them so that replaying is harmless. |
For channel and presence events, store the latest known state (occupied, member present) and
compare `time_ms` before overwriting, instead of counting events.
```ts title="handler.ts"
async function enqueueWebhookEvent(event: SockudoWebhookEvent) {
switch (event.name) {
case "message_version_created":
await jobs.enqueueOnce(`${event.message_serial}:${event.version_serial}`, event);
break;
case "member_added":
await presence.markPresent(event.channel, event.user_id);
break;
case "member_removed":
await presence.markAbsent(event.channel, event.user_id);
break;
default:
await audit.record(event);
}
}
```
Handlers should:
- verify the signature before any business logic
- return quickly and enqueue expensive work
- tolerate duplicates and out-of-order arrival
- preserve raw payloads for audit when compliance requires it
## Shutdown delivery
Graceful shutdown closes client connections promptly, but keeps webhook dependencies available
while their cleanup finishes. After the listener grace period, Sockudo waits for cleanup workers,
pending presence leases, and delayed `member_removed` and `channel_vacated` producers. It then
joins any in-flight batch enqueue and flushes the final buffer, even if the configured batch
interval has not elapsed. A V2 presence lease remains 15 seconds by default; the subsequent
reconnect check still delays the webhook by 3 seconds.
This drain has a budget of `queue.reliability.shutdown_timeout_ms`
(`QUEUE_SHUTDOWN_TIMEOUT_MS`, default 30000), separate from `shutdown_grace_period`. The memory
queue must finish its queued work before shutdown because it cannot retain jobs after exit.
If the final batch is only partly accepted, shutdown still drains accepted memory-queue jobs
within that budget before returning the enqueue error.
Broker-backed queues only need the final jobs enqueued before their normal worker shutdown;
pending durable jobs can be processed later. With the memory driver, a webhook endpoint that
fails delivery leaves the job in the delayed state for its retry backoff, so graceful
shutdown waits for those retries within the drain budget.
Keep the webhook receiver and queue/cache dependencies available during a rollout. Enqueue
failures are logged, and an exhausted drain budget warns that pending lifecycle events may be
lost. A hard kill or unavailable receiver can still prevent delivery. Allow your supervisor
enough time for both this drain and the queue worker shutdown budget, in addition to listener
grace and teardown. See [rolling deploys](/docs/server/scaling#rolling-deploys).
## Push notifications
Push delivery does not emit webhook events. Track push outcomes through the push publish status
API, metrics, and provider feedback described in [push operations](/docs/server/push-operations).
Sockudo retries a failed delivery in-process for `webhooks.retry.max_elapsed_time_ms`
(default 5 minutes). If it still fails, the queue job is retried by the queue
backend and dead-lettered after its attempt limit. On SQS that limit is the
queue's `maxReceiveCount`, or `queue.reliability.max_attempts` when no redrive
policy exists. Delivery is at-least-once: when one endpoint fails, a retried job
re-POSTs the same batch to endpoints that already succeeded. Make receivers
idempotent.
## Security
- Accept webhooks only over HTTPS.
- Validate the signature against the raw body before parsing business logic.
- Rotate app secrets with an overlap window, accepting old and new credentials during the switch.
- Use endpoint `headers` for an additional shared token if your gateway requires one; never put
secrets in the URL.
- Avoid logging provider tokens, secrets, or encrypted payloads.
- Rate limit webhook endpoints separately from public API endpoints.
## Next steps
- [Mutable messages](/docs/server/mutable-messages): Understand message and version serials behind message_version_created.
- [Observability](/docs/server/observability): Metrics, logs, and persisting completed AI runs from webhooks.
- [Scaling](/docs/server/scaling): Queues, rolling deploys, and multi-node webhook delivery.
- [Configuration reference](/docs/reference/configuration): Every [webhooks] and [queue] key in one place.
---
# Push notifications (https://sockudo.io/docs/server/push-notifications)
> Configure and operate Sockudo's core push notification platform for FCM, APNs, Web Push, HMS, and WNS.
Sockudo ships a durable push notification pipeline next to its realtime server. WebSockets reach
clients that are connected right now; push notifications reach devices that are offline,
backgrounded, suspended by the OS, or outside the active channel session. Use this page when you
need to register devices, target them by channel, user, or device, and deliver through FCM, APNs,
Web Push, HMS, or WNS with idempotency, retries, and observable status.
For ActivityKit start, update, end, push-token rotation, and iOS 18 broadcast channels, see
[Apple Live Activities](/docs/server/apple-live-activities). For metrics, alerts, and outage
runbooks, see [Push operations](/docs/server/push-operations).
```mermaid
flowchart TB
D[Device or browser] -->|provider token| B[Your backend]
B -->|register device| S[Sockudo push API]
B -->|subscribe to channel| S
A[Application event] -->|publish or rule| S
S --> Q[(Push queue)]
Q --> F[FCM] & AP[APNs] & W[Web Push] & H[HMS] & N[WNS]
F & AP & W & H & N --> R[Device receives push]
```
## How push works
Every notification moves through the same five stages. Each stage has its own API surface and its
own failure modes, so it helps to keep them separate in your head.
| Stage | What happens | Where it lives |
| --- | --- | --- |
| 1. Registration | Your backend stores a **device**: a stable `id`, an optional `clientId`, and the provider **recipient** (FCM registration token, APNs device token, Web Push subscription, HMS token, or WNS channel URI). | `POST /apps/{appId}/push/deviceRegistrations` |
| 2. Subscription | A device is mapped to a realtime **channel** so channel-targeted publishes can find it. | `POST /apps/{appId}/push/channelSubscriptions` |
| 3. Publish | You submit a **publish**: one or more targets plus a payload. Sockudo validates, renders provider payloads, persists the work durably, and returns `202` with a `publishId`. | `POST /apps/{appId}/push/publish` |
| 4. Provider delivery | Planner, shard, and provider workers resolve targets to devices and send to FCM, APNs, Web Push, HMS, or WNS. | Push queue stages (`push.publish.v1`, `push.shards.v1`, `push.delivery..v1`) |
| 5. Status and feedback | Feedback workers record each provider outcome, schedule retries, dead-letter exhausted work, and invalidate dead tokens. | `GET /apps/{appId}/push/publish/{publishId}/status` |
The key mental model: **a `202` means "durably accepted", not "delivered"**. Delivery is
asynchronous, and the publish status record is the source of truth for what happened next.
### Concepts
| Concept | Meaning |
| --- | --- |
| Device registration | A device record tied to one provider recipient and an optional `clientId`. Responses expose only a `tokenHash`, never the raw token. |
| Device identity token | A per-device secret issued on admin registration. It lets a device-scoped caller update its own registration and subscriptions. |
| Channel subscription | A mapping between a device and a realtime channel for push targeting. |
| Credential | Provider configuration for FCM, APNs, Web Push, HMS, or WNS. |
| Publish | A push request accepted by Sockudo and fanned out asynchronously. |
| Publish status | The operational record for accepted, scheduled, dispatched, failed, or cancelled push work. |
| Template | A stored, localized notification body that a publish references by `templateId`. |
## Provider support
| Provider | Platforms | Cargo feature | Recipient (`transportType`) | Notes |
| --- | --- | --- | --- | --- |
| FCM | Android, web, cross-platform app backends | `push-fcm` | `gcm` with `registrationToken` | Best default for Android and Firebase-backed apps. |
| APNs | iOS, iPadOS, macOS, watchOS, tvOS, visionOS | `push-apns` | `apns` with `deviceToken` | Use sandbox for development tokens and production for App Store/TestFlight tokens. |
| Web Push | Browsers and installed PWAs | `push-webpush` | `web` with `endpoint`, `p256dh`, `auth` | Endpoints must be public HTTPS URLs; payload size and browser support vary. |
| HMS | Huawei devices | `push-hms` | `hms` with `registrationToken` | Use when shipping to Huawei Mobile Services environments. |
| WNS | Windows apps | `push-wns` | `wns` with `channelUri` | Use for Windows notification channels. |
Sockudo normalizes admission, idempotency, queueing, retries, status retention, and feedback.
Provider-specific limits still apply after the request leaves Sockudo.
## Set up the server
1. Build Sockudo with the push features you need. `monolith` runs the planner, dispatcher,
provider, feedback, retry, repair, and cleanup workers inside the server process. The `full`
feature set includes every push provider.
```bash
cargo build -p sockudo --release --features "push-fcm,push-apns,push-webpush,monolith"
```
2. Choose durable storage and a durable queue. `storage_driver` accepts `memory`, `postgres`,
`mysql`, `dynamodb`, `surrealdb`, or `scylladb`. `queue_driver` is `memory` or a broker
driver. A broker `queue_driver` runs on the shared `[queue]` backend and must match
`queue.driver`. Push needs per-message delays, so it starts only on `redis`, `redis-cluster`,
standard `sqs` and `pulsar`; see [push queue backends](/docs/reference/configuration#push).
3. Enable each provider and configure its credentials (see
[Provider credentials](#provider-credentials)).
```toml title="config.toml"
[push]
storage_driver = "postgres"
queue_driver = "redis"
allow_memory_drivers = false
fcm_enabled = true
apns_enabled = true
webpush_enabled = true
publish_status_ttl_days = 30
analytics_retention_days = 30
scheduler_interval_secs = 5
cleanup_interval_secs = 300
cleanup_batch_size = 1000
cleanup_max_deleted_per_tick = 100000
[push.apns]
# Required whenever apns_enabled = true.
topic = "com.example.app"
```
4. Start the server and confirm each provider worker reports ready in the logs. A provider whose
credentials are missing logs a warning and does not start its dispatch worker.
In `mode = "production"`, `storage_driver = "memory"` and `queue_driver = "memory"` are rejected
unless `allow_memory_drivers = true` or `PUSH_ALLOW_MEMORY_DRIVERS=true` is set explicitly. Memory
drivers are node-local and lose state on restart or dead-node loss.
APNs connection pooling, Live Activity topics, broadcast and channel-management endpoints, and
timeouts are configured under `[push.apns]`. Retry policy lives under `[push.retry]`, the provider
circuit breaker under `[push.circuit_breaker]`, per-app defaults under `[push.default_quotas]`, and
log redaction under `[push.payload_redaction]`. See the
[configuration reference](/docs/reference/configuration) for every key.
## Provider credentials
Provider workers in a monolith process load credentials at boot, from environment variables or
from a credential stored through the push API. Most provider credential variables also accept an
unprefixed alias, for example `FCM_PROJECT_ID`, `APNS_TEAM_ID`, or `VAPID_PRIVATE_KEY`.
| Provider | Environment credentials | Stored credential |
| --- | --- | --- |
| FCM | `PUSH_FCM_SERVICE_ACCOUNT_JSON_PATH` or `PUSH_FCM_SERVICE_ACCOUNT_JSON`. `PUSH_FCM_PROJECT_ID` is optional when the service account JSON contains `project_id`. `PUSH_FCM_PROVIDER_TOKEN` (with `PUSH_FCM_PROJECT_ID`) remains only for short-lived legacy static-token dispatch. | Upload to `push/credentials/fcm` and set `PUSH_FCM_APP_ID` (and optionally `PUSH_FCM_CREDENTIAL_ID`, default `fcm`). |
| APNs | `PUSH_APNS_TEAM_ID`, `PUSH_APNS_KEY_ID`, and `PUSH_APNS_PRIVATE_KEY_PATH` or `PUSH_APNS_PRIVATE_KEY` for token auth; or `PUSH_APNS_PROVIDER_TOKEN`. Topic from `[push.apns].topic` or `PUSH_APNS_TOPIC`. | Upload to `push/credentials/apns` and set `PUSH_APNS_APP_ID` (and optionally `PUSH_APNS_CREDENTIAL_ID`, default `apns`). Stored material can be `p12`, `pem`, or `teamId` + `keyId` + `privateKey`. |
| Web Push | `PUSH_WEBPUSH_VAPID_PRIVATE_KEY` (required) and `PUSH_WEBPUSH_VAPID_CONTACT` (defaults to `mailto:sockudo-webpush@example.com`). | Accepted by `push/credentials/webpush`; the dispatch worker reads the VAPID key from the environment. |
| HMS | `PUSH_HMS_APP_ID` and a pre-generated `PUSH_HMS_PROVIDER_TOKEN`. Optional `PUSH_HMS_ENDPOINT`. | Accepted by `push/credentials/hms`; the dispatch worker reads the environment. |
| WNS | A pre-generated `PUSH_WNS_PROVIDER_TOKEN`. | Accepted by `push/credentials/wns`; the dispatch worker reads the environment. |
Point APNs workers at the sandbox with `PUSH_APNS_ENDPOINT=https://api.sandbox.push.apple.com` (or
`[push.apns].endpoint`) when you test development-build tokens.
```bash title=".env"
PUSH_FCM_ENABLED=true
PUSH_FCM_SERVICE_ACCOUNT_JSON_PATH=/run/secrets/fcm-service-account.json
PUSH_APNS_ENABLED=true
PUSH_APNS_TOPIC=com.example.app
PUSH_APNS_TEAM_ID=TEAMID1234
PUSH_APNS_KEY_ID=KEYID12345
PUSH_APNS_PRIVATE_KEY_PATH=/run/secrets/AuthKey_KEYID12345.p8
PUSH_WEBPUSH_ENABLED=true
PUSH_WEBPUSH_VAPID_PRIVATE_KEY=...
PUSH_WEBPUSH_VAPID_CONTACT=mailto:ops@example.com
```
### Store a credential through the API
Credential endpoints require the `push-admin` capability. Secret fields are encrypted before they
are stored and are never returned.
```http
POST /apps/{appId}/push/credentials/apns
Content-Type: application/json
{
"credentialId": "apns",
"version": 2,
"teamId": "TEAMID1234",
"keyId": "KEYID12345",
"privateKey": "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----"
}
```
```json title="201 Created"
{
"app_id": "my-app",
"credential_id": "apns",
"provider": "apns",
"version": 2
}
```
| Endpoint | Body fields |
| --- | --- |
| `POST /apps/{appId}/push/credentials/fcm` | `serviceAccountJson` (object) |
| `POST /apps/{appId}/push/credentials/apns` | `p12` + `p12Password`, or `pem`, or `teamId` + `keyId` + `privateKey` |
| `POST /apps/{appId}/push/credentials/webpush` | `publicKey`, `privateKey` |
| `POST /apps/{appId}/push/credentials/hms` | `hmsAppId`, `clientSecret` |
| `POST /apps/{appId}/push/credentials/wns` | `packageSid`, `clientSecret` |
| `GET /apps/{appId}/push/credentials` | Lists stored credentials (metadata only), paged with `limit` and `cursor`. |
Every credential body also accepts optional `credentialId` (defaults to the provider name) and
`version` (defaults to `1`).
Set `PUSH_CREDENTIAL_ENCRYPTION_KEY` before uploading credentials so Sockudo stores decryptable
AES-GCM envelopes that workers can load later. Changing provider capability or credentials
requires a provider worker restart; stored credentials are reloaded when a supervised provider
worker restarts.
## Authorization model
Every push endpoint is a signed app API request, authenticated with the app key and secret exactly
like the rest of the [HTTP API](/docs/server/http-api). On top of that, the
`x-sockudo-push-capability` header selects one of two capabilities:
| Capability | When to use | What it may do |
| --- | --- | --- |
| `push-admin` (default when the header is absent) | Your trusted backend. | Everything: credentials, templates, publishes, status, dead letters, any device or subscription. |
| `push-subscribe` | Device-scoped calls that act on one existing device. | Update an existing device registration, and read, add, or remove that device's channel subscriptions. Requires `x-sockudo-device-identity-token` (or an `Authorization` bearer token) matching the device. |
An admin registration of a new device returns a one-time `deviceIdentityToken`. Send
`x-sockudo-rotate-device-identity-token: true` with an admin registration to issue a new one.
Mobile and browser clients should call your backend, never Sockudo's push API directly with app
credentials. Your backend authenticates the user, validates device ownership, and forwards to
Sockudo.
## Register devices
Device IDs should be stable per installation. Provider tokens rotate; register again with the same
`id` whenever the platform SDK gives you a new token. The response tells you whether anything
changed.
```http
POST /apps/{appId}/push/deviceRegistrations
Content-Type: application/json
x-sockudo-push-capability: push-admin
{
"appId": "my-app",
"id": "ios-7f3c2a",
"clientId": "user-42",
"formFactor": "phone",
"platform": "ios",
"timezone": "Europe/Berlin",
"locale": "de-DE",
"deviceSecret": "server-assigned",
"push": {
"state": "ACTIVE",
"recipient": { "transportType": "apns", "deviceToken": "APNS_DEVICE_TOKEN" }
}
}
```
```json title="201 Created"
{
"change": "inserted",
"tokenHash": "9c1e...",
"device": {
"appId": "my-app",
"id": "ios-7f3c2a",
"clientId": "user-42",
"formFactor": "phone",
"platform": "ios",
"timezone": "Europe/Berlin",
"locale": "de-DE",
"lastActiveAtMs": 0,
"pushState": "ACTIVE",
"pushFailureCount": 0,
"recipient": { "transportType": "apns", "provider": "apns", "tokenHash": "9c1e..." }
},
"deviceIdentityToken": "DEVICE_IDENTITY_TOKEN"
}
```
- `formFactor`: `phone`, `tablet`, `desktop`, `tv`, `car`, `watch`, `embedded`, or `other`.
- `platform`: `android`, `ios`, `browser`, `windows`, `macos`, `watchos`, `tvos`, or `other`.
- `push.state`: `ACTIVE`, `FAILING`, or `FAILED`. Register new devices as `ACTIVE`.
- `deviceSecret` is part of the device schema, but the server always replaces it with the hash of
the issued device identity token; send any placeholder.
- `change` is `inserted`, `updated`, or `unchanged`.
With the Node server SDK, the same call from your backend looks like this:
```ts title="routes/push.ts"
app.post("/api/push/devices", async (req, res) => {
const user = requireUser(req);
const result = await sockudo.activateDevice({
appId: process.env.SOCKUDO_APP_ID!,
id: req.body.deviceId,
clientId: user.id,
formFactor: "phone",
platform: req.body.platform,
timezone: req.body.timezone,
locale: req.body.locale,
deviceSecret: "server-assigned",
push: {
state: "ACTIVE",
recipient: { transportType: "gcm", registrationToken: req.body.fcmToken },
},
});
res.json({ tokenHash: result.tokenHash });
});
```
Other device operations:
| Request | Capability | Result |
| --- | --- | --- |
| `GET /apps/{appId}/push/deviceRegistrations/{id}` | admin, or subscribe with the device's identity token | The device (without raw tokens). |
| `GET /apps/{appId}/push/deviceRegistrations?limit=&cursor=` | admin | `{ "items": [...], "next_cursor": ..., "has_more": ... }` |
| `DELETE /apps/{appId}/push/deviceRegistrations/{id}` | admin, or subscribe with the device's identity token | `204 No Content` |
| `DELETE /apps/{appId}/push/deviceRegistrations?clientId=user-42` | admin | `{ "deleted": n, "consistency": "eventual" }` |
List endpoints accept `limit` between 1 and 1000 and return an opaque `next_cursor`.
Web Push `endpoint` values must be `https` URLs with a host and no user info. Localhost, `.local`,
and private or local IP ranges are rejected unless `PUSH_ALLOW_LOCAL_WEB_PUSH_ENDPOINTS=true` is set
for local tests. `PUSH_WEBPUSH_DENIED_HOSTS` and `PUSH_WEBPUSH_ALLOWED_HOSTS` further restrict
accepted hosts.
## Subscribe devices to channels
A channel subscription lets a publish that targets `{ "type": "channel" }` reach every device
subscribed to that channel. Pass the `tokenHash` returned by registration.
```http
POST /apps/{appId}/push/channelSubscriptions
Content-Type: application/json
x-sockudo-push-capability: push-admin
{
"appId": "my-app",
"channel": "orders",
"deviceId": "ios-7f3c2a",
"clientId": "user-42",
"provider": "apns",
"tokenHash": "9c1e..."
}
```
The response is `201 Created` with the stored subscription. With the Node SDK:
```ts
await sockudo.upsertChannelPushSubscription({
appId: process.env.SOCKUDO_APP_ID!,
channel: "orders",
deviceId: "ios-7f3c2a",
clientId: "user-42",
provider: "apns",
tokenHash: registration.tokenHash,
credentialVersion: 1,
});
```
| Request | Result |
| --- | --- |
| `GET /apps/{appId}/push/channelSubscriptions?channel=orders` | Devices subscribed to a channel. |
| `GET /apps/{appId}/push/channelSubscriptions?deviceId=ios-7f3c2a` | Channels a device is subscribed to. `push-subscribe` callers must pass their own `deviceId`. |
| `GET /apps/{appId}/push/channelSubscriptions/channels` | Every channel with at least one push subscription (admin only). |
| `DELETE /apps/{appId}/push/channelSubscriptions?channel=orders` or `?deviceId=...` | `{ "deleted": n }`. One of `channel` or `deviceId` is required. |
Use the same authorization model as realtime channel subscription. If the user cannot subscribe to
`private-orders` over WebSocket, do not subscribe their device to `private-orders` for push.
```mermaid
sequenceDiagram
participant App as Client app
participant API as Your backend
participant S as Sockudo
participant P as Push provider
App->>API: current provider token
API->>API: verify user and device
API->>S: register device
API->>S: subscribe device to channel
API->>S: publish push
S-->>API: 202 publishId
S->>P: provider request
P-->>S: accepted or failed
API->>S: get publish status
S-->>API: state and counters
```
## Publish
A publish combines **targets** (`recipients`) with a **payload**. Publish requires `push-admin`.
```http
POST /apps/{appId}/push/publish
Content-Type: application/json
x-sockudo-push-capability: push-admin
{
"publishId": "order-ord_123-packed",
"recipients": [
{ "type": "channel", "channel": "orders" },
{ "type": "client", "client_id": "user-42" }
],
"payload": {
"title": "Order updated",
"body": "Order ord_123 is packed",
"collapseKey": "order-ord_123",
"templateData": { "order_id": "ord_123" }
}
}
```
```json title="202 Accepted"
{
"publishId": "order-ord_123-packed",
"status": "accepted",
"expectedRecipients": 2,
"fanoutRegime": "fastPath",
"renderedPayloads": [
{
"provider": "apns",
"payload": {
"headers": { "apns-push-type": "alert", "apns-priority": "10", "apns-collapse-id": "order-ord_123" },
"aps": { "alert": { "title": "Order updated", "body": "Order ord_123 is packed" }, "sound": null },
"data": { "order_id": "ord_123" }
},
"usedOverride": false
}
]
}
```
`renderedPayloads` contains one preview per provider (FCM, APNs, Web Push, HMS, WNS, and the
realtime transport); the example above shows only APNs. `fanoutRegime` is `fastPath` below
`push.fanout_fast_threshold` expected recipients and `shardPath` above it.
With the Node SDK:
```ts
const response = await sockudo.publishPush({
publishId: "order-ord_123-packed",
recipients: [
{ type: "channel", channel: "orders" },
{ type: "client", client_id: "user-42" },
],
payload: {
title: "Order updated",
body: "Order ord_123 is packed",
templateData: { order_id: "ord_123" },
},
});
console.log(response.publishId);
```
### Targets
| `type` | Fields | Resolves to |
| --- | --- | --- |
| `device` | `device_id` | One registered device. |
| `client` | `client_id` | Every device registered with that `clientId`. |
| `channel` | `channel` | Every device subscribed to that channel. |
| `recipient` | `recipient` (a raw provider recipient, for example `{ "transportType": "gcm", "registrationToken": "..." }`) | One provider token, without a stored device. Requires `push-admin`. |
| `registeredTopic`, `userTopic` | `topic` | Topic-scoped targets. |
| `providerTopic` | `provider`, `topic` | A provider-native topic. |
| `providerCondition` | `provider`, `condition` | A provider-native condition expression. |
| `indexedFilter` | `filter` | An indexed device filter. |
### Payload fields
| Field | Meaning |
| --- | --- |
| `title`, `body` | Notification text. |
| `icon` | An `https` image URL. |
| `sound` | Platform sound name. |
| `collapseKey` | Collapses superseded notifications (APNs `apns-collapse-id`, Web Push `topic`, FCM/HMS `collapse_key`). |
| `templateData` | Arbitrary JSON delivered as the provider `data` object, and the source for template placeholders. |
| `templateId` | References a stored template. |
### Idempotency
Publishing is idempotent in two ways. Reusing a `publishId` returns the original publish instead of
creating new work. Sending an identical request body (same targets, payload, overrides, and timing)
also resolves to the original publish. Idempotency records last until `expiresAtMs`, or 24 hours
when no expiry is set.
Use a deterministic `publishId` such as `order-ord_123-packed`. Your own retries, queue
redeliveries, and double-clicks then collapse into one notification instead of several.
`sync: true` or `?mode=sync` does not block on delivery. The request is still admitted
asynchronously and the response carries `X-Sockudo-Forced-Async: publish-api-is-async`. Use
`POST /apps/{appId}/push/batch/publish` with a JSON array of publish requests to admit several at
once; it returns `{ "items": [...] }` with one accepted response per request.
## Track publish status
```http
GET /apps/{appId}/push/publish/order-ord_123-packed/status
x-sockudo-push-capability: push-admin
```
```json title="200 OK"
{
"appId": "my-app",
"publishId": "order-ord_123-packed",
"state": "succeeded",
"counters": {
"planned": 2,
"dispatched": 2,
"succeeded": 2,
"failed": 0,
"expired": 0,
"retryScheduled": 0,
"retryAttempted": 0,
"deadLettered": 0
},
"fanoutRegime": "fastPath"
}
```
| `state` | Meaning |
| --- | --- |
| `queued` | Durably accepted; waiting for the planner. The publish response reports this as `accepted`. |
| `planning` | Targets are being resolved to devices. |
| `throttled` | Delivery is paused by rate or quota limits. |
| `dispatching` | Provider requests are in flight or scheduled for retry. |
| `succeeded` | Every planned delivery succeeded. |
| `partiallySucceeded` | Some deliveries succeeded and some failed, expired, or dead-lettered. The publish response label is `partially_succeeded`. |
| `failed` | Every delivery failed terminally. |
| `expired` | `expiresAtMs` passed before delivery. |
| `dead_lettered` | Retries were exhausted with no successes. |
| `quota_exceeded` | Rejected at admission by a fanout or delivery quota. |
| `cancelled` | Cancelled before dispatch. |
`planned` is the acceptance-time audience estimate. Terminal states are absorbing: once a publish
is `succeeded`, `failed`, `expired`, `dead_lettered`, `partiallySucceeded`, `quota_exceeded`, or
`cancelled`, it never moves again.
## Payload fidelity, templates, and overrides
Publish responses include `renderedPayloads`, one provider-specific preview per provider. The same
renderer is used by admission and dispatch. Provider workers send the previewed provider payload,
except for runtime-only fields such as provider tokens, APNs device tokens, generated authorization
headers, and provider request IDs.
`providerOverrides` are provider-specific complete payloads. When an override exists for a
provider, it replaces the generic payload mapping for that provider. APNs override headers become
APNs request headers, Web Push `headers.ttl`, `headers.urgency`, and `headers.topic` become Web
Push request headers, and FCM/HMS/WNS overrides are sent as the provider body with runtime recipient
tokens added by the worker. Overrides are validated and size-checked before dispatch.
```json
{
"recipients": [{ "type": "client", "client_id": "user-42" }],
"payload": { "title": "Order updated", "body": "Order ord_123 is packed" },
"providerOverrides": [
{
"provider": "fcm",
"payload": {
"message": {
"notification": { "title": "Order updated", "body": "Packed and ready" },
"android": { "priority": "high" }
}
}
}
]
}
```
### Templates
Store localized content once and reference it by `templateId`:
```http
POST /apps/{appId}/push/templates
Content-Type: application/json
{
"appId": "my-app",
"templateId": "order-packed",
"defaultLocale": "en",
"locales": {
"en": { "title": "Order packed", "body": "Order {{ data.order_id }} is packed" },
"de": { "title": "Bestellung verpackt", "body": "Bestellung {{ data.order_id }} ist verpackt" }
}
}
```
The response is `201 Created` with the stored template. `defaultLocale` must exist in `locales`.
`GET /apps/{appId}/push/templates`, `GET /apps/{appId}/push/templates/{id}`, and
`DELETE /apps/{appId}/push/templates/{id}` list, read, and remove templates.
Templates are resolved during publish admission. A `payload.templateId` must reference an existing
template; missing templates fail the publish request before work is accepted. Sockudo selects the
template locale from `payload.templateData.locale`, falling back from exact locale to language and
then the template's `defaultLocale`. Template fields fill missing payload fields, while explicit
payload fields win. Template provider overrides are applied first and request `providerOverrides`
win when both define the same provider. The resolved payload is stored in the accepted publish work,
so later template edits do not change queued delivery or retry attempts.
Template placeholders use bounded `{{ data.path }}` substitutions against `payload.templateData`.
Missing placeholders fail admission with a precise invalid-template error; objects and arrays are
not coerced into strings.
## Delayed delivery and cancellation
```ts
const delayed = await sockudo.publishPush({
notBeforeMs: Date.parse("2026-05-19T18:00:00Z"),
expiresAtMs: Date.parse("2026-05-19T18:10:00Z"),
recipients: [{ type: "client", client_id: "user-42" }],
payload: { title: "Reminder", body: "Your room starts soon" },
});
```
`notBeforeMs` delays provider dispatch; `expiresAtMs` prevents late sends. Both are enforced at
delivery time, including on retries.
The public HTTP API does not expose a separate schedule-create endpoint. Use
`POST /apps/{appId}/push/publish` with `notBeforeMs` for delayed delivery. Persisted scheduled jobs
created by internal scheduler/store integrations can be cancelled before emission with
`DELETE /apps/{appId}/push/scheduled/{jobId}`, which returns `204 No Content`.
## Channel-triggered push rule
Push rules convert normal channel events into push work, with no extra API call from your backend.
This is useful for product events and AI agent completion notifications.
```toml title="config.toml"
[[push_rules]]
enabled = true
channel_pattern = "notifications:*"
event_filter = ["agent-complete", "order-ready"]
rate_limit_per_second = 100
[push_rules.payload_mapping]
title_field = "title"
body_field = "body"
template_data_field = "data"
include_remaining_fields = true
```
A realtime publish such as this one now also produces a channel-targeted push to every device
subscribed to `notifications:user-42`:
```json
{
"name": "agent-complete",
"channel": "notifications:user-42",
"data": {
"title": "Agent finished",
"body": "Your answer is ready",
"data": {
"session_id": "sess_01J"
}
}
}
```
- `channel_pattern` supports an exact name, `*`, or a trailing wildcard such as `notifications:*`.
- `event_filter` must list 1 to 32 event names.
- The event `data` must be a JSON object with string `title` and `body` fields (or the fields named
by `payload_mapping`). Events that do not match produce no push.
## Admission readiness
Sockudo fails closed before accepting publish work it already knows cannot be processed safely. The
push publish API requires a healthy push queue, production-safe storage, local pipeline workers, and
local provider-worker capability. Raw FCM/APNs/Web Push/HMS/WNS recipients require the matching
provider worker in this process. Channel, client, and device targets require at least one active
provider worker without scanning large channel subscription sets during admission; shard-path
publishes also require a local shard worker. The planner still resolves exact devices
asynchronously.
| Response | Cause |
| --- | --- |
| `202` with `status: "quota_exceeded"` | Quota rejection. Sockudo persists a terminal `quota_exceeded` publish status and idempotency record without appending publish log work or enqueueing the pipeline. |
| `429` with `Retry-After` | Queue pressure (a critical stage crossed its lag or oldest-age threshold), or the per-app acceptance rate (`[push.default_quotas].acceptance_rps`, overridable with `PUSH_ACCEPTANCE_RATE_LIMIT`) was exceeded. |
| `503` | Readiness failure: queue unhealthy, memory drivers in production, missing provider feature or credentials, or no capable worker. |
| `400` | Invalid targets, payload, template, override, or Live Activity fields. |
| `403` | Missing `push-admin` capability. |
Errors use the standard body `{ "error": "...", "code": "...", "status": 503 }`.
## Cluster queue delivery guarantees
Push queue delivery is at-least-once. Accepted publish work is durable before the API returns, and
every delivery result is idempotently applied by feedback workers. A provider send can still be
duplicated if a worker crashes after the provider accepts the request but before `push.results.v1`
is produced or before the original delivery batch is acknowledged. Sockudo keeps the duplicate
window bounded with stable delivery idempotency keys, deterministic retry context, and duplicate
feedback suppression, but it does not claim exactly-once provider delivery.
Set `collapseKey` on notifications that represent state ("order is packed") so a rare duplicate
replaces the previous notification on the device instead of stacking.
In monolith deployments, each node consumes only stages it can process. Planner, shard, feedback,
retry, and dead-letter stages are consumed only by nodes with the matching local worker enabled.
Provider delivery stages such as `push.delivery.fcm.v1` are consumed only by nodes whose provider
worker is active and credential-ready at boot. Changing provider capability or credentials requires
worker restart; stored credentials are reloaded when a supervised provider worker restarts.
External queue adapters pull broker messages into a bounded node-local ready/pending handoff before
Sockudo workers ack, nack, or dead-letter them. If the local worker does not finish before the local
lease timeout, the adapter requeues the envelope with backoff. Queue health and lag reported by the
generic queue manager adapter distinguish this node's local pull-ahead depth and oldest actionable
handoff age from broker-wide backlog, so operators can alert on Sockudo-owned worker staleness with
`sockudo_push_queue_oldest_age_seconds` and use backend-native tools for deeper broker visibility.
The monolith repair worker also scans durable publish-log entries that remain `queued` past
`push.repair_min_age_secs` and recreates missing `push.publish.v1` work if an accepted publish
lost its queue message before planning began.
## Retry and dead letters
Provider throttling, 5xx responses, and transport failures are classified as retryable provider,
quota, or network failures. Feedback workers schedule retry entries on `push.retry.v1` with the
original delivery job context, a deterministic retry idempotency key, the next attempt number, the
first-attempt timestamp, retry deadline, and the provider error class. Retry scheduler workers
consume only due entries, respect provider `Retry-After` deadlines when configured, and otherwise
use bounded exponential backoff with deterministic jitter.
```toml title="config.toml (defaults)"
[push.retry]
max_attempts = 5
initial_backoff_ms = 1000
max_backoff_ms = 60000
max_elapsed_secs = 86400
jitter = true
jitter_ratio_percent = 20
respect_retry_after = true
```
Retries stop at the earliest of `max_attempts`, `max_elapsed_secs`, or the publish/job
`expires_at_ms`. Exhausted retry work is emitted to `push.deadletters.v1` and the publish status
moves to `dead_lettered` unless some deliveries already succeeded, in which case it becomes
partially succeeded. If the publish expiry passes before a retry can run, the publish moves to
`expired`. Pending retry work increments `retryScheduled`; retry dispatch increments
`retryAttempted`; `dispatched` counts terminal provider outcomes so retry attempts do not inflate
completion accounting.
Publish timing is enforced at delivery time. `notBeforeMs` delays provider dispatch; provider
workers split mixed due/future batches so due jobs are not blocked behind future jobs. `expiresAtMs`
prevents late sends; expired delivery jobs are recorded as `expired` and are not sent to providers.
Retry scheduling preserves the original timing window and does not run past publish or job expiry.
Operators inspect aggregate status with `GET /apps/{appId}/push/publish/{publishId}/status`, retry
and dead-letter queue lag through the health/backpressure surfaces, and metrics such as
`sockudo_push_queue_oldest_age_seconds{stage=...,state=...}`,
`sockudo_push_retry_scheduled_total`, `sockudo_push_retry_attempted_total`,
`sockudo_push_retry_dead_lettered_total`, `sockudo_push_retry_malformed_total`, and
`sockudo_push_provider_failures_total{failure_class=...}`. Inspect and replay dead letters with
`GET /apps/{appId}/push/deadLetters` and
`POST /apps/{appId}/push/deadLetters/{deadLetterId}/replay`; see
[Push operations](/docs/server/push-operations).
Lifecycle safety checks expose `sockudo_push_invariant_violations_total{invariant=...}`. The bounded
`status_transition` label means a stale, non-monotonic status write was ignored. The metric does
not include payloads, device tokens, or provider credentials.
Publish-status mutations use storage-level compare-and-swap in every backend. Short-lived write
contention increments `sockudo_push_status_cas_conflicts_total{component=...}` and is retried with
a fixed bound. `sockudo_push_status_cas_exhausted_total{component=...}` means that bound was
exhausted; the worker returns a structured error instead of silently accepting a lost update.
Component labels come from a fixed internal set and neither metric includes payloads, tokens,
credentials, app IDs, or publish IDs.
All installations must stop old push admission and worker processes before starting
revision-aware workers; do not run old blind status writers alongside them. PostgreSQL and MySQL
must apply the push schema-version-2 migration while workers are stopped. DynamoDB, SurrealDB, and
ScyllaDB store the revision in the status document envelope and need no table migration, but still
require the same drain-and-replace deployment boundary.
Upgrade note: provider workers now enqueue internal `deliveryFeedback` payloads on
`push.results.v1` so feedback workers can schedule retries with replayable job context. Older
`deliveryResult` payloads are still accepted, but retryable legacy results without job context are
dead-lettered instead of being retried blindly.
## Provider outcomes and device invalidation
Every provider response becomes a delivery outcome (`accepted`, `rejected`, `retryable`,
`expired`, or `cancelled`) plus a failure class. The outcome decides whether the delivery is retried;
the class decides what happens to the device and how the failure is counted.
| `failure_class` | Typical cause | Device effect |
| --- | --- | --- |
| `device_terminal` | The token is permanently invalid (see below). | Device registration is deleted. |
| `device_transient` | A device-specific, possibly temporary failure. | Failure count increments; the device moves from `ACTIVE` to `FAILING`, then `FAILED`. |
| `provider_transient` | Provider `5xx`. | None. |
| `provider_quota` | Provider `429` or quota response. | None. |
| `credential_auth` | Bad key, project, topic, environment, or VAPID configuration. | None. |
| `caller_payload` | The provider rejected the payload. | None. |
| `network_transport` | DNS, TLS, or connection failure. | None. |
| `unknown` | Unclassified response. | None. |
Sockudo deletes or terminally invalidates a device only for device-specific terminal failures:
- FCM `UNREGISTERED`, `registration-token-not-registered`, or an `INVALID_ARGUMENT` response that
specifically identifies the registration token as invalid.
- APNs `BadDeviceToken` or `Unregistered`.
- Web Push, HMS, and WNS expired subscription/channel responses such as `404` or `410`.
Provider-wide failures do not increment device failure counts and do not delete devices. This
includes FCM `SENDER_ID_MISMATCH` or project mismatch, APNs topic/environment/auth failures, Web
Push VAPID/auth failures, provider `429` quota responses, provider `5xx` responses, DNS/TLS/network
transport failures, and caller payload errors. These outcomes still update publish status, schedule
retry or dead-letter work according to the retry policy, and emit classified metrics.
Operators should treat `credential_auth` and project/topic mismatch failures as credential or
configuration incidents: rotate or fix the provider credential, project id, APNs topic/environment,
or VAPID configuration, then inspect `GET /apps/{appId}/push/deadLetters` and replay entries marked
`replayable` or republish from the original source as appropriate. Do not purge device registries
for these classes.
The `sockudo_push_provider_failures_total{failure_class=...}` metric separates these classes. The
`sockudo_push_token_invalidation_guard_total` metric and `token-invalidation-guard` meta event fire
when a publish crosses the built-in invalidation spike threshold: at least 10 failed deliveries and
at least 20% of the planned recipients.
## Retention and cleanup
The cleanup worker runs in monolith deployments when `push.cleanup_interval_secs > 0`. It removes
terminal publish statuses older than `push.publish_status_ttl_days`, delivery events and operator
invalidation events older than `push.analytics_retention_days`, expired idempotency records, and
expired scheduler locks. It does not delete active scheduled jobs, active retry work, credentials,
templates, or active device registrations.
Cleanup is incremental: each tick is bounded by `push.cleanup_batch_size` per category and
`push.cleanup_max_deleted_per_tick` overall. See [Push operations](/docs/server/push-operations)
for metrics, alerts, and outage runbooks.
## Capacity planning
Push fanout has a different bottleneck profile than WebSocket fanout:
- provider rate limits
- token invalidation churn
- per-platform payload size limits (4096 bytes for FCM, APNs device, Web Push, and HMS payloads)
- queue depth and worker concurrency (`dispatch_worker_count`, `dispatch_max_outbound_requests`)
- status retention writes
- scheduled publish scans
- provider callback volume
Before a campaign, test admission and provider dispatch separately. A healthy API admission rate
does not prove provider delivery capacity.
## Benchmarking
Use the repository push benchmark for repeatable scenarios. `--mode` is `seed` (register devices
and subscriptions), `admission` (publish load), `e2e`, or `all`.
```bash
node scripts/push-benchmark.mjs \
--url http://127.0.0.1:6001 \
--app-id app-id \
--key app-key \
--secret app-secret \
--mode all \
--provider fcm \
--devices 10000 \
--channels 100 \
--concurrency 50 \
--duration 60
```
Track accepted publishes, queue depth, provider dispatch latency, provider error classes, and status
write latency.
The Criterion benchmark `push_retry_scheduler/memory_drain_100k_due_retry_entries_budget_lt_5s`
seeds and drains 100k due retry entries through the memory queue/store path as a regression guard
for retry scheduling overhead.
## Troubleshooting
| Symptom | Check |
| --- | --- |
| `202` returned but no notification | Inspect publish status and provider errors. |
| `400 invalid JSON body` on registration | Include `appId`, `deviceSecret`, `push.state`, `timezone`, and `locale`; check field casing. |
| `403 push-admin capability is required` | The request sent `x-sockudo-push-capability: push-subscribe` to an admin-only endpoint. |
| `401 invalid deviceIdentityToken` | The device-scoped call used a missing or rotated identity token. |
| APNs rejects token | Verify bundle ID, environment, topic, and token freshness. |
| Web Push rejected | Verify VAPID key and contact, and that the endpoint host is public HTTPS. |
| FCM unauthorized | Verify service account and project. |
| Channel push misses users | Verify channel push subscriptions and `client_id` mapping. |
| Large push is rejected | Check platform payload limits and remove nonessential data. |
| Push publish returns `503` | Check push queue health, memory-driver production guard, provider feature flags, provider credentials, and dispatch worker count. |
| Push publish returns `429` | Check publish-log, shard, delivery result, retry, dead-letter, active provider delivery queue depth, and `sockudo_push_queue_oldest_age_seconds`. |
| Publish remains `dispatching` | Check retry scheduler workers, `push.retry.v1` lag, dead-letter lag, and provider retry-after deadlines. |
| Publish becomes `dead_lettered` | Inspect `GET /apps/{appId}/push/deadLetters`, retry metrics, provider outage history, and `max_attempts`/`max_elapsed_secs`; replay only entries marked `replayable`. |
| Cleanup failures increase | Check `sockudo_push_cleanup_errors_total`, store connectivity, and batch limits. |
Push payloads should wake the app and include stable IDs. Fetch authoritative application state
after the user opens the notification.
## Next steps
- [Apple Live Activities](/docs/server/apple-live-activities): Start, update, and end ActivityKit activities, and broadcast to iOS 18 channels.
- [Push operations](/docs/server/push-operations): Metrics, alerts, dead-letter replay, and provider outage runbooks.
- [AI Transport push](/docs/server/ai-transport-push-notifications): Notify users when an AI agent finishes while they are away.
- [Configuration reference](/docs/reference/configuration): Every push, retry, quota, and APNs configuration key.
---
# Apple Live Activities (https://sockudo.io/docs/server/apple-live-activities)
> Run ActivityKit start, update, end, and iOS 18 broadcast delivery through Sockudo's durable push pipeline.
# Apple Live Activities
This page covers driving iOS Live Activities from your server through Sockudo: starting an
activity remotely, updating and ending it, and broadcasting one update to millions of devices
through iOS 18 channels. Use it when your app shows live state on the Lock Screen or Dynamic Island
(rides, deliveries, scores, builds) and that state changes on your backend.
Sockudo supports the complete server-side ActivityKit lifecycle:
- token-based push-to-start, update, and end notifications
- automatic `.push-type.liveactivity` topics
- direct priority 5 (`conservePower`) and 10 (`immediate`), plus broadcast-only priority 1 (`lowPower`)
- `stale-date`, `dismissal-date`, relevance score, alerts, and iOS 18 input push token/channel fields
- iOS 18 broadcast channel create, inspect, delete, update, and end
- durable admission, scheduling, idempotency, retry, dead letters, status, metrics, and backpressure from the normal push pipeline
Sockudo builds and validates the APNs wire payload. Callers use camel-case typed fields and never
need to construct hyphenated `aps` keys or APNs headers themselves. The implementation follows
Apple's [ActivityKit remote update contract](https://developer.apple.com/documentation/ActivityKit/starting-and-updating-live-activities-with-activitykit-push-notifications),
[APNs request contract](https://developer.apple.com/documentation/usernotifications/sending-notification-requests-to-apns),
[broadcast request contract](https://developer.apple.com/documentation/usernotifications/sending-broadcast-push-notification-requests-to-apns),
and [channel management contract](https://developer.apple.com/documentation/usernotifications/sending-channel-management-requests-to-apns).
## Three ways to reach an activity
Every Live Activity request is a normal [push publish](/docs/server/push-notifications#publish)
with two additions: a Live Activity recipient and a typed `liveActivity` object. Which recipient you
use depends on what you are doing.
| Goal | Recipient `transportType` | Token or ID | Allowed events |
| --- | --- | --- | --- |
| Start a new activity remotely | `apnsLiveActivity` | The app's **push-to-start token** | `start` |
| Update or end one specific activity | `apnsLiveActivity` | That activity's **update token** | `update`, `end` |
| Update or end every activity watching shared content | `apnsLiveActivityBroadcast` | A broadcast **channel ID** plus its `storagePolicy` | `update`, `end` |
Direct tokens cost one APNs request per activity. A broadcast channel costs one APNs request no
matter how many devices subscribe, which is why shared content (a match score, a flight status)
belongs on a channel.
```mermaid
sequenceDiagram
participant App as iOS app
participant BE as Your backend
participant S as Sockudo
participant APNs as APNs
App->>BE: push-to-start token
BE->>S: publish start
S->>APNs: liveactivity push
APNs-->>App: activity starts
App->>BE: activity update token
BE->>S: publish update or end
S->>APNs: liveactivity push
APNs-->>App: content state changes
```
## Apple app setup
Enable Push Notifications and Live Activities for the application and widget extension. Add
`NSSupportsLiveActivities` to the app's Info.plist. If the product needs higher-frequency updates,
also add `NSSupportsLiveActivitiesFrequentUpdates`; users can still disable frequent updates, so
the app should observe ActivityKit authorization changes and tolerate coalescing.
Every activity update token and push-to-start token can rotate. Observe the async token sequences
for the entire activity lifetime, upload every new value to your backend, and stop using the old
value immediately. The Sockudo Swift client includes cancellable observers for both sequences.
## Server configuration
1. Build the server with `push-apns`. Add `monolith` when the same process should run the durable
planner, dispatcher, APNs delivery, feedback, repair, and cleanup workers:
```bash
cargo build -p sockudo --release --features "push-apns,monolith"
```
Without `monolith`, run the push worker roles separately; the HTTP API and broadcast-channel
management surface remain available in the server process.
2. Enable APNs, Live Activities, and (optionally) broadcast under `[push]` and `[push.apns]`:
```toml title="config.toml"
[push]
apns_enabled = true
dispatch_worker_count = 8
dispatch_max_outbound_requests = 256
[push.apns]
topic = "com.example.rides"
live_activities_enabled = true
bundle_id = "com.example.rides"
live_activity_topic = "" # derives com.example.rides.push-type.liveactivity
broadcast_enabled = true
endpoint = "https://api.push.apple.com"
broadcast_endpoint = "https://api-broadcast.push.apple.com"
management_endpoint = "https://api-manage-broadcast.push.apple.com:2196"
default_broadcast_expiration_secs = 3600
connect_timeout_ms = 5000
request_timeout_ms = 10000
pool_idle_timeout_secs = 90
max_idle_connections_per_host = 128
tcp_keepalive_secs = 60
http2_keepalive_interval_secs = 30
http2_keepalive_timeout_secs = 10
ca_certificate_path = "" # optional extra PEM roots
```
3. Provide APNs credentials, either as a stored credential (`POST /apps/{appId}/push/credentials/apns`
plus `PUSH_APNS_APP_ID`) or through `PUSH_APNS_TEAM_ID`, `PUSH_APNS_KEY_ID`, and
`PUSH_APNS_PRIVATE_KEY_PATH`. See
[Provider credentials](/docs/server/push-notifications#provider-credentials).
4. Start the server. Configuration is validated at boot; an invalid combination fails fast with
the offending `push.apns.*` key.
Startup validation enforces these rules:
| Rule | Error when violated |
| --- | --- |
| `live_activities_enabled` requires `push.apns_enabled`. | `push.apns.live_activities_enabled requires push.apns_enabled` |
| `broadcast_enabled` requires `live_activities_enabled`. | `push.apns.broadcast_enabled requires push.apns.live_activities_enabled` |
| `topic` must be set when APNs is enabled. `bundle_id` falls back to `topic`. | `push.apns.topic must not be empty when APNs is enabled` |
| `live_activity_topic`, when set, must equal `.push-type.liveactivity`. | `push.apns.live_activity_topic must equal .push-type.liveactivity` |
| `default_broadcast_expiration_secs` must be between 1 and 28800 (eight hours). | `push.apns.default_broadcast_expiration_secs must be between 1 and 28800` |
| All three endpoints must be `https` URLs without user information. | `push.apns. must be an HTTPS URL without user information` |
`ca_certificate_path` adds root certificates to the built-in public roots for every APNs, broadcast,
and channel-management connection. Use it for private egress proxies or for staged rollouts against
an APNs-compatible mock; leave it empty to trust only Apple's public certificate chain.
Use all three sandbox hosts for sandbox tokens and channels:
- `https://api.sandbox.push.apple.com`
- `https://api-broadcast.sandbox.push.apple.com`
- `https://api-manage-broadcast.sandbox.push.apple.com:2195`
Broadcast channel management supports the same APNs token or mTLS credential stored for dispatch.
For token authentication, configure `teamId`, `keyId`, and a `.p8` private key, or a short-lived
provider token. APNs provider JWTs are cached for 55 minutes and invalidated once on
`ExpiredProviderToken`; Sockudo does not regenerate them more frequently than Apple's recommended
interval. Secrets remain in the existing encrypted credential store or secret environment
variables.
`lowPower` maps to APNs priority 1 and is valid only for broadcast requests. Direct token delivery
uses `conservePower` (5, the default) or `immediate` (10).
## The liveActivity object
| Field | Type | Meaning |
| --- | --- | --- |
| `event` | `start`, `update`, or `end` | The ActivityKit event. |
| `timestamp` | UNIX seconds | Required and positive. Keep it current and monotonically ordered per activity. |
| `contentState` | object | Required. Must match your `Activity.ContentState`. |
| `attributesType` | string | The `ActivityAttributes` type name. Start only. |
| `attributes` | object | Static activity attributes. Start only. |
| `alert` | object | Alert shown with the push. Required for start. |
| `staleDate` | UNIX seconds | When the content becomes stale. Update only. |
| `dismissalDate` | UNIX seconds | When an ended activity leaves the Lock Screen. End only. |
| `relevanceScore` | number | Finite and non-negative. Orders multiple activities. |
| `inputPushToken` | boolean | iOS 18: ask the started activity for an update token. Start only. |
| `inputPushChannel` | string | iOS 18: subscribe the started activity to a broadcast channel. Start only. |
| `priority` | `lowPower`, `conservePower`, `immediate` | APNs priority 1, 5, or 10. Defaults to `conservePower`. |
Sockudo rejects a publish with `400` when:
- a Live Activity recipient is present but `liveActivity` is missing;
- `liveActivity` is combined with an APNs entry in `providerOverrides`;
- a `start` omits `attributesType`, `attributes`, or `alert`, or sets `staleDate` or `dismissalDate`;
- an `update` sets start fields, `dismissalDate`, `inputPushToken`, or `inputPushChannel`;
- an `end` sets start fields, `staleDate`, `inputPushToken`, or `inputPushChannel`;
- `inputPushToken` and `inputPushChannel` are both set;
- `lowPower` is used for direct (non-broadcast) delivery;
- a broadcast recipient carries a `start` event.
## Start from a push-to-start token
Broadcast notifications cannot start an activity. Send a start request to a push-to-start token:
```http
POST /apps/{appId}/push/publish
Content-Type: application/json
x-sockudo-push-capability: push-admin
{
"publishId": "ride-184-start-v1",
"recipients": [{
"type": "recipient",
"recipient": {
"transportType": "apnsLiveActivity",
"activityToken": "PUSH_TO_START_TOKEN"
}
}],
"payload": {},
"liveActivity": {
"event": "start",
"timestamp": 1725000000,
"contentState": { "status": "driverAssigned", "etaMinutes": 4 },
"attributesType": "RideAttributes",
"attributes": { "rideID": "ride-184" },
"alert": { "title": "Driver assigned", "body": "Arriving in 4 minutes" },
"inputPushToken": true,
"priority": "immediate"
}
}
```
```json title="202 Accepted"
{
"publishId": "ride-184-start-v1",
"status": "accepted",
"expectedRecipients": 1,
"fanoutRegime": "fastPath"
}
```
The real response also includes `renderedPayloads`, one preview per provider; it is omitted here.
On iOS 18, use either `inputPushToken: true` or `inputPushChannel: "CHANNEL_ID"`, never both.
The channel form lets the newly started activity subscribe directly to a broadcast channel.
## Update or end one activity
Use the activity's current update token with `transportType: "apnsLiveActivity"`. An update may
set `staleDate`; an end may set `dismissalDate`. Keep `timestamp` current and monotonically ordered
for a given activity.
```json title="Update"
{
"publishId": "ride-184-update-42",
"recipients": [{
"type": "recipient",
"recipient": {
"transportType": "apnsLiveActivity",
"activityToken": "CURRENT_ACTIVITY_UPDATE_TOKEN"
}
}],
"payload": {},
"liveActivity": {
"event": "update",
"timestamp": 1725000042,
"contentState": { "status": "arriving", "etaMinutes": 1 },
"staleDate": 1725000162,
"relevanceScore": 0.9,
"priority": "conservePower"
}
}
```
```json title="End"
{
"publishId": "ride-184-end",
"recipients": [{
"type": "recipient",
"recipient": {
"transportType": "apnsLiveActivity",
"activityToken": "CURRENT_ACTIVITY_UPDATE_TOKEN"
}
}],
"payload": {},
"liveActivity": {
"event": "end",
"timestamp": 1725000300,
"contentState": { "status": "completed", "etaMinutes": 0 },
"dismissalDate": 1725003900
}
}
```
Reusing `publishId` makes application retries deterministic: a retried request returns the
original publish instead of sending a second push. Derive it from the activity and update
sequence, as in `ride-184-update-42`.
## Broadcast at large scale
1. Create a channel through the signed, push-admin API:
```http
POST /apps/{appId}/push/liveActivities/channels
Content-Type: application/json
{"storagePolicy":"mostRecent"}
```
```json title="201 Created"
{ "channelId": "CHANNEL_ID", "storagePolicy": "mostRecent" }
```
2. Store the returned `channelId` and `storagePolicy` with the event it represents, and hand the
channel ID to apps so they can start activities with `inputPushChannel`.
3. Publish updates to the channel with an `apnsLiveActivityBroadcast` recipient (example below).
4. Delete the channel when the event is over.
`storagePolicy` is `noStorage` (the default) or `mostRecent`. `noStorage` maximizes the broadcast
budget and always sends `apns-expiration: 0`. `mostRecent` lets APNs retain only the latest
notification for offline devices, for at most eight hours. The policy is immutable: include the
returned `storagePolicy` in every publish target so Sockudo can apply the matching expiration rules.
Apple permits up to 10,000 channels per app in each environment. Reconcile the APNs inventory with
your event registry using the read endpoints:
| Request | Response |
| --- | --- |
| `GET /apps/{appId}/push/liveActivities/channels` | `200` with `{ "channels": ["CHANNEL_ID", ...] }` |
| `GET /apps/{appId}/push/liveActivities/channels/{channelId}` | `200` with `{ "channelId": "...", "storagePolicy": "..." }` |
| `DELETE /apps/{appId}/push/liveActivities/channels/{channelId}` | `204 No Content` |
Channel endpoints return `422 feature_disabled` unless both `push.apns_enabled` and
`push.apns.broadcast_enabled` are set. APNs `429` and `5xx` responses from channel management
surface as `503`; other APNs rejections surface as `400`.
```json title="Broadcast update"
{
"publishId": "match-90-score-3-1",
"recipients": [{
"type": "recipient",
"recipient": {
"transportType": "apnsLiveActivityBroadcast",
"channelId": "CHANNEL_ID",
"storagePolicy": "mostRecent"
}
}],
"payload": {},
"expiresAtMs": 1725003600000,
"liveActivity": {
"event": "update",
"timestamp": 1725000000,
"contentState": { "home": 3, "away": 1 },
"priority": "conservePower"
}
}
```
A broadcast target is one provider request regardless of subscriber count. Consequently,
`expectedRecipients` is `1`; delivery status means APNs accepted the broadcast, not that every
device displayed it. Use Apple's Push Notifications Console and delivery metrics for downstream
visibility.
Delete retired channels promptly, but never reuse or publish to a deleted ID. APNs may still
deliver an already stored message after channel deletion.
## Swift token observation
```swift title="RideActivityController.swift"
let updateTask = SockudoLiveActivityTokens.observePushTokens(for: activity) {
activityID, token in
await backend.replaceLiveActivityToken(activityID: activityID, token: token)
}
if #available(iOS 17.2, *) {
let startTask = SockudoLiveActivityTokens.observePushToStartTokens(
for: RideAttributes.self
) { token in
await backend.replacePushToStartToken(token)
}
}
```
Keep the returned task for as long as tokens are needed and cancel it when ownership ends. Tokens
are delivered as hexadecimal strings, ready to send as `activityToken`.
Mobile apps should send tokens only to their authenticated backend. That backend calls Sockudo;
app secrets and push-admin capability headers must never ship in the application.
## Local end-to-end example
`examples/apple-live-activities-backend` runs the whole path without Apple credentials: a mock
APNs in Docker that enforces Apple's wire contract, a ride-tracking backend built on the Node
server SDK, and a scenario that verifies every request the server makes. Its `run.sh` builds
Sockudo with `push-apns,monolith` and trusts the mock's CA through `ca_certificate_path`. The
example also ships an iOS Simulator app with a Live Activity widget that exercises the Swift token
observers against the backend; the Simulator needs real sandbox APNs credentials to display pushed
updates, because `simctl push` cannot deliver `liveactivity` pushes.
## Capacity and failure handling
Sockudo reuses long-lived HTTP/2 connections, enables adaptive flow-control windows, and bounds
each worker's concurrent outbound requests. Direct-token fanout remains O(number of activities);
use broadcast channels for audiences that share content. Increase worker count and outbound
concurrency gradually while watching APNs 429 responses, queue lag, retry age, circuit state, and
provider latency. More sockets do not guarantee more APNs throughput.
| Condition | Behavior |
| --- | --- |
| Payload too large | The APNs device payload limit is 4096 bytes and the broadcast limit is 5120 bytes. Sockudo checks the final serialized body for the actual destination. |
| APNs `429` or `5xx` | Retried with jitter and `Retry-After`. |
| APNs `5xx` | Holds only that payload for Apple's required 15-minute delay; other Live Activity deliveries keep flowing, and repeated 5xx failures trip the normal provider circuit breaker. |
| Invalid token or channel, configuration failure, caller payload error | Terminal and observable, without leaking tokens. |
| Retried request | Reuses the same `apns-id` (direct) or `apns-request-id` (broadcast). Sockudo supplies both deterministically. |
The dedicated `sockudo_push_apns_live_activity_requests_total{mode,event,status,app}` counter
separates direct and broadcast outcomes while the common push queue, retry, provider-failure, and
latency metrics retain the full operational picture.
## Next steps
- [Push notifications](/docs/server/push-notifications): Devices, subscriptions, publish, status, and provider credentials.
- [Push operations](/docs/server/push-operations): Metrics, alerts, dead-letter replay, and APNs outage runbooks.
- [Swift client](/docs/clients/swift): The Sockudo Swift client and its Live Activity token observers.
- [Configuration reference](/docs/reference/configuration): Every push.apns key with defaults.
---
# Push operations (https://sockudo.io/docs/server/push-operations)
> Operate Sockudo push cleanup, metrics, backpressure, and provider outage runbooks.
This page is the operator's guide to a running push pipeline: what to measure, what to alert on,
how to inspect and replay dead letters, and what to do when a provider or a node fails. Read it
before your first production campaign, and keep it open during an incident. For the API and
delivery model itself, see [Push notifications](/docs/server/push-notifications).
Push operations revolve around five questions:
1. Is work being admitted?
2. Is it moving through each queue stage?
3. Are providers accepting it?
4. Can stale queued work be repaired?
5. Is old operational state being cleaned up?
## Queue stages
Accepted publishes move through a fixed set of queue stages. Metrics label each stage with a stable
`stage` value, which is what you alert on.
```mermaid
flowchart TB
A[Publish API] --> P[push.publish.v1]
P --> S[push.shards.v1]
P --> D[push.delivery.provider.v1]
S --> D
D --> R[push.results.v1]
R --> T[push.retry.v1]
T --> D
R --> X[push.deadletters.v1]
```
| Queue topic | Metric `stage` label | Consumer | Lag usually means |
| --- | --- | --- | --- |
| `push.publish.v1` | `publish_log` | Planner workers | Planner capacity or queue health. |
| `push.shards.v1` | `shard_jobs` | Shard workers | Large channel fanout. |
| `push.delivery..v1` | `delivery_jobs_` | Provider workers | Provider throughput or credentials. |
| `push.results.v1` | `delivery_results` | Feedback workers | Feedback or status-store write capacity. |
| `push.retry.v1` | `retry_schedule` | Retry scheduler workers | Provider outage or retry policy. |
| `push.deadletters.v1` | `dead_letters` | Operators | Work that needs inspection and replay. |
Small publishes go straight from the planner to provider delivery (`fastPath`); publishes at or
above `push.fanout_fast_threshold` expected recipients are split into shards first (`shardPath`).
## Backpressure
Publish admission rejects new work before a backlog becomes unbounded. When a threshold is crossed,
`POST /apps/{appId}/push/publish` returns `429` with `Retry-After`.
| Control | Default | Effect |
| --- | --- | --- |
| `PUSH_PUBLISH_LOG_MAX_LAG` | `100000` | Maximum `publish_log` depth before admission returns `429`. |
| `PUSH_CRITICAL_QUEUE_MAX_LAG` | `100000` | Maximum depth for the other critical stages. |
| `push.backpressure_lag_threshold_secs` / `PUSH_BACKPRESSURE_LAG_THRESHOLD_SECS` | `60` | Oldest ready or inflight item age tolerated. `0` relies only on depth. |
| `PUSH_BACKPRESSURE_RETRY_AFTER_SECONDS` | `5` | `Retry-After` value sent with queue backpressure. |
| `PUSH_BACKPRESSURE=true` | unset | Forces admission to return `429` (manual brake during an incident). |
Setting `PUSH_BACKPRESSURE=true` stops new admission while existing work drains and retries. It is
a faster, safer lever than scaling workers into a provider that is already throttling you.
## Retention and cleanup
The monolith push cleanup worker runs when `push.cleanup_interval_secs > 0`.
| State | Retention |
| --- | --- |
| Terminal publish statuses | `push.publish_status_ttl_days` |
| Delivery events | `push.analytics_retention_days` |
| Operator invalidation events | `push.analytics_retention_days` |
| Expired idempotency records | The record `expires_at_ms`; legacy non-epoch expiries are retained |
| Expired scheduler locks | Removed after `expires_at_ms` |
| Completed scheduled jobs | Deleted synchronously when emitted or cancelled |
| Active scheduled jobs and active retries | Retained until they run, expire, or are cancelled |
| Credentials, templates, and active device registrations | Never removed by cleanup |
Cleanup is bounded by `push.cleanup_batch_size` per category and
`push.cleanup_max_deleted_per_tick` overall. SQL stores use bounded deletes. Document stores clean
new writes through internal app and time indexes where available; families without a time index are
cleaned by bounded app-partition scans. Memory cleanup is for tests and local development only.
```toml title="config.toml (defaults)"
[push]
publish_status_ttl_days = 30
analytics_retention_days = 30
cleanup_interval_secs = 300
cleanup_batch_size = 1000
cleanup_max_deleted_per_tick = 100000
```
## Dead letters
Dead-letter queue messages remain owned by the configured queue backend. Operators can inspect
queue-native dead-letter metadata through `GET /apps/{appId}/push/deadLetters`, filter by
`provider` (`fcm`, `apns`, `webpush`, `hms`, or `wns`), `sinceMs`, and `untilMs`, and page with
`limit` (1 to 1000) plus `cursor`. Both endpoints require the `push-admin` capability.
```http
GET /apps/{appId}/push/deadLetters?provider=apns&sinceMs=1725000000000&limit=50
x-sockudo-push-capability: push-admin
```
```json title="200 OK"
{
"items": [
{
"deadLetterId": "dl_01J...",
"appId": "my-app",
"publishId": "order-ord_123-packed",
"provider": "apns",
"stage": "delivery_result",
"key": "BATCH_ID",
"reason": "PROVIDER_ERROR_CLASS",
"occurredAtMs": 1725000123456,
"replayable": true
}
],
"next_cursor": null,
"has_more": false
}
```
Responses include safe metadata only: dead-letter id, app/publish ids, provider when known, stage,
key, reason, timestamp, and `replayable`. They do not expose original push payloads, recipient
tokens, endpoints, or credential material.
Use `POST /apps/{appId}/push/deadLetters/{deadLetterId}/replay` to re-enqueue a replayable
dead-letter's original queue item after fixing the underlying incident:
```json title="202 Accepted"
{
"deadLetterId": "dl_01J...",
"replayed": true,
"messageId": "..."
}
```
Marker-only dead letters that were emitted without retained original queue payload stay
inspectable but are returned as `replayable: false`. Replaying one returns `400` with the reason;
an unknown id returns `404`.
Replay re-enqueues the original work unchanged. If the credential, topic, or provider incident is
still active, the replayed work will fail and dead-letter again.
## Metrics
Watch these metrics by app, provider, stage, or bounded category labels:
| Signal | Metrics |
| --- | --- |
| Admission | `sockudo_push_publish_accepted_total`, `sockudo_push_quota_acceptance_rejections_total` |
| Queue lag | `sockudo_push_publish_log_lag_seconds`, `sockudo_push_delivery_jobs_lag_seconds`, `sockudo_push_queue_oldest_age_seconds`, queue backend lag |
| Provider outcomes | `sockudo_push_dispatched_total`, `sockudo_push_dispatch_duration_seconds`, `sockudo_push_provider_failures_total` |
| Retries | `sockudo_push_retry_scheduled_total`, `sockudo_push_retry_attempted_total`, `sockudo_push_retry_deferred_total`, `sockudo_push_retry_expired_total`, `sockudo_push_retry_malformed_total` |
| Dead letters | `sockudo_push_retry_dead_lettered_total`, dead-letter queue depth, `GET /apps/{appId}/push/deadLetters` |
| Repair | `sockudo_push_repair_scanned_total`, `sockudo_push_repair_requeued_total`, `sockudo_push_repair_skipped_total` |
| Credentials | `sockudo_push_provider_failures_total{failure_class="credential_auth"}` |
| Device invalidation | `sockudo_push_token_invalidations_total`, `sockudo_push_token_invalidation_guard_total` |
| Status integrity | `sockudo_push_invariant_violations_total`, `sockudo_push_status_cas_conflicts_total`, `sockudo_push_status_cas_exhausted_total` |
| Live Activities | `sockudo_push_apns_live_activity_requests_total` |
| Cleanup | `sockudo_push_cleanup_scanned_total`, `sockudo_push_cleanup_deleted_total`, `sockudo_push_cleanup_errors_total`, `sockudo_push_cleanup_tick_duration_seconds` |
| Worker health | `sockudo_push_worker_exits_total` |
`sockudo_push_queue_oldest_age_seconds` carries a `stage` label (see [Queue stages](#queue-stages))
and a `state` label of `ready` or `inflight`. `failure_class` values are `device_terminal`,
`device_transient`, `provider_transient`, `provider_quota`, `credential_auth`, `caller_payload`,
`network_transport`, and `unknown`.
Do not add labels for `publish_id`, `device_id`, tokens, endpoint URLs, or raw provider reasons.
## Alerts
Start with these alert shapes and tune them against normal traffic:
| Alert | Suggested condition |
| --- | --- |
| Retry backlog age | `sockudo_push_queue_oldest_age_seconds` for `retry_schedule` ready or inflight work exceeds the retry SLO |
| Dead-letter rate | `sockudo_push_retry_dead_lettered_total` increases above baseline |
| Provider auth failures | `sockudo_push_provider_failures_total{failure_class="credential_auth"}` is nonzero |
| Invalidation spike | `sockudo_push_token_invalidation_guard_total` is nonzero or invalidation ratio jumps |
| Dispatching age | publish statuses remain `dispatching` beyond retry max elapsed plus provider timeout |
| Queue lag | critical stage depth crosses `PUSH_CRITICAL_QUEUE_MAX_LAG` or oldest actionable age crosses `PUSH_BACKPRESSURE_LAG_THRESHOLD_SECS` |
| Repair requeues | `sockudo_push_repair_requeued_total` increases outside a queue-loss or worker-crash incident |
| Status write exhaustion | `sockudo_push_status_cas_exhausted_total` increases |
| Worker exits | `sockudo_push_worker_exits_total` increases |
| Cleanup failures | `sockudo_push_cleanup_errors_total` increases |
## Runbooks
### APNs outage
1. Confirm `sockudo_push_provider_failures_total{provider="apns"}` and the APNs HTTP status family.
2. If failures are `credential_auth`, rotate or roll back the APNs key/topic/environment
configuration and restart provider workers so credentials reload.
3. If failures are provider transient or quota, keep devices intact, watch retry and dead-letter
rate, and reduce campaign admission until `push.delivery.apns.v1` lag drains.
### FCM outage
1. Separate project/auth failures from provider `5xx` or quota responses using
`sockudo_push_provider_failures_total{provider="fcm",failure_class=...}`.
2. For auth failures, fix the service account or stored credential and restart workers.
3. For a provider outage, leave device registrations in place, verify retries are scheduled, and
throttle new fanout if retry lag or publish-log lag crosses the configured backpressure
thresholds.
### Credential rotation failure
1. Stop new publishes for the affected provider if admission is still accepting work.
2. Restore the last known-good credential or upload a corrected credential, then restart provider
workers.
3. Watch `credential_auth` failures, provider worker exits, and `dispatching` publish age.
4. Replay dead letters marked `replayable` that failed during the incident, or republish from the
original source.
Do not delete devices for credential failures. Sockudo never increments device failure counts for
`credential_auth`, and purging the registry would turn a recoverable incident into permanent
audience loss.
### Dead node with pending queue work
Use the queue backend's visibility timeout and redelivery tools first. Sockudo queue adapters keep a
bounded local ready/pending handoff; if a node dies, broker redelivery depends on the selected
backend. After the replacement worker starts, verify `push.publish.v1`, `push.shards.v1`,
`push.results.v1`, `push.retry.v1`, and provider delivery stages are draining.
### Backlog or backpressure
Check which stage crossed lag first. Publish-log lag points to planner capacity or queue health.
Shard lag points to channel fanout. Provider delivery lag points to provider throughput or
credentials. Retry lag points to provider outage or retry policy. Use
`sockudo_push_queue_oldest_age_seconds` to separate a shallow but stale stage from ordinary depth
growth. Dead-letter lag means operators need to inspect `GET /apps/{appId}/push/deadLetters`, fix
the underlying provider/credential/queue incident, then replay only entries marked `replayable` or
republish from the original source.
### Publish-log repair
When `push.repair_interval_secs > 0`, the monolith repair worker scans durable publish-log entries
that are still in `queued` state after `push.repair_min_age_secs` and re-enqueues the missing
`push.publish.v1` queue item. It is meant for accepted publish work whose queue message was lost or
acknowledged by a crashing worker before planning began. It does not recreate active retry,
provider-delivery, or terminal publish work.
```toml title="config.toml (defaults)"
[push]
repair_interval_secs = 30
repair_min_age_secs = 30
repair_batch_size = 100
```
## Next steps
- [Push notifications](/docs/server/push-notifications): Delivery model, publish API, provider credentials, and device invalidation.
- [Apple Live Activities](/docs/server/apple-live-activities): ActivityKit delivery, broadcast channels, and APNs failure handling.
- [Observability](/docs/server/observability): Prometheus metrics, logging, and tracing across the server.
- [Scaling](/docs/server/scaling): Horizontal deployment and queue backends for multi-node clusters.
---
# MCP server (https://sockudo.io/docs/server/mcp)
> Let AI agents operate Sockudo through the Model Context Protocol with scoped tokens, audited tools, resources, and prompts.
Sockudo ships a Model Context Protocol (MCP) server so agents such as Claude can inspect and
operate a deployment through a typed, permissioned tool surface instead of hand-built HTTP calls.
Use it when you want an agent to debug channels, triage incidents, read history, or publish
events while you keep control over exactly what it may touch.
The protocol layer is the official [`rmcp`](https://crates.io/crates/rmcp) SDK; Sockudo adds
signed API access, scopes, tools, resources, and prompts.
## How it works
The MCP tools are a thin layer over the public [HTTP API](/docs/server/http-api). Every tool call
is turned into a signed HTTP API request and runs through the same validation, idempotency, rate
limits, metrics, feature gates, and role restrictions as any external caller. Tool results are the
documented HTTP API response shapes, so anything you know about the HTTP API applies.
```mermaid
sequenceDiagram
participant A as Agent
participant M as MCP endpoint
participant R as Sockudo API router
A->>M: tools/call with Bearer token
Note over M: Scope check, rate limit, app allow-list
M->>R: Signed HTTP API request
R-->>M: JSON response
M-->>A: Tool result and structuredContent
```
Two deployment shapes share the same code:
| Shape | Transport to Sockudo | MCP transport | Use when |
| --- | --- | --- | --- |
| Embedded (`sockudo` binary, `mcp` feature) | In-process: the server drives its own API router with self-signed requests | Streamable HTTP on `/mcp` (shared or dedicated port) | Hosted agents, shared team access, production |
| Standalone (`sockudo-mcp` binary) | Signed HTTP to a remote deployment | stdio or Streamable HTTP | Claude Desktop, Claude Code, local IDEs |
In both shapes the MCP server holds the app credentials and signs requests itself. Agents
authenticate with an MCP bearer token, and tools such as `sockudo_list_apps` return keys and
sanitized policy only.
## Enable the embedded server
1. Build with the `mcp` feature (it is also part of the `full` feature set):
```bash
cargo build -p sockudo --release --features "v2,mcp,redis,postgres"
```
2. Add an `[mcp]` section with at least one token:
```toml title="config.toml"
[mcp]
enabled = true
path = "/mcp"
# port = 6100 # optional dedicated listener (host defaults to the server host)
allowed_hosts = [] # empty = accept any Host (fine behind a trusted proxy)
allowed_origins = [] # browser origins; empty disables Origin checks
rate_limit_per_minute = 600 # per token; 0 disables
request_timeout_ms = 30000
session_ttl_seconds = 1800
[[mcp.tokens]]
name = "ops-agent"
token = "${SOCKUDO_MCP_OPS_TOKEN}" # >= 16 chars; use env interpolation
scopes = ["read", "write"]
apps = ["*"]
[[mcp.tokens]]
name = "readonly-dashboard"
token = "${SOCKUDO_MCP_RO_TOKEN}"
scopes = ["read"]
apps = ["app-1"]
```
3. Start Sockudo and send an `initialize` request to confirm the endpoint answers:
```bash
curl -sS http://127.0.0.1:6001/mcp \
-H "Authorization: Bearer $SOCKUDO_MCP_OPS_TOKEN" \
-H "Content-Type: application/json" -H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"1"}}}'
```
4. Register the endpoint with your agent (see [Connect a client](#connect-a-client)).
### Configuration reference
| Key | Default | Meaning |
| --- | --- | --- |
| `enabled` | `false` | Serve MCP. Requires the `mcp` Cargo feature. |
| `path` | `"/mcp"` | URL path of the Streamable HTTP endpoint. |
| `host` | unset | Bind host for a dedicated listener; defaults to the server host. |
| `port` | unset | Dedicated MCP port. When unset, MCP shares the main HTTP listener. |
| `allowed_hosts` | `[]` | Allowed `Host` authorities (DNS-rebinding protection). Empty accepts any host. |
| `allowed_origins` | `[]` | Allowed browser origins. Empty disables `Origin` validation. |
| `allow_anonymous` | `false` | Accept unauthenticated requests with `anonymous_scopes`. Development only. |
| `anonymous_scopes` | `["read"]` | Scopes granted to anonymous callers. |
| `tokens` | `[]` | Bearer token principals (`name`, `token`, `scopes`, `apps`). |
| `request_timeout_ms` | `30000` | Per-tool upstream timeout. |
| `max_body_bytes` | `1048576` | Maximum MCP POST body size (minimum `1024`). |
| `session_ttl_seconds` | `1800` | Idle session lifetime for session-based protocol versions. |
| `rate_limit_per_minute` | `600` | Per-token tool and resource call budget. `0` disables. |
| `disabled_tools` | `[]` | Tool names to hide entirely. |
| `instructions` | unset | Extra text appended to the MCP server instructions. |
Each `[[mcp.tokens]]` entry takes a unique `name` (used in audit logs), a `token` of at least 16
characters, `scopes` (default `["read"]`), and `apps` (default `["*"]`).
Startup validation fails when:
- `enabled = true` and there are no tokens, unless `allow_anonymous = true`.
- `path` is not an absolute path, or shadows a Sockudo API route (`/apps`, `/app`, `/up`, `/live`,
`/usage`, `/stats`, `/operator`, `/accept-traffic`, `/metrics`) while sharing the main listener.
- `port` equals the main HTTP port or the metrics port.
- A token name is empty or duplicated, a token is shorter than 16 characters, or a scope is not
one of `read`, `write`, `admin`.
If any `[[mcp.tokens]]` entry is configured, `allow_anonymous` has no effect and Sockudo logs a
warning. When it does apply, every caller gets the anonymous principal, so never enable it on a
reachable listener.
### Tokens from the environment
`MCP_TOKEN`, `MCP_TOKEN_NAME`, `MCP_TOKEN_SCOPES`, and `MCP_TOKEN_APPS` add a token without
touching the file. The name defaults to `env`, scopes to `read,write`, and apps to `*`; an existing
file token with the same name is replaced. Every other `[mcp]` key has an `MCP_*` override; see
[Environment variables](/docs/reference/environment-variables#mcp).
```bash
MCP_ENABLED=true \
MCP_TOKEN="$(openssl rand -hex 24)" \
MCP_TOKEN_NAME=ops-agent \
MCP_TOKEN_SCOPES=read,write \
MCP_TOKEN_APPS=app-1 \
sockudo --config config.toml
```
## Connect a client
Clients authenticate with `Authorization: Bearer `. A missing or invalid token is rejected
before MCP processing with HTTP `401` and a JSON body:
```json
{ "error": "unauthorized", "message": "invalid bearer token" }
```
Claude Code:
```bash
claude mcp add --transport http sockudo https://rt.example.com/mcp \
--header "Authorization: Bearer $SOCKUDO_MCP_OPS_TOKEN"
```
Claude API (MCP connector):
```json
{
"mcp_servers": [{ "type": "url", "url": "https://rt.example.com/mcp", "name": "sockudo",
"authorization_token": "" }],
"tools": [{ "type": "mcp_toolset", "mcp_server_name": "sockudo" }]
}
```
For interactive exploration, point `npx @modelcontextprotocol/inspector` at `/mcp` with the same
`Authorization` header.
## Standalone binary
`sockudo-mcp` runs outside the server and talks to a remote deployment over signed HTTP. Use it for
desktop agents and IDEs, where stdio is the natural transport.
```bash
cargo install --path crates/sockudo-mcp --features cli
# stdio for Claude Desktop / Claude Code
sockudo-mcp --url https://rt.example.com --app app-1:key:secret --scopes read,write
claude mcp add sockudo -- sockudo-mcp --url https://rt.example.com --app app-1:key:secret
# Streamable HTTP with tokens
sockudo-mcp --transport http --listen 127.0.0.1:6100 \
--token 'ops/read+write=<16+ char token>' --url https://rt.example.com --app app-1:key:secret \
--metrics-url http://rt.example.com:9601/metrics
```
| Flag | Env | Default | Meaning |
| --- | --- | --- | --- |
| `--url` | `SOCKUDO_URL` | `http://127.0.0.1:6001` | Base URL of the Sockudo HTTP API (may include a path prefix). |
| `--app` | `SOCKUDO_MCP_APPS` | none | `app_id:app_key:app_secret`; repeatable or comma-separated. |
| `--app-id`, `--app-key`, `--app-secret` | `SOCKUDO_APP_ID`, `SOCKUDO_APP_KEY`, `SOCKUDO_APP_SECRET` | none | Single-app convenience form. |
| `--transport` | `SOCKUDO_MCP_TRANSPORT` | `stdio` | `stdio` or `http`. |
| `--listen` | `SOCKUDO_MCP_LISTEN` | `127.0.0.1:6100` | Listen address for `--transport http`. |
| `--path` | `SOCKUDO_MCP_PATH` | `/mcp` | URL path for `--transport http`. |
| `--token` | `SOCKUDO_MCP_TOKENS` | none | `token`, `name=token`, or `name/scope+scope=token`; repeatable or comma-separated. |
| `--scopes` | `SOCKUDO_MCP_SCOPES` | `read,write` | Scopes for stdio and for tokens without explicit scopes. |
| `--allow-anonymous` | `SOCKUDO_MCP_ALLOW_ANONYMOUS` | off | Serve HTTP without authentication (development only). |
| `--metrics-url` | `SOCKUDO_METRICS_URL` | none | Prometheus URL; enables `sockudo_server_metrics`. |
| `--timeout-ms` | `SOCKUDO_MCP_TIMEOUT_MS` | `30000` | Per-request timeout. |
| `--allowed-hosts` | `SOCKUDO_MCP_ALLOWED_HOSTS` | empty | Allowed `Host` authorities for HTTP. |
| `--allowed-origins` | `SOCKUDO_MCP_ALLOWED_ORIGINS` | empty | Allowed browser origins for HTTP. |
| `--disable-tool` | `SOCKUDO_MCP_DISABLED_TOOLS` | none | Tools to hide; repeatable or comma-separated. |
| `--instructions` | `SOCKUDO_MCP_INSTRUCTIONS` | none | Extra text appended to the server instructions. |
Over stdio, stdout is the protocol channel. `sockudo-mcp` writes logs to stderr, so wrap it in
scripts that do not print to stdout.
## Scopes and safety
Scopes are cumulative: `admin` implies `write`, and `write` implies `read`.
| Scope | Grants | Examples |
| --- | --- | --- |
| `read` | Inspection | `sockudo_list_channels`, `sockudo_get_history`, `sockudo_server_stats`, `sockudo_server_metrics` |
| `write` | Publishing and mutation (implies `read`) | `sockudo_trigger_event`, `sockudo_update_message`, `sockudo_publish_annotation`, `sockudo_push_publish`, `sockudo_sign_channel_auth` |
| `admin` | Destructive and connection-affecting operations (implies `write`) | `sockudo_terminate_user_connections`, `sockudo_reset_history`, `sockudo_purge_history`, `sockudo_revoke_capability_tokens`, `sockudo_replay_push_dead_letter` |
- `tools/list` only returns tools the token may call. Calling a tool above the token's scope
returns JSON-RPC error `-32003` with `reason: "missing_scope"` and `required_scope`.
- Tokens carry an app allow-list; requests for other apps fail with `-32003` and
`reason: "app_not_allowed"`.
- When a token exceeds `rate_limit_per_minute`, calls fail with `-32004` and
`retry_after_seconds`. The embedded limiter is in-memory, so each node enforces its own budget.
- App secrets never leave the server. `sockudo_list_apps` and `sockudo_get_app` return keys and
sanitized policy only; webhook headers are redacted.
- Every tool call emits an audit log line on target `sockudo_mcp::audit` with principal, tool, app,
channel, outcome, and latency.
Every `admin` tool requires `confirm: true`; calls without it fail argument validation before any
request is made. `sockudo_reset_history`, `sockudo_purge_history`, and
`sockudo_reset_presence_history` also require a `reason`, which is forwarded to the server.
`sockudo_revoke_capability_tokens` accepts an optional `reason` recorded with the revocation.
```json title="tools/call: reset a history stream"
{
"jsonrpc": "2.0",
"id": 7,
"method": "tools/call",
"params": {
"name": "sockudo_reset_history",
"arguments": {
"app_id": "app-1",
"channel": "chat:room-1",
"reason": "stream marked reset_required after storage incident",
"confirm": true
}
}
}
```
`admin` tools disconnect users, rotate history streams, and delete retained data. Give agents
`read` or `read` plus `write` by default, and hide individual tools you never want exposed with
`disabled_tools`.
### Errors an agent can act on
Protocol-level problems (bad arguments, missing scope, rate limits) are JSON-RPC errors. Failures
from Sockudo itself come back as tool results with `isError: true` and a machine-readable envelope,
so the agent can read the server's `code` and `error` and recover:
```json title="isError result for an upstream 4xx"
{
"error": "upstream_error",
"message": "",
"details": {
"http_status": 404,
"upstream": { "...": "the server's JSON error body, unchanged" }
}
}
```
Other envelope `error` values are `unknown_app`, `missing_app_id`, `transport_error`, `timeout`,
`invalid_request`, and `internal_error`.
## Tools
| Area | Tools |
| --- | --- |
| Discovery and server | `sockudo_list_apps`, `sockudo_get_app`, `sockudo_server_info`, `sockudo_server_health`, `sockudo_server_accept_traffic`, `sockudo_server_stats`, `sockudo_server_usage`, `sockudo_server_metrics` |
| Channels | `sockudo_list_channels`, `sockudo_get_channel`, `sockudo_get_presence_users` |
| Publish | `sockudo_trigger_event`, `sockudo_trigger_batch_events` |
| Durable history | `sockudo_get_history`, `sockudo_get_history_state`, `sockudo_reset_history`, `sockudo_purge_history` |
| Versioned messages | `sockudo_get_message`, `sockudo_list_message_versions`, `sockudo_update_message`, `sockudo_delete_message`, `sockudo_append_message` |
| Annotations | `sockudo_list_annotations`, `sockudo_publish_annotation`, `sockudo_delete_annotation` |
| Presence history | `sockudo_get_presence_history`, `sockudo_get_presence_history_state`, `sockudo_get_presence_snapshot`, `sockudo_reset_presence_history` |
| Connections and tokens | `sockudo_terminate_user_connections`, `sockudo_force_reconnect_user`, `sockudo_revoke_capability_tokens` |
| Push | `sockudo_push_publish`, `sockudo_push_batch_publish`, `sockudo_push_publish_status`, `sockudo_list_push_devices`, `sockudo_get_push_device`, `sockudo_list_push_channel_subscriptions`, `sockudo_list_push_subscription_channels`, `sockudo_list_push_dead_letters`, `sockudo_replay_push_dead_letter`, `sockudo_delete_push_scheduled_job`, `sockudo_list_push_credentials`, `sockudo_list_push_templates`, `sockudo_get_push_template` |
| Auth helpers | `sockudo_sign_channel_auth`, `sockudo_sign_user_auth`, `sockudo_verify_webhook_signature` |
Start an agent session with `sockudo_list_apps` to discover valid `app_id` values, then
`sockudo_server_info`, which reports the server version, transport, tool count, and the caller's
own scopes and apps:
```json title="sockudo_server_info result (abridged)"
{
"mcp_server": {
"name": "sockudo",
"version": "5.2.0",
"transport": "in-process",
"tools": 48,
"request_timeout_ms": 30000,
"rate_limited": true
},
"principal": { "name": "ops-agent", "scopes": ["read", "write"], "apps": "*" }
}
```
Hide tools with `disabled_tools = ["sockudo_purge_history"]`. Tools that hit a feature the server
does not enable (for example push without the `push` feature) return the server's
`feature_disabled` or `404` error rather than failing at startup. `sockudo_server_stats` and
`sockudo_server_usage` require `http_api.usage_enabled` on the server.
## Resources and prompts
Resources use the `sockudo://` scheme:
| Resource | Contents |
| --- | --- |
| `sockudo://server/info`, `server/health`, `server/stats` | Server description, health check, operator stats |
| `sockudo://apps`, `apps/{app_id}` | Apps visible to the token and one app's sanitized policy |
| `sockudo://apps/{app_id}/channels` | Occupied channels for an app |
| `sockudo://apps/{app_id}/channels/{channel}` | One channel, with `/history`, `/presence`, and `/messages/{message_serial}` sub-resources |
| `sockudo://docs/http-api`, `docs/channels`, `docs/operations` | Embedded reference docs for the agent |
Prompts `sockudo_debug_channel`, `sockudo_incident_triage`, `sockudo_design_realtime_feature`, and
`sockudo_audit_app_security` walk an agent through common workflows. Argument completion is
available for `app_id` and `channel`.
## Observability
Prometheus families (with the configured prefix):
| Metric | Labels | Meaning |
| --- | --- | --- |
| `mcp_requests_total` | `port`, `outcome` | Protocol requests by outcome (`tools_call`, `resources_read`, `rate_limited`, `unauthorized`, ...). |
| `mcp_tool_calls_total` | `port`, `tool`, `outcome` | Tool calls by tool and outcome (`ok`, `upstream_4xx`, `forbidden_scope`, `timeout`, ...). |
| `mcp_tool_latency_ms` | `port`, `tool` | Tool execution latency. |
Tool outcomes are `ok`, `invalid_arguments`, `forbidden_scope`, `forbidden_app`, `unauthorized`,
`upstream_4xx`, `upstream_5xx`, `timeout`, `unknown_app`, `transport_error`, `internal_error`, and
`unknown_tool`. A rising `forbidden_scope` count usually means an agent is trying tools above its
grant; a rising `upstream_4xx` count usually means bad arguments the agent is not correcting.
## Testing
- `cargo test -p sockudo-mcp --all-features` runs protocol, signing, catalog, and Streamable HTTP
transport tests; `cargo test -p sockudo --features mcp mcp::` drives the in-process transport
through the real API middleware.
- `make mcp-smoke` builds both binaries, starts a scratch server from `tests/mcp/config.toml`, and
runs `tests/mcp/smoke_http.py` (embedded endpoint, admin and read-only tokens) and
`tests/mcp/smoke_stdio.py` (standalone binary over stdio).
- For interactive checks use `npx @modelcontextprotocol/inspector` against `/mcp` with the
`Authorization: Bearer` header, or register the endpoint in Claude Code and call
`sockudo_server_info`.
## Performance notes
- The embedded transport never touches a socket: requests are signed and dispatched to the router
with `tower::Service::oneshot`. Cost per call is one HMAC-SHA256, one MD5 for POST bodies, and the
normal handler work.
- `tools/list` responses are prebuilt once per scope combination and served as an `Arc` clone.
- Responses are passed through as the server's JSON bytes; only object bodies are parsed once more
to populate `structuredContent`.
- Sessions are managed by `rmcp` with idle expiry (`session_ttl_seconds`); the `2026-07-28`
protocol revision is served statelessly.
## Next steps
- [HTTP API](/docs/server/http-api): The request and response shapes every MCP tool returns.
- [Security](/docs/server/security): Harden listeners, tokens, and app credentials before exposing MCP.
- [Observability](/docs/server/observability): Scrape mcp_* metrics and route the audit log.
- [Environment variables](/docs/reference/environment-variables#mcp): Every MCP_* override for containerized deployments.
---
# AI Transport overview (https://sockudo.io/docs/server/ai-transport-overview)
> How Sockudo maps AI Transport sessions onto existing V2 realtime, history, recovery, push, and presence primitives.
AI Transport turns a streamed LLM answer into durable, ordered channel state. Instead of tying a
response to one HTTP request, the agent writes every token into a Sockudo channel, and any device
can follow along live, reconnect mid-answer, or reload the finished transcript later. Read this page
first if you are building a chat UI, a copilot, or a long-running agent and you want answers that
survive refreshes, flaky networks, and device switches.
AI Transport is a Protocol V2 convention layer, not a separate product. A session is a channel,
every run is normal channel traffic, and the server supplies durable, ordered, recoverable
primitives that SDKs reduce into UI state.
Sockudo AI Transport is heavily inspired by [Ably AI Transport](https://ably.com/docs/ai-transport).
Ably's public AI Transport docs and SDK were the main product inspiration; Sockudo adapts that
model to Protocol V2, versioned messages, durable history, presence, push, and the local
`@sockudo/client` SDK. Sockudo AI Transport is community-built and community-maintained. It is not
an Ably product and is not supported by Ably.
## Why durable streaming
A plain streaming endpoint (`fetch` plus server-sent events) works until something interrupts it.
Close the tab, lose Wi-Fi in a lift, or open the same chat on your phone, and the half-finished
answer is gone: the tokens only ever existed inside one HTTP response.
AI Transport moves the answer out of the response and into a channel:
- **Refresh-safe.** The answer lives in durable history, so a reloaded page reads the same
transcript.
- **Multi-device.** Every subscriber of the session channel sees the same tokens, in the same
order.
- **Resumable.** A client that drops mid-answer resumes from its last serial and receives only what
it missed.
- **Observable.** Run starts, ends, cancellations, and orphaned streams surface as webhooks and
Prometheus metrics.
The HTTP request still exists, but it only starts the work. It returns `202 Accepted` and the answer
flows through the channel.
## Enable AI Transport
1. Build Sockudo with the `ai-transport` Cargo feature (it is not in the default feature set):
```bash
cargo build -p sockudo --features "v2,ai-transport,redis,postgres"
```
2. Enable durable history and versioned messages. Startup rejects `ai_transport.enabled = true`
unless both `[history].enabled` and `[versioned_messages].enabled` are true.
3. Turn on AI Transport and scope it to channel prefixes:
```toml title="config.toml"
[ai_transport]
enabled = true
[[ai_transport.channels]]
prefix = "private-ai-"
```
4. Name every AI session channel so it starts with a configured prefix, for example
`private-ai-user-42-sess-01J`. Prefix matching is a plain string prefix: `private-ai:user-42`
does **not** match `private-ai-`.
AI validation applies only to channels that match an `[[ai_transport.channels]]` prefix. Publishing
an AI event name (or `extras.ai` metadata) to any other channel fails with `104003`
`ai_event_not_permitted` and the message `AI Transport is not enabled for this channel`.
## Mental model
Four ideas cover almost everything:
| Concept | What it is | In Sockudo terms |
| --- | --- | --- |
| Session | One conversation between a user and an agent. | One private or presence channel. |
| Run | One user turn plus the agent's reply to it, identified by `run-id`. | A sequence of ordinary channel events that share `extras.ai.transport.run-id`. |
| Output message | The assistant's reply, which grows token by token. | One mutable message: a `message.create` followed by `message.append` operations. |
| Transcript | What the user sees after reload. | Durable history with every mutable message substituted by its latest aggregate. |
A run moves through a small lifecycle. The client sends input, the agent announces the run, streams
output, and closes the run with a reason:
```mermaid
sequenceDiagram
participant U as User client
participant B as Your backend
participant S as Sockudo
participant A as Agent worker
participant H as History store
U->>S: publish ai-input (run-id, invocation-id)
S->>H: persist input
U->>B: POST /api/chat (runId, sessionName)
B-->>U: 202 Accepted
B->>A: start run
A->>S: ai-run-start
A->>S: create ai-output (status=streaming)
A->>S: message.append per model chunk
S->>H: store every append
S-->>U: rolled-up appends
A->>S: final append (status=complete)
A->>S: ai-run-end (run-reason=complete)
S-->>U: terminal state and run end
```
Your backend and the agent worker are often the same process: the `/api/chat` route accepts the
request, returns `202`, and keeps streaming in the background. Ownership is split deliberately:
- **Clients** may publish only `ai-input` and `ai-cancel`, and only with a verified identity.
- **Agents** publish `ai-output`, the run lifecycle events, and the step events through trusted
app-key HTTP. A WebSocket client that publishes an agent event receives `104003` with the message
`agent AI events require trusted app-key publish`.
### Why resume works
Deliveries on a durable V2 channel carry a `stream_id` and a monotonic `serial`. A reconnecting
client sends its last position, and Sockudo replays from the hot replay buffer or, if that has expired,
from durable history (reported as `hot` or `cold` recovery). When Sockudo cannot prove the range is
contiguous, it fails closed with `stream_reset` or `position_expired` rather than delivering a
transcript with a silent hole. A late joiner can also request history with `until_attach: true`,
which returns everything at or below its subscription attach serial, so history plus live delivery
forms one gap-free sequence.
### Why every original mutation is kept
The version store and durable history record every `message.append` exactly as received. Live
WebSocket delivery may coalesce a burst of appends into one frame
([append rollup](/docs/server/token-streaming-rollup)), but that is the only place coalescing
happens. Persistence, history, recovery, webhooks, push, billing, and rate limits all see the
original operations. Keeping storage unrolled means a client that reduces the full log and a client
that received rolled-up frames always arrive at the same final message.
## The primitives
AI Transport reuses Sockudo subsystems rather than adding AI-only storage:
| Primitive | Plain-language role | Sockudo subsystem |
| --- | --- | --- |
| Session | The conversation container. | One channel, usually private or presence, for a user/agent conversation. |
| Input | What the user said. | `ai-input` channel event from a verified user-capable connection or trusted server. |
| Output | What the agent is saying, as it grows. | `ai-output` mutable message created and appended by the agent/server side. |
| Run lifecycle | When a reply starts, pauses, resumes, or ends. | `ai-run-start`, `ai-run-suspend`, `ai-run-resume`, `ai-run-end`, and `ai-cancel` events with bounded `extras.ai.transport` headers. |
| Mutable state | Editing a message without losing its history. | Existing versioned messages: create, update, delete, append, summary. |
| Replay | Catching up after a disconnect or late join. | Durable history, rewind, `until_attach`, and hot-to-durable recovery. |
| Identity | Who is allowed to say what. | V2 capability tokens, signed channel auth, and server app-key HTTP auth. |
| Agent state | Whether the agent is idle, thinking, or streaming. | Presence channels plus V2 `sockudo:presence_update`. |
| Notifications | Telling a user who left that the answer is ready. | Existing push device/channel-subscription pipeline and optional `[[push_rules]]`. |
### Event names
| Event | Published by | Purpose |
| --- | --- | --- |
| `ai-input` | Client or trusted server | User input for a run. |
| `ai-cancel` | Client or trusted server | Ask the agent to stop one or more runs. |
| `ai-output` | Agent (app key) | The assistant's mutable output message. |
| `ai-run-start` | Agent (app key) | A run began. |
| `ai-run-suspend` | Agent (app key) | A run paused without a terminal outcome; it can continue later. |
| `ai-run-resume` | Agent (app key) | A suspended run continues under a new `invocation-id`. |
| `ai-run-end` | Agent (app key) | A run finished with `run-reason` `complete`, `cancelled`, or `error`. |
| `ai-step-start`, `ai-step-end` | Agent (app key) | Open and close a re-attemptable step inside a run. |
Legacy `ai-turn-start` and `ai-turn-end` are still accepted as inbound aliases for migration; new
code uses the run vocabulary. The full header registry is in
[AI Transport conventions](/docs/server/ai-transport-conventions).
## What already exists
These features are not new AI-only subsystems:
- `MessageAction` and `VersionStore` for mutable messages.
- `HistoryStore`, rewind, and two-tier recovery.
- Presence and presence-history.
- Annotation publish/delete/summary projection.
- Push providers, device registrations, channel subscriptions, publish status, queues, retries,
feedback, and scheduler.
- Protocol V2 capability tokens and revocation.
- V1 isolation: V1 clients keep Pusher-compatible output.
## What `ai-transport` adds
The feature adds validation and operational conventions:
- well-known AI event names
- bounded `extras.ai.transport` and `extras.ai.codec`
- anti-spoof checks for `*-client-id` headers
- stream lifecycle tracking and active-stream limits
- append rollup as WebSocket egress optimization
- orphan stream cancellation
- conformance, benchmark, load, and chaos suites
The formal wire reference is the
[AI Transport wire protocol](https://github.com/sockudo/sockudo/blob/master/docs/specs/ai-transport-wire-protocol.md)
specification in the repository.
## Build your first session
The JavaScript SDK `@sockudo/ai-transport` handles run IDs, optimistic UI, history decoding, and
the append codec for you. Install it next to the realtime client:
```bash
pnpm add @sockudo/client @sockudo/ai-transport ai @ai-sdk/vue
```
### Agent route
The agent side creates a session for the channel, registers the run the client announced, and pipes
the model stream into it.
```ts title="app/api/chat/route.ts"
// App code: a ClientLike whose publishes and mutations use app-key HTTP.
// App code: resolves the signed-in user and checks channel ownership.
export async function POST(req: Request) {
const body = await req.json();
const channelName: string = body.sessionName;
const userId = await requireSessionOwner(req, channelName);
const session = createAgentSession({ client: realtimeClient(), channelName });
const run = session.createRun({
runId: body.runId,
invocationId: body.invocationId,
inputEventId: body.inputEventId,
clientId: userId,
onCancel(request) {
return request.runOwners.get(body.runId) === userId;
},
});
const work = (async () => {
await run.start();
const result = streamText({
model: "openai/gpt-5-mini",
instructions: "You are a concise incident-room assistant.",
prompt: latestText(body.messages),
abortSignal: run.abortSignal,
});
await run.streamResponse(toUIMessageStream({ stream: result.stream }));
await run.end("complete");
})().finally(() => session.close());
void work;
return new Response(null, { status: 202 });
}
function latestText(messages: { parts?: { text?: string }[] }[] = []): string {
return messages.at(-1)?.parts?.map((part) => part.text ?? "").join("") ?? "";
}
```
The client SDK posts `sessionName` (the channel name), `runId`, `invocationId`, `inputEventId`,
`clientId`, and the UI `messages` to `/api/chat`. Treat every field as untrusted input: derive the
user from your own session cookie or token and confirm that user owns `sessionName` before starting
work.
Agent events (`ai-output`, `ai-run-*`, `ai-step-*`) are rejected on WebSocket connections. The
`client` you pass to `createAgentSession` must route `publish`, `appendMessage`, `updateMessage`,
`deleteMessage`, and `history` through signed app-key HTTP. The Nuxt demo's `realtimeClient()` in
`client-sdks/sockudo-ai-transport-js/demo/server/utils/sockudo.ts` is a complete reference adapter
built on `adaptSockudoClient`.
On serverless platforms, keep the background work alive past the response (for example with the
platform's `waitUntil`), or run the agent in a long-lived worker.
### Browser client
The browser connects with Protocol V2, authenticates with a short-lived capability token so its
identity is verified, and hands the session to AI SDK `useChat`.
```ts title="src/chat.ts"
const realtime = await createSockudoRealtimeClient("app-key", {
clientOptions: {
wsHost: "realtime.example.com",
forceTLS: true,
channelAuthorization: { endpoint: "/api/sockudo/channel-auth" },
authCallback: async ({ reason }) => {
const res = await fetch("/api/sockudo/token", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ reason }),
});
return res.json(); // { token, exp }
},
},
});
const channelName = "private-ai-user-42-sess-01J";
const provider = provideChatTransport({
api: "/api/chat",
channelName,
client: realtime,
clientId: "user-42",
});
const { sendMessage } = useChat({
id: channelName,
transport: provider.chatTransport.value,
});
await sendMessage({ text: "Investigate the last failed deploy." });
```
`createSockudoRealtimeClient` always connects with `protocolVersion: 2` and rejects any other
explicit version. To wrap a `Sockudo` instance you already created, pass it to
`adaptSockudoClient` instead. React and Svelte bindings live at
`@sockudo/ai-transport/vercel/react` and `@sockudo/ai-transport/vercel/svelte`. See
[Authentication for AI Transport](/docs/server/ai-transport-authentication) for the token endpoint.
## Session channel model
Use channel names that encode tenant/user/session ownership in your backend, and start them with a
configured AI prefix:
```text
private-ai-tenant-7-user-42-sess-01J...
presence-ai-tenant-7-sess-01J...
```
Add a `presence-ai-` prefix to `[[ai_transport.channels]]` if AI events will flow on presence
channels. Colon-delimited names such as `private-ai:tenant-7:...` are only valid when the app
defines a matching [channel namespace](/docs/reference/configuration); without one, Sockudo
rejects the name with `Unknown channel namespace`.
Clients should never assert `client_id` or tenant identity in message bodies. Derive channel access
from signed auth or capability-token claims, then scope `subscribe`, `publish`, `history`,
`message_append_own`, and related capabilities to the exact channel or a narrow wildcard such as
`private-ai-user-42-*`.
## AI SDK agents and durability
Sockudo durability and AI SDK agent durability solve different recovery problems. Use both when you
run long-lived agents.
| Layer | Owns | Recovers from |
| --- | --- | --- |
| Sockudo AI Transport | Published user inputs, assistant output chunks, approval responses, run lifecycle, and cursors. | Browser refresh, mobile handoff, reconnect, late join, transcript replay, and audit workflows. |
| AI SDK `WorkflowAgent` or agent store | Pending agent steps, sleeps, retries, tool execution state, model-local context, and checkpoints. | Worker crash, deployment restart, long-running wait states, and agent workflow resume. |
| Model provider runtime | Provider-native realtime sessions, reasoning controls, uploaded files, and provider tool context. | Provider-specific retries, session resume, or media reconnect semantics. |
Sockudo history is intentionally not a workflow checkpoint store. Persist the AI SDK workflow run id
or agent checkpoint id in run metadata, then correlate it with `channelName`, `runId`, and
`invocationId`. When a workflow resumes, it can reload its own checkpoint and continue appending to
the same Sockudo session channel; clients recover the visible transcript from Sockudo.
## Compatibility
AI Transport is V2-only. V1 connections can still coexist on the same server, but they do not see
V2 extras, mutable-message fields, recovery frames, annotations, or AI validation semantics.
## Next steps
- [Conventions](/docs/server/ai-transport-conventions): Event names, extras headers, idempotency, and the deployment matrix.
- [Providers](/docs/server/ai-transport-providers): Stream from Vercel AI SDK, OpenAI, Anthropic, or a custom model gateway.
- [Authentication](/docs/server/ai-transport-authentication): Issue capability tokens and scope who can read, write, and append.
- [Token streaming and rollup](/docs/server/token-streaming-rollup): How high-rate appends are coalesced on egress without losing data.
---
# Ably REST, WebSocket, and AI Transport compatibility (https://sockudo.io/docs/server/ably-ai-transport-compatibility)
> Sockudo's evidence-backed Ably REST and WebSocket compatibility surface, excluding Live Objects.
Sockudo can expose an Ably-compatible REST and WebSocket surface when the server
is built with Cargo feature `ably-compat`. The evidence-backed scope is **Ably
REST and WebSocket compatibility, excluding Live Objects**. It also supports
Ably AI Transport clients through Sockudo's native mutable-message, history,
recovery, presence, annotation, stats, and push services.
Use this page when you want existing `ably-js`, `ably-go`, or
`@ably/ai-transport` clients to talk to a Sockudo server, or when you need to know
exactly which behaviors are covered by test evidence. The compatibility layer is
a translation facade: it decodes Ably frames and REST requests and calls the same
native services that Protocol V2 clients use, so there is no second copy of
message, presence, or history state.
Sockudo and this compatibility layer are community-built and
community-maintained. They are not Ably products, and Ably does not provide
support for them. Use the Sockudo community and project support channels for
help.
Use **Ably REST and WebSocket compatibility, excluding Live Objects** in public
material. Do not describe this as full Ably compatibility.
## How the facade fits
```mermaid
flowchart LR
A[Ably SDK client] --> F[Ably compat facade]
F --> M[Native message service]
F --> P[Native presence service]
M --> V[Version store]
M --> H[Durable history]
M --> N[Native fanout]
```
The facade lives in the optional `sockudo-ably-compat` workspace crate. The
server constructs one `AblyCompatRuntime` per server instance and merges its
routes. Socket writers and attachment state remain local to that process;
authorization, session ownership, recovery, and durable channel state use the
shared authorities described below. Ably realtime is WebSocket-only: Comet,
XHR polling or streaming, SSE, long polling, and other fallback transports are
intentionally unsupported.
The `ably-compat` Cargo feature also enables the `ai-transport` and `push`
features. Runtime options live under `[ably_compat]`; see the
[configuration reference](/docs/reference/configuration).
## Evidence status
The pinned manifest classifies 41 target files. Exactly the two Live Objects
files and the multiple/non-WebSocket transport file are excluded, and there are
no per-test exclusions. Node defaults, browser execution, strict execution of
upstream-pending bodies, and AI Transport have separate reports and pass/fail
states. A result from one lane is never used to claim another lane passed.
Pull requests also run a moving latest-upstream gate. It resolves and records
the current `main` commits of `ably-js`, `ably-go`, and
`ably-ai-transport-js`. It runs ably-js over WebSocket, the official ably-go
unit and JSON/MsgPack integration suites, and the complete AI Transport unit
and integration suites. The structurally defined exceptions are the two Live
Objects files, the multi-transport file, the Comet inventory assertion, and two
local-routing TLS/port assertions. The Go and AI Transport suites have no test
exclusions. A second gate installs the newest published `ably`,
`@ably/pubsub-device`, `@ably/pubsub-server`, and `@ably/ai-transport` packages
and runs the local smoke harness against them.
| Lane | Pinned source-build result |
| --- | --- |
| Node defaults | 575/575 |
| Strict completeness | 250/250 |
| Chromium defaults | 574/574, no browser boundary errors |
| Chromium strict | 250/250 |
| AI Transport | 50/50 |
A new published Sockudo tag must still pass the released-binary workflow before
that tag is promoted as verified.
## Compatibility matrix
| Surface | Status | Scope |
| --- | --- | --- |
| Ably Realtime WebSocket JSON/MsgPack protocol | Node, Chromium, and Go source evidence green | Root WebSocket endpoint accepts Ably `ProtocolMessage` connect, attach, detach, message, presence, auth, heartbeat, ACK, and NACK flows covered by pinned `ably@2.21.0` and ably-go JSON/MsgPack lanes. Non-WebSocket transports are intentionally unsupported. |
| Ably REST time/token/history/message reads and publish | Node, Chromium, and Go source evidence green | `/time`, `/keys/{keyName}/requestToken`, `/keys/{keyName}/revokeTokens`, `/messages` batch publish, `/channels/{channel}/messages`, the Go SDK's `/channels/{channel}/history` alias, message lookup, version lookup, and presence reads/history exist for the selected SDK path. Full Ably REST API parity is not claimed. |
| Run lifecycle vocabulary | Supported | Native AI Transport docs and fixtures use `ai-run-start`, `ai-run-suspend`, `ai-run-resume`, `ai-run-end`, and `ai-cancel`. |
| Legacy turn vocabulary | Legacy alias | `ai-turn-start`, `ai-turn-end`, `turn-id`, and `turn-reason` remain accepted for migration, but new docs and demos use run vocabulary. |
| Mutable messages and output streaming | Pinned Node and Chromium evidence green | Ably `publish`, `appendMessage`, `updateMessage`, `deleteMessage`, `getMessage`, `getMessageVersions`, and history map to Sockudo versioned messages. |
| Message annotations | Supported for the pinned annotation suite | Realtime action `21`, REST create/delete/list, summaries, pagination, JSON, and MsgPack translate onto Sockudo's native annotation store and projection pipeline. Protocol V1 remains unchanged. |
Compatibility projections read a shared commit-time message envelope. This
keeps IDs, publisher identity, timestamps, encoding chains, supported extras,
history/recovery positions, and version operation metadata consistent across
live delivery and REST reads while leaving Protocol V1's projection unchanged.
## Distributed state authority
Clustered compatibility requires Redis or Redis Cluster for coordination and a
non-memory durable history/version driver. Sockudo does not use a process-local
`DashMap` as cross-node authority and does not fall back from an unavailable
coordination backend to private memory.
| State | Class | Authority and retention | Backend failure behavior |
| --- | --- | --- | --- |
| App keys and configured rotated keys | Durable configuration | `AppManager` plus the deployment's versioned key configuration; every node must load the same generation. | Unknown or stale generations reject authentication. Secrets are never logged. |
| Opaque tokens | Replicated ephemeral | Shared cache record for the token TTL. A process-local decoded copy is only an optimization after the shared record is validated. | Token resolution fails closed when shared state cannot be read. |
| Nonce replay and token revocations | Replicated ephemeral security state | Atomic shared-cache claims. Revocations live for at least the maximum accepted token TTL. | Replay/revocation checks fail closed. Revocation polling disconnects sockets on every node; renewal changes the shared authorization generation so stale timers cannot restore access. |
| WebSocket writer, auth timer, channel gate, negotiated attachment | Local connection state | The node owning the socket. Gates and queues have count and byte bounds and end with the transport. | The socket closes when safe local delivery cannot continue; another node never fabricates ownership. |
| Session owner and recovery key | Replicated ephemeral | Compare-and-swap owner lease in shared cache, bounded by the connection-state TTL. Recovery keys are single-use and key rotation invalidates the previous key. | Resume fails closed when the owner lease or continuity record cannot be verified. A successful claim on another node supersedes the old owner. |
| Publish idempotency claim and acknowledgement | Replicated ephemeral plus durable repair | Atomic cache claim/receipt for the idempotency TTL; canonical history/version records repair an ambiguous post-commit receipt. | A backend error does not permit a second winner. A retry is acknowledged only after its canonical commit is proven. |
| Current presence | Replicated ephemeral | Per-connection typed membership, replicated by the horizontal adapter and tied to the node-health lease. Abrupt transport loss starts a bounded 15-second removal lease, independently of the two-minute connection-state lease. | A recovered live owner cancels pending removal. Duplicate leave/dead-node notifications are idempotent. Node death removes every orphan connection; failure to coordinate cleanup is degraded and observable rather than reported as a clean snapshot. |
| Presence transitions | Durable | Configured durable history backend and its retention policy. | Current membership can remain available, but continuity-sensitive history fails closed while degraded. |
| Attachment high-water and reconnect continuity | Local attachment plus durable canonical position | The live attachment owner captures the high-water; durable history/version streams prove reconnect ranges. | `untilAttach` requires routing affinity to the attachment owner. Resume/recover returns reset-required when stream identity or a contiguous range cannot be proven. |
| Remote delivery suppression | Local bounded projection state | Each subscriber node keeps a contiguous per-stream delivery high-water plus a bounded set of out-of-order serials; Redis fanout carries the complete canonical envelope. | Serials at or below proven continuity remain suppressed without retaining an unbounded ledger. Missing canonical identity is not guessed; reorder-window overflow is counted and fails reset-required instead of evicting duplicate evidence. |
| Mutable versions and delivery serials | Durable | Configured `VersionStore`; create, predecessor compare-and-apply, operation receipt, and serial allocation are one backend commit. | Mutation and recovery fail closed on conflicts, gaps, or backend errors. |
| Annotation serials, create receipts, event stream, and summaries | Durable | The configured non-memory version-storage driver owns atomic per-channel serial allocation, stable create IDs, canonical events, materialized summaries, and bounded retention. | Annotation writes and rebuilds fail closed; no node-local projection is used in a cluster. |
| Stats buckets | Replicated retained aggregate | Bounded local ingestion workers merge minute buckets into shared cache with TTL. | Queue drops and backend failures are counted. Queries cannot treat an unflushed or unreadable bucket as a complete cluster barrier. |
| Push registrations, subscriptions, statuses, publish log, feedback, and schedules | Durable | Configured push storage and queue; conditional status/idempotency/worker claims use backend operations. | Production rejects accidental memory drivers. Failed provider/queue/storage work remains observable through durable status and backlog metrics. |
Memory cache, history, version, annotation, push-storage, and push-queue
implementations are single-node development choices only. Clustered startup
rejects combinations that would split an authority between processes.
Mutable REST and realtime operations call the native typed mutation service.
That service atomically validates the predecessor, owns the delivery position,
commits the aggregate and individual version, enforces stream limits, and stores
the operation receipt in the configured `VersionStore`. The compatibility layer
only decodes/encodes Ably JSON or MsgPack shapes. It has no parallel mutable
state, counter, or handler-to-handler call.
## Protocol behavior
The sections below describe how the facade handles specific Ably protocol
surfaces on top of those shared authorities.
### Publish identity and retries
Ably messages may omit `name`, `data`, or both; the facade does not manufacture
an event name. The commit envelope retains the original encoding chain,
`headers`, `ref`, `ai`, and `push` extras, message ID, authenticated publisher
client ID, originating connection ID, and server publish timestamp. Live
delivery, durable history, rewind, recovery, message reads, and mutation
projections all read those same commit-time facts.
The facade also preserves AI Transport codec metadata under `extras.ai`,
including `type`, nested `fields`, `stream`, and `ends`. These values are opaque
to Sockudo; native `transport` and `codec` headers retain their existing
validation. `extras.ai.codec` may also be the string codec id that Ably AI
Transport stamps on every message; it is relayed verbatim so Ably decoders can
match it. `extras.headers` accepts string, number, boolean, and `null` values,
as Ably does; `null` is returned as `null` on JSON and MessagePack deliveries
and is omitted from Protobuf deliveries, which have no null header value.
`extras.ephemeral` accepts a boolean and is returned on live deliveries. With
ephemeral handling enabled, ephemeral messages bypass durable history, replay
storage, and webhooks through the native publish pipeline.
Live append delivery selects one projection per attachment: subscribers with a
known message base receive the append fragment; new subscribers receive a full
`message.update` before subsequent appends. Base tracking is bounded; after a
base is evicted or a mutation is filtered out, the next append supplies a full
update again. Matching subscribers share the same encoded projection.
`X-Ably-ClientId` is decoded as strict standard-base64 UTF-8, matching the Ably
SDK request format. A query/header identity mismatch, fixed-token mismatch,
message identity mismatch, or mutation-operation identity mismatch is rejected
before persistence. REST `message.connectionKey` is accepted only when it
identifies a currently live connection in the same app; the target identity is
then used for authorization and delivery stamping. `connectionId` remains a
server-assigned output field.
`message.id` uses Sockudo's shared publish-idempotency coordinator. The first
node atomically claims the app/channel/id tuple, commits through the normal
history/version/fanout pipeline, and stores the exact serial receipt. An
identical REST or realtime retry returns that receipt without publishing again;
the same ID with a different canonical payload or publisher identity fails
deterministically. Redis and Redis Cluster use atomic compare-and-swap scripts,
so a handler-local acknowledgement cache is never the cross-node authority.
The durable envelope stores only the hashed idempotency identity and payload
fingerprint. If a node exits after history/version persistence but before the
receipt compare-and-swap, another node reads the authoritative durable record,
atomically repairs the pending receipt, and returns the original acknowledgement
without repeating persistence or fanout. Ordinary publishes retain their
message-ID acknowledgement shape; versioned publishes retain the exact original
message, history, delivery, and version serial tuple.
Realtime echo combines the connection `echoMessages` setting, the channel
`echo` parameter, and the optional message-level `extras.echo` override. Only
the actual originating connection is suppressed; another connection with the
same client ID still receives the message. A publish frame is validated in full
before the first commit, then receives exactly one ACK or NACK covering its
inbound `msgSerial`/`count` range, including a frame containing multiple
messages.
### Presence across reconnects
Ably `DISCONNECT` and `CLOSE` have different presence lifecycles. `DISCONNECT`
ends the current WebSocket transport and starts the Ably presence grace period.
The default is 15 seconds; the `remainPresentFor` transport parameter may reduce
it to a minimum of one second. Recovery during that window keeps the
authoritative member set and cancels the pending removal. After the grace period,
Sockudo removes the members and fans out their leave transitions while retaining
the connection recovery state for its independent two-minute lease. `CLOSE` is
terminal: it removes the connection's members immediately and invalidates its
recovery state. Both paths still unsubscribe the terminated transport session
from local channel delivery.
On automatic presence re-entry, Sockudo preserves a supplied member ID only
when it is a valid `connectionId:msgSerial:index` ID owned by the current
connection. This keeps same-connection re-entry idempotent while a missing,
malformed, or foreign-connection ID is replaced with a server-assigned ID.
Presence snapshots remain keyed by the `(clientId, connectionId)` pair, so the
same client ID on distinct connections remains distinct.
### Batch REST routing
The Ably compatibility router accepts both `POST /messages` request families.
A raw `Rest.request()` batch spec returns the legacy ordered channel array with
`201` when every publish commits. If any channel fails, already committed
channels remain successful and the route returns `400`/`40020` with
`{ error, batchResponse }`; every entry retains its original position and its
own `messageId` or `error`. The `batchPublish()` array body returns an ordered
array of `{ successCount, failureCount, results }` envelopes with `200`, including
per-channel partial failures.
Every message is parsed and validated before its channel work starts. Successful
items call the native `MessageService` and are not acknowledged until its
idempotency, durable history/version, fanout, metrics, webhook, and optional push
path returns a commit receipt. `GET /presence` reads each channel independently
through the native `PresenceService`; `/keys/{keyName}/revokeTokens` writes each
valid target through the shared revocation store. A failure in one presence read
or revocation write is therefore an item failure, not a fabricated whole-batch
success or a rollback of unrelated items.
Batch work is bounded and ordered. At most eight channel/target operations run
concurrently. A request accepts at most 100 specs, 1,000 channel results, 10,000
channel-message publish operations, and 10 MiB of encoded JSON or MsgPack. The
per-spec channel and message limits are additionally capped by
`event_limits.max_channels_at_once` and `event_limits.max_batch_size`. Known
resources reject unsupported methods immediately with `405` and `Allow`; unknown
resources return `404`. JSON/MsgPack content types, Ably error headers, and
pagination links use the shared REST encoders.
### Channel names
The Ably facade parses channel names independently from Sockudo's native
Pusher-compatible validator. After REST path decoding, Ably channels may contain
spaces, namespace colons, braces, quotes, and Unicode. Empty names, control
characters, reserved leading `:` names, malformed qualifiers, and names larger
than 16 KiB of UTF-8 are rejected with Ably error code `40010`. These rules apply
consistently to REST channel operations and realtime attach, publish, and
presence frames; Protocol V1 validation is unchanged.
Qualified names such as `[?rewind=1]room` and `[filter=...]room` retain their
full name on Ably wire messages, capability checks, attach state, and errors.
Their base name (`room`) is the sole identity used by Sockudo fanout, history,
recovery, presence, and mutable-message services. Attaching both a base and a
qualified name therefore creates two subscription projections over one channel
authority, not duplicate durable state.
Derived `[filter=]` subscriptions compile the decoded JMESPath expression
once at attach. The expression is size-bounded and filters both live messages and
recovery backlog before wire encoding; malformed base64, UTF-8, or JMESPath fails
only that channel with code `40010` and leaves the connection usable.
Compiled predicates are shared by expression through a bounded 1,024-entry,
1 MiB cache. Cache hit, miss, eviction, current-entry, and current-byte counters
are included in the compatibility runtime metrics snapshot. Compilation happens
outside the cache lock, and delivery evaluates the cached predicate over the
real Ably message projection, including extras and headers.
Realtime attach negotiates channel modes as typed flags. `params.modes` takes
precedence over the protocol mode flags, unknown parameters are not reflected,
and an attach without explicit modes starts from the supported Ably default set.
The requested/default modes are intersected with the authenticated token's
channel capabilities. Unsupported modes are omitted from `ATTACHED` rather than
rejecting an otherwise authorized attachment; an attachment fails with `40160`
only when the intersection is empty. This mode intersection never widens
authority: annotation operations and object publishing still require their
explicit capabilities, object subscription follows Ably's `subscribe` or
`object-subscribe` rule, and an operation outside the granted modes is rejected.
String, boolean, and finite numeric attach parameters are normalized at the
JSON/MsgPack protocol boundary; binary, null, array, object, and non-finite
values are rejected instead of widening runtime parameter handling.
The negotiated parameters and flags are echoed on every `ATTACHED` outcome.
Each connection retains a typed attachment contract containing the parsed
qualified/base name, accepted parameters, explicit/default modes, compiled
filter, attach position, and presence state. Reattach requests that omit these
fields reuse the retained contract instead of reverting to unrelated defaults.
Publish, presence entry, message subscription, and presence subscription are
enforced independently; a disallowed publish or presence operation is NACKed
with `40160` without failing the connection or another channel.
REST history uses Sockudo's opaque durable-history cursor. When more results are
available, the facade emits Ably-compatible relative `first` and `next` Link
relations and preserves the original direction, limit, and bounds without
reflecting credentials into the link. JSON and MsgPack pagination use the same
cursor and base-channel state.
For realtime rewind, the selected durable/versioned message is collected before
`ATTACHED`; `HAS_BACKLOG` is set exactly when a replay item follows that frame.
Rewind accepts positive message counts and positive `s`, `m`, or `h` time forms,
is capped by the resolved channel history policy, and is delivered oldest first.
An `ATTACH_RESUME` request retains the channel position but does not replay the
rewind selection a second time.
### Ably VCDIFF delta delivery
An Ably channel may request `params.delta = "vcdiff"` on `ATTACH`. The accepted
value is echoed on `ATTACHED`; other delta modes are not accepted or reflected.
This is an Ably-only egress projection and is separate from Sockudo's native
Pusher/V2 delta events and `[delta_compression]` algorithm settings.
Each attached subscriber retains at most one base for each requested channel,
keyed by the last delivered Ably message ID. A base is limited to 64 KiB and
expires after 120 seconds; the periodic compatibility sweep removes idle expired
bases. Detach, connection close, reattach, recovery discontinuity, and a native
stream-generation change reset the base. VCDIFF is emitted only when encoding
succeeds and the delta is smaller than the complete encoded payload. Otherwise
the subscriber receives a full message and that full payload becomes the next
eligible base.
JSON deltas use the portable `json/utf-8/vcdiff/base64` encoding chain and
`extras.delta = { from, format: "vcdiff" }`. Unsupported, binary, or cipher
encoding chains pass through unchanged and break the chain rather than being
reinterpreted as JSON. JSON and MsgPack subscribers use the same payload and ID
semantics. Missing bases and decode errors are handled by ably-js as `40018`
reattach recovery; a client that requested VCDIFF without installing its plugin
fails the channel with `40019`. The reattach starts with a full canonical
message before delta delivery resumes.
Only the per-subscriber wire projection is compressed. Durable history, hot
replay, recovery inputs, webhooks, annotations, push, and cross-node fanout all
retain and distribute the original complete message. Run
`cargo bench -p sockudo-ably-compat --bench ably_vcdiff --features delta` to
measure CPU time, allocation count/bytes, and encoded/full ratios for similar
and dissimilar 1 KiB and 64 KiB payloads at 1, 100, and 1,000 subscribers.
Subscriber registration happens before the canonical attach high-water mark is
captured. A bounded count-and-byte gate holds concurrent fanout until `ATTACHED`,
drops gated rows already covered by rewind/recovery, and then drains only later
positions in order. Gate overflow or a stream-generation change fails that
channel closed instead of silently losing a publish.
### Presence
Ably virtual connections register with Sockudo's typed `PresenceService` and a
bounded current-member registry keyed by app, base channel, connection, and
client. Enter, update, leave, detach, explicit close, recovery expiry,
failed-resume re-entry, ATTACHED `HAS_PRESENCE`, and SYNC all project from that
authority. Duplicate disconnects are idempotent; two connections with one
client ID remain separate members while native first-join/last-leave facts are
preserved. Presence extras, data, timestamps, encoding chains, and
`connectionId:serial:index` member IDs are retained. Large SYNC sets move
members into ordered 100-item continuation frames without per-chunk member
clones. Inbound action-16 frames request the current authoritative SYNC instead
of failing the connection.
Virtual members are also written to Sockudo's native horizontal presence
registry with stable virtual socket identities. Native new-node presence-state
sync rebuilds current snapshots, while a dedicated protocol-neutral broadcast
envelope carries live transitions to Ably subscribers on other nodes. Receiving
nodes consume that envelope before Protocol V1 fanout, so compatibility
replication cannot appear as a Pusher event.
`GET /channels/{channel}/presence` reads the current native snapshot with limit,
client-id, connection-id, and opaque pagination cursor support. Presence
transitions are recorded through Sockudo's configured `PresenceHistoryStore`;
the Ably history route supports direction, time bounds, the same filters,
policy-capped limits, and cursor-preserving `first`/`next` Link relations. Sparse
filters use bounded native-store scans and return a continuation rather than
performing unbounded work. The route projects native records back to Ably
actions and retains data, encoding, extras, timestamps, and IDs. A degraded or
reset-required durable stream returns `503`/`50003` instead of presenting an
unproven history as complete. Qualified attaches and REST reads use the same
base-channel registry and durable history, so qualifiers never create parallel
presence state.
The current-member hot path uses a lock-free authenticated-app index,
per-connection channel handles, inline storage for the common one-connection
client, and bounded sharded capacity reservations. Same-client transitions
remain serialized to preserve exact first-join/last-leave facts, while
disjoint clients avoid shared channel-lookup and global-capacity cache lines.
Empty app and connection channel caches are bounded and pruned only when no
current member or connection handle depends on them.
Run `scripts/presence-bench-guard.sh` (or `make presence-bench-guard`) after a
presence hot-path change. The guard compares authoritative and legacy cycles in
the same Criterion process and requires the authoritative confidence interval
to remain faster for both single-worker and eight-worker disjoint-client loads.
### Message annotations
Realtime annotation action `21` and
`GET|POST /channels/{channel}/messages/{messageSerial}/annotations` are edge
projections over Sockudo's native `AnnotationStore`; the facade does not keep a
second annotation store. Create and delete events retain their annotation
action, message serial, type, name, client identity, data, encoding, server
timestamp, and monotonically ordered annotation serial. A caller-supplied
annotation ID makes an identical retry idempotent.
Channel modes and capabilities are separate concerns. `annotation_publish`
permits create, `annotation_subscribe` permits raw action-21 delivery, and
`annotation-delete-own` / `annotation-delete-any` authorize delete independently
from mutable-message permissions. Ordinary subscribers receive only
`message.summary`; raw annotations are emitted only when the negotiated
annotation-subscribe mode is present. Multiple/counting summaries rename the
native `clientCounts` member to Ably's `clientIds` only at this edge.
REST annotation pages negotiate JSON or MsgPack and return relative `first` and
`next` Link relations. The cursor is opaque and scoped to the authenticated app,
base channel, and original message serial; credentials are never reflected into
links. Live fanout and recovery use the same durable delivery positions as the
native summary/raw events. An unprovable history gap therefore fails through
the existing recovery path instead of fabricating annotation state.
The surface requires global `[annotations].enabled`, versioned messages, and an
app/channel policy with annotations enabled. Protocol V1 receives neither the
raw annotation event nor the V2/Ably summary projection.
### Realtime delivery bounds
Compatibility delivery is injected at the native local delivery boundary. It
is interest-gated, synchronous, and non-blocking when no Ably session is
attached to a channel. Active sessions use separate bounded control and data
queues, each limited by both frame count and bytes. The data queue honors the
server's `[websocket].max_messages` and `[websocket].max_bytes` bounds; unset
limits retain bounded compatibility defaults. ACK, ERROR, heartbeat, and
close-control frames are prioritized over data frames. A data overflow marks
continuity lost, stops further data for that session, and sends a recoverable
`90003` error when possible; the next attach/recovery reads canonical Sockudo
hot replay or durable history.
Messages are encoded once for each active JSON/MessagePack group and shared as
immutable bytes across that group's subscribers. Session, token, channel, and
queue state have hard bounds and periodic expiry. No compatibility lock is held
while encoding or fanning out, and the compatibility projection does not alter
Protocol V1 delivery. `/operator/stats/ably-runtime` exposes `data_encoded` for
these shared projections separately from `encoded`, which also includes
connection, channel, ACK, and heartbeat control frames.
For process-local deployments, compatibility coordination state is kept in the
runtime rather than the general bounded memory cache. Redis and Redis Cluster
remain the shared coordination authority for multi-node deployments; the
configured cache still persists compatibility statistics in both deployment
modes.
In Redis-backed deployments, connection recovery keys are stored for the
advertised connection-state TTL in the shared cache as well as the local
runtime. A reconnect routed to another node can therefore validate the key and
recover each reattached channel from canonical hot replay or durable history; a
missing or expired `recover` key returns `80018`. The active key lease is
refreshed at half the advertised TTL, then receives a full TTL from an
ungraceful disconnect; an explicit `CLOSE`
invalidates the active key after sending `CLOSED`. Cached state never includes
app secrets or provider credentials.
Realtime remains WebSocket-only. Operator listeners configured with
`[ably_compat].realtime_admission = "placement_constraint"` return a genuine
`DISCONNECTED`/`50320` placement response, allowing the SDK to reconnect to a
configured fallback host over WebSocket. This admission mode does not add
Comet, XHR, SSE, polling, or streaming transports. Realtime messages without an
explicit ID derive the stable Ably ID `connectionId:msgSerial:index`; retrying
an unacknowledged frame therefore reuses the native idempotency path instead of
duplicating persistence or fanout.
ATTACH processing is bounded by `ably_compat.attach_timeout_ms` across native
presence snapshot, hot recovery, and durable history dependencies. A timeout
cancels that work, removes the partially registered subscriber, and returns a
channel-scoped `DETACHED`/`50003`, allowing the SDK's normal channel retry timer
to run without leaving a delivery route active before `ATTACHED`.
Token expiry and revocation checks are generation-stamped. An `AUTH` update
atomically replaces credential identity, client identity, capability, expiry,
and revocation metadata, so an older timer or delayed revocation cannot close a
renewed session. Revocations are stored in the configured shared cache and are
visible to compatibility sessions on every node. The targets `clientId`,
`revocationKey`, and `channel`, `issuedBefore`, and the delayed
`allowReauthMargin` flow are supported. Tokens, JWTs, key secrets, and signing
material are not written to compatibility logs.
### Statistics and push
Authenticated `GET /stats` supports minute, hour, day, and month units;
inclusive interval-ID or epoch-millisecond `start` and `end` bounds;
forward/backward direction; policy-capped limits; opaque, query-scoped cursors;
and credential-free `first`/`next` Link relations. Canonical UTC minute buckets
are merged through the configured cache using compare-and-swap, and larger units
are deterministic read-time projections. A shared persistent cache such as
Redis provides cross-node and restart-safe history; the memory backend remains
node-local.
Live counters cover inbound and delivered outbound message counts and encoded
bytes, API and token-request outcomes, connections, channels, presence, and push
admission. A bounded batching worker persists publish, attach, and connection
observations before their application acknowledgements; nonblocking outbound
delivery observations report drops on saturation. The worker's backlog,
capacity, drops, flush failures, completed batches, observation count, and lag
are available from `GET /operator/stats/aggregation`. Authenticated conformance
interval ingestion is separately disabled by default and, when enabled, writes
through the same typed minute store rather than returning canned fixture data.
The Ably push projection is compiled only with `ably-compat`, which also enables
Sockudo's native `push` feature. Device registrations and channel subscriptions
use the configured native `DynPushStore`. `/push/publish` and ordinary publishes
with `extras.push` create native `PublishIntent` records through `PushPipeline`;
they do not publish directly from the REST handler. Planning, queue admission,
provider batching, retry, feedback, aggregate status, repair, and retention use
the same workers and durable records as Sockudo's native push API.
The Ably-channel test transport is the native realtime provider. Its dispatch
worker publishes `__ably_push__` through `MessageService` and reports an accepted
provider outcome only after real realtime fanout succeeds. Retries reuse a
bounded delivery id so duplicate provider attempts remain idempotent. Client-ID
channel subscriptions are represented by typed client-scoped native
subscriptions and resolve the client's current durable device set at planning
time. `push-admin` and `push-subscribe` capabilities remain channel scoped;
device-scoped operations verify the hashed device identity token, and updates do
not rotate that token. Recipient scans are paged, and provider credentials,
device tokens, private keys, and raw payloads are not logged.
For an in-process deployment, build with `monolith` so planner, retry, feedback,
scheduler, repair, and cleanup workers run beside the compatibility realtime
provider worker. External FCM, APNs, Web Push, HMS, and WNS outcomes are never
synthesized: delivery requires the corresponding native provider feature,
worker, and real credentials.
The standard Sockudo Docker image includes `monolith`. Custom images that
override the `SOCKUDO_FEATURES` build argument must include it when Ably push
compatibility uses a broker-backed `push.queue_driver`.
## Additional compatibility surfaces
| Surface | Status | Scope |
| --- | --- | --- |
| History and recovery | Supported for tested Pub/Sub subset | The harness covers forward/backward REST pagination, `untilAttach`, history projection, and recovery of missed append fragments. Broader Ably history API parity is not claimed. |
| Presence enter/update/leave | Supported for tested Pub/Sub subset | Realtime presence is ACKed, server-stamped, synchronized, and fanned out; REST current presence and durable presence history share the same base-channel authority. Full Ably presence API parity is not claimed. |
| Message annotations | Supported for the pinned annotation suite | Realtime create/delete and raw subscription, summary projection, REST pagination, and JSON/MsgPack all use the native annotation subsystem. |
| Keys and capabilities | Supported for tested Pub/Sub subset | `App.key`/`App.secret` is the primary credential; opt-in `[ably_compat]` keys share its app state. Key and token capabilities are enforced for REST publish/history/status/message reads and realtime attach/publish/presence operations. |
| Token lifecycle | Supported for tested WebSocket subset | Canonical `access_token` and `client_id` connection parameters are accepted with documented camel-case aliases. Signed HS256 Ably JWTs and issued opaque tokens carry identity, capability, and expiry into the connection. In-band `AUTH` renews authorization without changing the connection ID; expiry and revocation emit `DISCONNECTED` 40142 and 40141. Capability downgrades fail affected channels only. |
| LiveObjects/object modes | Intentionally out of scope | The current compatibility target does not implement Ably LiveObjects. |
| Ably Push | Supported for the pinned REST subset | Device registration, channel subscription, listing/deletion, direct publish, and `extras.push` use native durable storage, queue, planner, provider dispatch, feedback, retry, status, scheduler, and cleanup paths. Full provider-platform parity is not claimed. |
| Binary MsgPack Ably protocol | Supported for the tested surface | Realtime Pub/Sub and REST publish/history/time are selected from pinned `ably@2.21.0` with `useBinaryProtocol: true`. Full Ably platform MsgPack parity is not claimed. |
| Full Ably platform API parity | Intentionally out of scope | This surface is for AI Transport compatibility, not a replacement for all Ably services. |
## Test commands
1. Start Sockudo locally with the Ably compatibility feature, AI Transport,
durable history, and versioned messages enabled. The default repository
config uses `app-key:app-secret` and an AI channel prefix of `private-ai-`.
```bash
cargo run -p sockudo --features "v2,ai-transport,ably-compat,redis,postgres,push,monolith" -- \
--config config/config.toml
```
2. From the pinned `sockudo-compatibility` harness checkout, install
dependencies:
```bash
cd sockudo-compatibility
npm ci
```
3. Run the compatibility lanes. Each lane writes its own report:
```bash
make conformance
make strict-completeness
make browser-install
make browser-conformance
make browser-strict
make go-conformance
make ait-conformance
```
`make browser-matrix` runs Chromium first, then the pinned Firefox and WebKit
revisions. Browser reports fail on console or page errors and leaked contexts in
addition to assertion, runner, pending, and accounting failures. Browser support
services are loopback-only and real; the runner blocks requests to Ably-hosted
infrastructure.
The targets read these environment variables:
| Variable | Default |
| --- | --- |
| `ABLY_KEY` | `app-key:app-secret` |
| `ABLY_ENDPOINT` | `127.0.0.1` |
| `ABLY_PORT` | `6001` |
| `ABLY_TLS` | `false` |
| `ABLY_CLIENT_ID` | script-specific |
| `ABLY_AGENT_CLIENT_ID` | `sockudo-ably-ait-agent` |
| `ABLY_CHANNEL` | generated per run |
| `ABLY_DEMO_PROMPT` | `Say hello from Sockudo AI Transport.` |
| `ABLY_BROWSER_ORIGIN_PORTS` | `4173,3001,5174` |
## Demos
`make ably-ai-demo` runs two headless demos:
1. A stock `@ably/ai-transport@0.10.0` chat demo using `createTransport` with
the `@ably/ai-transport/vercel` codec, pinned `ably@2.30.0`, and Sockudo as
the configured endpoint.
2. A recovery/history demo that disconnects during an AI output stream, appends
additional fragments while offline, reconnects with an Ably recovery key, and
verifies that only missed fragments replay and the final aggregate is intact.
These AI Transport demos are deliberately Node-based so CI can run them without
a browser.
## Evidence files
The working compatibility evidence lives in repository docs:
- `docs/ably-compat/ABLY_COMPAT_SCORECARD.md`
- `docs/ably-compat/ABLY_COMPAT_GAPS.md`
- `docs/ably-compat/TEST_PLAN.md`
Update those files whenever a supported/deferred status changes.
## Next steps
- [AI Transport overview](/docs/server/ai-transport-overview): The native run lifecycle the facade maps onto.
- [Mutable messages](/docs/server/mutable-messages): Versioned messages behind appendMessage and updateMessage.
- [History and recovery](/docs/server/history-recovery): Durable history, rewind, and reconnect continuity.
- [Configuration reference](/docs/reference/configuration): The [ably_compat] keys and their defaults.
---
# AI Transport conventions (https://sockudo.io/docs/server/ai-transport-conventions)
> Default-off AI Transport event and extras conventions for Protocol V2.
This page is the working reference for what an AI Transport message looks like on the wire: which
event names exist, who may publish them, which `extras.ai` headers Sockudo validates, and how
idempotent retries work. Read it when you write your own agent publisher, debug a rejected publish,
or plan a production deployment. The SDK applies these conventions for you; you need the details
when you step outside it.
## Enable and scope
AI Transport support is additive and default-off. Enable it with the `ai-transport` Cargo feature
and runtime `[ai_transport]` config, then scope validation to channel prefixes:
```toml title="config.toml"
[ai_transport]
enabled = true
[[ai_transport.channels]]
prefix = "private-ai-"
```
A channel is an AI channel when its name starts with one of the configured prefixes. Server-side
validation applies only on enabled AI channels. Publishing an AI event name or `extras.ai` metadata
to any other channel is rejected with `104003` `ai_event_not_permitted`.
## Events and who may publish them
AI lifecycle events are plain channel events: `ai-input`, `ai-output`, `ai-run-start`,
`ai-run-suspend`, `ai-run-resume`, `ai-run-end`, and `ai-cancel`. Agents additionally publish
`ai-step-start` and `ai-step-end` around re-attemptable steps inside a run.
| Event | WebSocket client | Trusted app-key HTTP |
| --- | --- | --- |
| `ai-input`, `ai-cancel` | Allowed with `publish` capability and a verified client identity | Allowed |
| `ai-output`, `ai-run-start`, `ai-run-suspend`, `ai-run-resume`, `ai-run-end` | Rejected with `104003` | Allowed |
| `ai-step-start`, `ai-step-end` | Rejected with `104003` | Allowed |
| `ai-turn-start`, `ai-turn-end` (legacy aliases) | Rejected with `104003` | Allowed |
WebSocket clients may publish `ai-input` and `ai-cancel` when their token allows `publish`; agent
events require trusted server-side HTTP publish until an explicit agent token marker is specified.
WebSocket AI publishes also require Protocol V2 and an authenticated `client_id` (a capability
token or a signed-in user); AI publishes from an anonymous connection are refused.
## The extras envelope
AI metadata is carried in the existing V2 `extras` envelope. It has two tiers:
- **`transport`** holds the headers Sockudo understands and validates: run identity, status,
lineage, and identity claims.
- **`codec`** holds SDK- or provider-specific fields. Sockudo bounds their size but never
interprets them.
```json
{
"extras": {
"ai": {
"transport": {
"run-id": "run-1",
"status": "streaming",
"role": "assistant"
},
"codec": {
"content-type": "application/json"
}
}
}
}
```
Each `transport` and `codec` tier is limited to 32 string keys. Keys must be at most 64 bytes.
Transport keys match `[a-z0-9-]+`; codec keys match `[A-Za-z0-9_-]+` for Ably/Vercel codec fields.
Transport values and ordinary codec values must be at most 256 bytes;
`extras.ai.codec.providerMetadata` may be up to 8 KiB because Vercel provider metadata is
JSON-serialized there. Transport keys are validated against the wire protocol registry. Codec keys
are opaque but still bounded.
Other keys placed directly under `extras.ai` (for example Ably AI Transport 0.9's `type`, `fields`,
`stream`, and `ends`) are preserved as opaque JSON, subject to the message size limit. They cannot
be named `transport` or `codec`.
`[ai_transport].max_transport_value_bytes` (default `256`) and
`[ai_transport].max_steer_codec_message_ids_bytes` (default `2048`) can only be raised. Startup
rejects a value below the protocol floor, because lowering either one would start rejecting traffic
that previously passed validation.
### Transport header registry
Unknown transport keys are rejected with `104000` `ai_invalid_transport_header`.
| Key | Accepted values |
| --- | --- |
| `run-id` | Non-empty opaque string. |
| `run-client-id` | Verified client identity, or the empty sentinel for an unknown owner (trusted publishes only). |
| `run-reason` | `complete`, `cancelled`, `error`. |
| `run-continue` | `true` or `false`; marks a client input as re-entering an existing run. |
| `codec-message-id` | Non-empty opaque string; equals Sockudo `message_serial` for streamed mutable messages. |
| `input-codec-message-id` | Non-empty opaque string. |
| `parent` | Parent `codec-message-id`. |
| `fork-of` | Fork source `codec-message-id`. |
| `msg-regenerate` | Non-empty `codec-message-id` of the assistant message being regenerated. |
| `supersedes` | Non-empty `run-id` of a suspended reply replaced by a client tool-result fork. |
| `stream` | `true` or `false`. |
| `stream-id` | Non-empty opaque string. |
| `status` | `streaming`, `complete`, `cancelled`. |
| `discrete` | `true` or `false`. |
| `invocation-id` | Non-empty opaque string. |
| `event-id` | Non-empty opaque string. |
| `input-client-id` | Verified client identity. |
| `step-id` | Non-empty opaque string; stable logical step identity across attempts. |
| `step-start-serial` | Non-empty serial of the matching `ai-step-start` attempt. |
| `step-reason` | `complete`, `failed`, `cancelled`. |
| `step-client-id` | Verified client identity, or the empty sentinel for an unknown participant (trusted publishes only). |
| `steer-codec-message-ids` | Non-empty JSON-stringified array of steer IDs; 2 KiB ceiling instead of 256 bytes. |
| `role` | `user`, `assistant`, `system`, `tool`, `agent`. |
| `error-code` | Non-empty opaque string. |
| `error-message` | Human-readable string within the value limit. |
| `model` | Non-empty opaque string; optional model identifier. |
Legacy transport aliases `turn-id`, `turn-client-id`, `turn-reason`, and `turn-continue` are
accepted for inbound and stored compatibility; `turn-reason` additionally accepts `suspended`.
Ably-native regenerate publishes use `extras.ai.transport.msg-regenerate` as the codec-message-id
of the assistant message being regenerated.
### Identity headers cannot be spoofed
Any `*-client-id` transport value on an untrusted WebSocket publish must match the verified
signed-in client identity; a mismatch is rejected with `104002` `ai_client_id_spoof`. Trusted
app-key publishes may preserve empty `run-client-id` and `step-client-id` values as unknown-owner
sentinels; derived identities treat those empty values as absent. For an untrusted client, an empty
value is still a spoof.
## Canonical run sequences
These are the event orders the SDKs produce and expect. Every streamed output is one mutable
message, so "create", "append", and "update" below are `sockudo:message.*` operations on the same
`message_serial`.
| Scenario | Sequence |
| --- | --- |
| Normal streaming run | `ai-run-start` (with `run-id`, `invocation-id`, `run-client-id`); create `ai-output` with `stream=true`, `status=streaming`; zero or more appends with `status=streaming`; final append or update with `status=complete`; `ai-run-end` with `run-reason=complete`. |
| Client cancellation | Client publishes `ai-cancel` with `run-id`; agent emits a final append or update with `status=cancelled`; `ai-run-end` with `run-reason=cancelled`. |
| Error | Stream starts normally; agent emits an update with `status=complete`, `error-code`, and `error-message` (or no final content if no message was created); `ai-run-end` with `run-reason=error`. |
| Suspend and resume | Agent emits `ai-run-suspend` with `run-id` and the current `invocation-id`; the continuation emits `ai-run-resume` with the same `run-id` and a new `invocation-id`. |
| Regenerate | New output create carries `msg-regenerate=` and parents at the same input as the regenerated reply. |
| Edit | Edited user input is a new `ai-input` or mutable update with `parent=`; downstream outputs fork from that parent. |
A stream stays "active" until a final append or update carries `status=complete` or
`status=cancelled`. If your agent crashes before that, the orphan janitor cancels it after
`ai_transport.rollup.orphan_ttl_ms` (see below), but users wait the full TTL first.
## Idempotency and acknowledgements
`extras.ai.codec` may instead be a single string: the codec id that Ably AI Transport stamps on every message (for example `"vercel-ai-sdk-ui-message"`, `"openai-responses"`, `"ag-ui-events"`). Ably decoders skip messages whose id does not match exactly, so Sockudo relays the string verbatim on live delivery and history. A codec id is limited to 256 bytes and is not otherwise validated.
Existing `idempotency_key` behavior is unchanged. AI SDKs may additionally send `message_id` on
HTTP publish for idempotent creates, and mutation requests may send `op_id` for no-op replay of
append/update/delete operations. V2 publish and mutation acknowledgements expose `message_serial`,
`history_serial`, `delivery_serial`, and `version_serial` when those serials are assigned.
| Mechanism | Dedupe key | Retry behavior |
| --- | --- | --- |
| `message_id` on create | `(app_id, channel, message_id)` | Returns the original serials; no rebroadcast. At most 64 bytes on AI channels. |
| `op_id` on append, update, delete | `(app_id, channel, message_serial, action, op_id)` | Same payload returns the original acknowledgement with `status: "duplicate"`; a different payload is a conflict. |
| `X-Idempotency-Key` header or body `idempotency_key` | App and key | Returns the cached acknowledgement; the body value wins over the header. |
When the event name is an AI event or `message_id` is present, `POST /apps/{appId}/events` responds
with per-channel serials:
```json
{
"channels": {
"private-ai-user-42-sess-01J": {
"message_serial": "msg_...",
"history_serial": 123,
"delivery_serial": 456,
"version_serial": "ver_..."
}
}
}
```
`GET /apps/{appId}/channels/{channelName}` adds an `ai` block with `active_streams`,
`last_history_serial`, and `message_count` for channels that match an AI prefix.
## Deployment matrix
AI Transport requires durable history and versioned messages. Startup rejects
`ai_transport.enabled = true` unless both `[history].enabled` and `[versioned_messages].enabled` are
true.
| Adapter | Cache | History backend | Version store | Support |
| --- | --- | --- | --- | --- |
| `local` | `memory` | `memory` | `memory` | Single-node development only |
| `local` | Redis or memory | PostgreSQL, MySQL, DynamoDB, ScyllaDB, or SurrealDB | PostgreSQL, MySQL, DynamoDB, ScyllaDB, or SurrealDB | Single-node or externally isolated worker |
| `redis`, `redis-cluster`, `nats`, `pulsar`, `rabbitmq`, `google-pubsub`, `kafka`, `iggy` | Redis or Redis Cluster | PostgreSQL, MySQL, DynamoDB, ScyllaDB, or SurrealDB | PostgreSQL, MySQL, DynamoDB, ScyllaDB, or SurrealDB | Supported horizontal deployment |
| any horizontal adapter | `memory` or `none` | any | any | Rejected at startup |
| any horizontal adapter | any | `memory` | any | Rejected at startup |
| any horizontal adapter | any | any | `memory` | Rejected at startup |
The horizontal rejections exist because every node must agree on the same history, the same version
chains, and the same owner for an orphaned stream. A node-local store would split that authority.
Under AI Transport, versioned-message delivery serials are leased in 128-position blocks per node
and channel. The backing store still owns monotonic reservation authority; the lease cache reduces
append-stream reservation contention without changing stored serials or recovery semantics.
## Rollup and orphaned streams
Append rollup is egress-only and defaults to a 40 ms window; see
[Token streaming and rollup](/docs/server/token-streaming-rollup). Streaming messages are tracked in
the shared cache while their `extras.ai.transport.status` is `streaming`. Any node can claim a stale
stream after `ai_transport.rollup.orphan_ttl_ms`, which defaults to 60000 ms, then append a normal
versioned `message.update` with `status=cancelled`, `error-code=orphan_timeout`, and an
`error-message` explaining the timeout. The claim key prevents duplicate cancellation across nodes,
and the latest version is re-read before mutation so active streams that advanced meanwhile are
refreshed instead of cancelled.
Apps can subscribe to the webhook event type `ai_stream_orphaned` to observe synthetic orphan
cancellation. The event payload includes `channel`, `message_serial`, and `reason`. The other AI
webhook event types are `ai_run_started`, `ai_run_ended`, and `ai_cancel_requested`; see
[Webhooks](/docs/server/webhooks).
## Scale-out verification
The repository includes a three-node Redis/PostgreSQL AI Transport stack and verification scripts:
1. Start the stack:
```bash
docker compose -f docker-compose.ai-transport.yml up --build -d
```
2. Run the cross-node benchmark:
```bash
node scripts/ai-transport-3node-bench.mjs
```
3. Run the partition check:
```bash
scripts/ai-transport-jepsen-lite.sh
```
`scripts/ai-transport-3node-bench.mjs` runs a same-stack baseline plus a cross-node workload,
verifies every node reads the same final aggregate, and enforces the cross-node added append p99
budget. `scripts/ai-transport-jepsen-lite.sh` runs the same correctness check before, during, and
after a docker `pause`/`unpause` partition of one node; steady state uses `MAX_APPEND_ADDED_P99_MS`
and partition/heal phases use `PARTITION_MAX_APPEND_ADDED_P99_MS`.
## Next steps
- [Authentication](/docs/server/ai-transport-authentication): Capability tokens, identity, and who may publish each event.
- [Limits and quotas](/docs/server/ai-transport-limits-quotas): Every cap that applies to AI traffic and what happens when you hit it.
- [Mutable messages](/docs/server/mutable-messages): The versioned-message model behind ai-output streams.
- [Troubleshooting](/docs/server/ai-transport-troubleshooting): Decode rejection codes and fix common integration mistakes.
---
# AI Transport providers (https://sockudo.io/docs/server/ai-transport-providers)
> Stream model output from Vercel AI SDK, OpenAI, Anthropic, OpenAI-compatible APIs, and custom providers through Sockudo AI Transport.
Sockudo AI Transport is provider-neutral. Sockudo carries ordered, durable run state; your agent
chooses how to call the model and converts provider output into AI SDK UI message chunks. This page
shows how to connect each supported provider family, with complete agent code, and how to handle
tool calls, approvals, voice, and generated media. Read it when you wire your first model into a
session or switch providers.
## Pick an integration
The JavaScript SDK has three entry points for agents:
| Integration | Import | Best for |
| --- | --- | --- |
| Vercel AI SDK transport | `@sockudo/ai-transport/vercel` | Apps already using `ai`, `streamText`, `useChat`, Vue, React, or Svelte AI SDK UI primitives. |
| Direct provider adapters | `@sockudo/ai-transport/providers` | Workers that call OpenAI, Anthropic, OpenAI-compatible HTTP/SSE endpoints, or local model servers directly. |
| Core codec transport | `@sockudo/ai-transport` | Custom model protocols, custom UI projections, or non-Vercel message shapes. |
| You have | Use |
| --- | --- |
| Vercel AI SDK `streamText` already wired | `@sockudo/ai-transport/vercel` |
| A provider with OpenAI-compatible SSE | `createOpenAICompatibleProvider` |
| Official OpenAI SDK | `createOpenAISdkProvider` |
| Official Anthropic SDK | `createAnthropicSdkProvider` |
| A custom internal model gateway | `DirectLlmProvider` |
| A non-Vercel UI model | Core codec API from `@sockudo/ai-transport` |
| AI SDK agent or workflow output | Convert to UI message chunks, then stream through the Vercel transport |
| Realtime voice or generated video | Provider media path plus Sockudo transcript/control events |
Sockudo AI Transport is heavily inspired by [Ably AI Transport](https://ably.com/docs/ai-transport),
and Ably's public AI Transport docs and SDK were the main product inspiration for this layer. The
Sockudo implementation maps those ideas onto Sockudo Protocol V2, durable history, versioned
messages, presence, push, and the `@sockudo/client` SDK in this monorepo. Sockudo AI Transport is
community-built and community-maintained. It is not an Ably product and is not supported by Ably.
## Provider flow
Every integration has the same shape. The provider only ever streams chunks to your worker; the
worker turns them into mutable-message operations on the session channel.
```mermaid
sequenceDiagram
participant U as User client
participant S as Sockudo
participant A as Agent worker
participant M as Model provider
participant H as History store
U->>S: ai-input on the session channel
S->>H: persist the user message
A->>M: stream request
M-->>A: deltas, tool calls, finish
A->>S: create and append ai-output
S->>H: persist every mutation
S-->>U: live rolled-up appends
Note over U,H: a refreshed client rewinds history instead
```
The provider never needs to know about connection recovery, rewind, history cursors, branch trees,
presence, or multi-device fanout. It streams chunks. Sockudo makes those chunks durable and
observable.
Every example below passes a `realtimeClient()` to `createAgentSession`. That is app code: a
`ClientLike` whose `publish`, `appendMessage`, `updateMessage`, `deleteMessage`, and `history`
calls go through signed app-key HTTP, because Sockudo rejects agent events on WebSocket connections.
`client-sdks/sockudo-ai-transport-js/demo/server/utils/sockudo.ts` contains a complete
implementation built on `adaptSockudoClient`.
## Vercel AI SDK
Use Vercel AI SDK when you want provider packages, tool calling, and UI message streams to stay in
the `ai` ecosystem. This Nuxt route mirrors the repository demo:
```ts title="server/api/chat.post.ts"
export default defineEventHandler(async (event) => {
const body = await readBody(event);
const channelName: string = body.sessionName;
const userId = await requireSessionOwner(event, channelName);
const session = createAgentSession({
client: realtimeClient(),
channelName,
});
const run = session.createRun({
runId: body.runId,
invocationId: body.invocationId,
inputEventId: body.inputEventId,
clientId: userId,
onCancel(request) {
return (
request.filter.all === true ||
request.runOwners.get(body.runId) === userId
);
},
});
const work = (async () => {
await run.start();
try {
const result = streamText({
model: "openai/gpt-5-mini",
instructions: "You are a concise incident-room assistant.",
prompt:
body.messages
.at(-1)
?.parts?.map((part) => part.text ?? "")
.join("") ?? "",
abortSignal: run.abortSignal,
});
await run.streamResponse(toUIMessageStream({ stream: result.stream }));
await run.end("complete");
} catch {
await run.end("error");
}
})().finally(() => session.close());
event.waitUntil?.(work);
setResponseStatus(event, 202);
return null;
});
```
The client SDK posts `sessionName`, `runId`, `invocationId`, `inputEventId`, `clientId`, and the UI
`messages`. `onCancel` decides whether an `ai-cancel` may stop this run; checking `runOwners`
against the verified owner stops one user from cancelling another user's run.
The frontend can use the Vercel-compatible transport in React, Vue, or Svelte:
```ts title="src/chat.ts"
const realtime = await createSockudoRealtimeClient("app-key", {
clientOptions: {
wsHost: "realtime.example.com",
forceTLS: true,
channelAuthorization: { endpoint: "/api/sockudo/channel-auth" },
authUrl: "/api/sockudo/token",
},
});
const provider = provideChatTransport({
api: "/api/chat",
channelName: "private-ai-user-42-sess-01J",
client: realtime,
clientId: "user-42",
});
const { sendMessage } = useChat({
id: "private-ai-user-42-sess-01J",
transport: provider.chatTransport.value,
});
await sendMessage({ text: "Summarize the last deploy failure." });
```
React uses `ChatTransportProvider` and `useChatTransport` from
`@sockudo/ai-transport/vercel/react`; Svelte uses `createChatTransportStore` from
`@sockudo/ai-transport/vercel/svelte`.
## AI SDK 7 feature coverage
AI SDK 7 does not require a separate Sockudo server mode. The compatibility surface is the AI SDK UI
message stream: Sockudo persists ordered UI chunks, run lifecycle events, approval responses, and
metadata, while the app runtime keeps provider calls, tool execution, and agent-local state.
| AI SDK capability | Sockudo behavior |
| --- | --- |
| Reasoning controls | Pass model options through `streamText`; Sockudo carries emitted reasoning text and reasoning-file chunks. |
| Tool context and runtime context | Keep context on the trusted worker. Publish only safe derived metadata, never provider secrets or raw tool credentials. |
| Provider file and skill uploads | Let the provider own uploads and references. Sockudo can carry file, source, reasoning-file, or custom chunks that point at externally stored assets. |
| MCP Apps and custom UI parts | Preserve `custom` chunks and provider metadata. Rendering app iframes or custom widgets remains a client responsibility. |
| Tool approvals | Persist approval requests and responses as normal run state. The app still owns approval policy, signature checks, and tool execution authorization. |
| `WorkflowAgent` and `HarnessAgent` output | Stream their UI-message output through `toUIMessageStream`. Use the workflow store for agent checkpoints and Sockudo for transcript/reconnect history. |
| Realtime voice | Keep low-latency audio on the provider's realtime channel. Mirror transcripts, tool calls, approvals, and final state through Sockudo. |
| Video generation | Store generated media outside Sockudo and publish progress, asset URLs, thumbnails, or final file parts through the AI session channel. |
| Telemetry and lifecycle callbacks | Correlate AI SDK spans with `channelName`, `runId`, and `invocationId`. Sockudo metrics continue to describe transport, history, and fanout health. |
## Direct provider support
The `@sockudo/ai-transport/providers` entry point supports these built-in adapters:
| Provider path | Function | Notes |
| --- | --- | --- |
| OpenAI-compatible HTTP/SSE | `streamOpenAICompatibleText` | Uses Chat Completions-compatible `/chat/completions` streaming. |
| OpenAI-compatible reusable provider | `createOpenAICompatibleProvider` | Good for named provider registries. |
| OpenAI SDK Chat Completions | `streamOpenAIChatCompletion` / `createOpenAISdkProvider` | Uses a structural subset of the official OpenAI SDK. |
| OpenAI SDK Responses | `streamOpenAIResponse` / `createOpenAISdkProvider({ mode: "responses" })` | Maps output text and tool argument deltas to UI chunks. |
| Anthropic SDK Messages | `streamAnthropicMessage` / `createAnthropicSdkProvider` | Maps text, thinking, tool use, and finish reasons. |
OpenAI-compatible presets are:
| Name | Default base URL |
| --- | --- |
| `openai` | `https://api.openai.com/v1` |
| `openrouter` | `https://openrouter.ai/api/v1` |
| `groq` | `https://api.groq.com/openai/v1` |
| `togetherai` | `https://api.together.xyz/v1` |
| `fireworks` | `https://api.fireworks.ai/inference/v1` |
| `deepseek` | `https://api.deepseek.com` |
| `perplexity` | `https://api.perplexity.ai` |
| `mistral` | `https://api.mistral.ai/v1` |
| `xai` | `https://api.x.ai/v1` |
| `ollama` | `http://127.0.0.1:11434/v1` |
| `lmstudio` | `http://127.0.0.1:1234/v1` |
Local providers such as Ollama and LM Studio may omit `apiKey` when the local server does not
require one. Pass `baseURL` to target any other compatible endpoint.
### A complete direct-provider worker
`runDirectLlm(run, provider, request)` does the whole run for you: it calls `run.start()`, streams
the provider through `run.streamResponse`, maps the provider finish reason to a run end reason,
publishes `ai-run-end`, and returns `{ pipeResult, runEndReason }`. It also wires
`run.abortSignal` into the provider request, so `ai-cancel` stops the model call. Do not call
`run.start()` or `run.end()` yourself around it.
```ts title="worker/groq-agent.ts"
createOpenAICompatibleProvider,
runDirectLlm,
} from "@sockudo/ai-transport/providers";
const provider = createOpenAICompatibleProvider({
provider: "groq",
apiKey: process.env.GROQ_API_KEY,
model: "llama-3.3-70b-versatile",
});
export async function handleRun(job: {
channelName: string;
runId: string;
invocationId: string;
inputEventId: string;
userId: string;
prompt: string;
}) {
const session = createAgentSession({
client: realtimeClient(),
channelName: job.channelName,
});
const run = session.createRun({
runId: job.runId,
invocationId: job.invocationId,
inputEventId: job.inputEventId,
clientId: job.userId,
});
try {
const { runEndReason } = await runDirectLlm(run, provider, {
model: "llama-3.3-70b-versatile",
prompt: job.prompt,
maxOutputTokens: 800,
});
return runEndReason;
} finally {
session.close();
}
}
```
`ProviderTextRequest.model` is a required field in the TypeScript types, even when the provider was
created with a default `model`. A model set on the request overrides the provider default.
The remaining examples assume a `run` created the same way.
### OpenAI SDK
```ts title="worker/openai-agent.ts"
createOpenAISdkProvider,
runDirectLlm,
} from "@sockudo/ai-transport/providers";
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const chatProvider = createOpenAISdkProvider({
client: openai,
mode: "chat",
model: "gpt-4.1-mini",
});
await runDirectLlm(run, chatProvider, {
model: "gpt-4.1-mini",
messages: [
{ role: "system", content: "Answer as a production engineer." },
{ role: "user", content: "Why did reconnect recovery fail?" },
],
});
```
Use `mode: "responses"` for the Responses API. Provider-specific fields go in `body`:
```ts title="worker/openai-responses-agent.ts"
const responsesProvider = createOpenAISdkProvider({
client: openai,
mode: "responses",
model: "gpt-5-mini",
});
await runDirectLlm(run, responsesProvider, {
model: "gpt-5-mini",
prompt: "Explain the latest channel history page in plain English.",
body: {
reasoning: { effort: "low" },
},
});
```
### Anthropic SDK
```ts title="worker/anthropic-agent.ts"
createAnthropicSdkProvider,
runDirectLlm,
} from "@sockudo/ai-transport/providers";
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
const provider = createAnthropicSdkProvider({
client: anthropic,
model: "claude-sonnet-4-5",
system: "Be concise and include risk levels.",
});
await runDirectLlm(run, provider, {
model: "claude-sonnet-4-5",
prompt: "Audit this failed push notification delivery chain.",
});
```
The adapters are structural: they accept any object with the same `create(...)` streaming shape as
the official SDK clients, so the package does not import provider SDKs itself.
### Provider registry
Use a registry when the UI lets a user or tenant select a model provider.
```ts title="worker/providers.ts"
createAnthropicSdkProvider,
createDirectLlmProviderRegistry,
createOpenAICompatibleProvider,
} from "@sockudo/ai-transport/providers";
const anthropic = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });
export const providers = createDirectLlmProviderRegistry({
groq: createOpenAICompatibleProvider({
provider: "groq",
apiKey: process.env.GROQ_API_KEY,
model: "llama-3.3-70b-versatile",
}),
local: createOpenAICompatibleProvider({
provider: "ollama",
model: "llama3.2",
}),
anthropic: createAnthropicSdkProvider({
client: anthropic,
model: "claude-sonnet-4-5",
}),
});
const stream = await providers.streamText("local", {
model: "llama3.2",
prompt: "Return a JSON incident summary.",
});
```
`providers.get(name)` returns the provider (or `undefined`), which you can pass to `runDirectLlm`.
`providers.streamText` throws for an unknown name, so validate tenant input first.
## Custom provider
Any provider that returns `ReadableStream` can participate.
```ts title="worker/echo-provider.ts"
export const provider: DirectLlmProvider = {
async streamText(request) {
const words = (request.prompt ?? "").split(/\s+/);
return new ReadableStream({
start(controller) {
controller.enqueue({ type: "start" });
controller.enqueue({ type: "text-start", id: "answer" });
for (const word of words) {
controller.enqueue({
type: "text-delta",
id: "answer",
delta: `${word} `,
});
}
controller.enqueue({ type: "text-end", id: "answer" });
controller.enqueue({ type: "finish", finishReason: "stop" });
controller.close();
},
});
},
};
```
Emit `start` first and `finish` last, and give every text or reasoning part matching `*-start`,
`*-delta`, and `*-end` chunks with the same `id`.
## Tool calling and human approval
Tool calls are just streamed UI message chunks. The transport persists every tool-input delta and
the final tool state, so a user can approve from another tab or after a reconnect.
```mermaid
sequenceDiagram
participant A as Agent
participant S as Sockudo
participant W as Web client
participant T as Tool executor
A->>S: tool-input-start + deltas
S-->>W: pending approval appears
W->>S: approval response
T->>S: tool output
A->>S: continue ai-output append
```
For AI SDK 7 approval flows, treat the request and response as part of the transcript:
1. The agent streams a `tool-approval-request` chunk with a stable approval id and tool call id.
2. Sockudo stores that request on the `ai-output` mutable message and fans it out to every attached
client.
3. The client records an approval response on the assistant tool part. The Vercel chat transport
diffs the optimistic message overlay and publishes a `tool-approval-response` input.
4. The worker verifies the response, runs or denies the tool, then appends `tool-result`,
`tool-result-error`, or `output-denied` state.
Use channel auth and V2 capabilities to decide who can approve. If the provider supplies approval
signatures or automatic-approval markers, keep them with the approval chunk so the worker can verify
them before executing the tool.
Do not treat a client-rendered button click as sufficient authority by itself. The worker must check
that the approval came from a verified identity allowed to approve this tool call.
## Realtime voice and generated media
AI SDK realtime voice sessions are optimized for provider-native WebSocket or WebRTC media paths.
Sockudo should not proxy high-rate PCM frames, microphone buffers, or generated video bytes through
mutable-message appends. Use Sockudo as the durable control and transcript plane:
- publish session start, model, voice, and participant metadata
- append transcript deltas, final transcript text, tool calls, and approval state
- fan out presence for the user, agent, and handoff devices
- publish push notifications or completion events when the media job finishes
- store recordings, uploaded files, thumbnails, and generated videos in object storage, then publish
references as file, source, or custom chunks
This keeps reconnect, rewind, audit, and multi-device UI recovery in Sockudo without putting
latency-sensitive media on the same path as durable chat history.
Keep provider API keys server-side. Browser clients should only receive Sockudo public app keys,
private/presence auth responses, and short-lived Protocol V2 capability tokens.
## Next steps
- [AI Transport overview](/docs/server/ai-transport-overview): The run lifecycle and session channel model.
- [Token streaming and rollup](/docs/server/token-streaming-rollup): How fast provider streams are delivered to many devices.
- [Authentication](/docs/server/ai-transport-authentication): Issue the capability tokens your browser client needs.
- [Push notifications](/docs/server/ai-transport-push-notifications): Notify users when a long run finishes.
---
# Authentication for AI Transport (https://sockudo.io/docs/server/ai-transport-authentication)
> Capability tokens, claims, channel patterns, refresh, revocation, and server-authorized requests.
AI Transport trusts identity, never claims. Every `run-client-id` and `input-client-id` header, and
every own-scoped edit or append, is checked against an identity Sockudo verified itself. This page
shows how to issue that identity with short-lived Protocol V2 capability tokens, how to scope what
each user can do, and how to refresh and revoke tokens. You need it before letting real users
publish `ai-input` from a browser or mobile app.
## Three layers of authority
| Layer | Used for | Authority |
| --- | --- | --- |
| App-key HTTP auth | Server publishes, mutations, history reads, push management, revocation | Trusted backend with app secret |
| Private/presence channel auth | Subscription admission | Customer backend signs channel auth |
| Protocol V2 capability token | Client connection identity and channel-scoped capabilities | Customer backend issues short-lived JWT |
In practice: your **agent worker** uses the app key over HTTP, and your **browser or mobile client**
connects with a capability token that your backend minted for the signed-in user. Token auth is
additive; it never weakens the existing private and presence channel signature rules, so private
channels still need channel authorization.
Never trust `client_id` from message bodies or AI headers. Verified identity comes from signed-in
socket state, capability-token claims, or trusted server app-key context.
```mermaid
sequenceDiagram
participant C as Browser client
participant B as Your backend
participant S as Sockudo
C->>B: request token (session cookie)
B-->>C: JWT with client id and capabilities
C->>S: connect with token (protocol V2)
S-->>C: connected, identity verified
C->>S: publish ai-input on private-ai-user-42-...
S-->>C: sockudo:token_expired (40142)
C->>B: request fresh token
C->>S: sockudo:auth with new token
```
## Capability token claims
Capability tokens are HS256 JWTs signed with the app secret. Header `kid` must equal the app key.
Claims are:
```json
{
"x-sockudo-client-id": "user-42",
"x-sockudo-capability": {
"private-ai-user-42-*": [
"subscribe",
"publish",
"history",
"message_append_own"
]
},
"iat": 1764835200,
"nbf": 1764835200,
"exp": 1764838800,
"jti": "tok_01J..."
}
```
`x-sockudo-capability` accepts either the JSON object shown above or its stringified form. Each
object key is a channel pattern and its value is the list of operations granted on matching
channels. `exp`, `iat`, and `jti` are required; `nbf` is optional.
### Token limits
| Limit | Value | Result when exceeded |
| --- | --- | --- |
| Encoded token size | 8 KiB | Authentication fails. |
| Capability patterns | 100 | Authentication fails. |
| `x-sockudo-client-id` | 1 to 128 bytes | Authentication fails. |
| `jti` | 1 to 128 bytes | Authentication fails. |
| Lifetime (`exp` minus `iat`) | 24 hours | Authentication fails. |
| Clock skew tolerance | 30 seconds | Applied to `exp`, `nbf`, and future-dated `iat`. |
| Algorithm | HS256 only | Any other `alg` is rejected. |
### Operations
| Operation | Grants |
| --- | --- |
| `subscribe` | Subscribe to matching channels. |
| `publish` | Publish events, including `ai-input` and `ai-cancel` on AI channels. |
| `history` | Read channel history over the WebSocket. |
| `presence` | Join and update presence on matching presence channels. |
| `message_update_own`, `message_delete_own`, `message_append_own` | Mutate messages this client created. |
| `message_update_any`, `message_delete_any`, `message_append_any` | Mutate any message on matching channels. |
| `annotation-publish`, `annotation-subscribe`, `annotation-delete-own`, `annotation-delete-any` | Annotation operations, independent from `publish`/`subscribe`. |
| `push-admin`, `push-subscribe` | Channel-scoped push management. |
Message mutation operations accept their documented underscore spelling and equivalent hyphenated
spelling (`message-append-own`). Annotation and push operation names accept both spellings as well.
Any other operation name makes the token invalid.
A capability token replaces the connection's implicit permissions. `subscribe`, `publish`,
`history`, and `presence` are granted only on channels that match a listed pattern, so a token that
forgets `history` cannot read history, even on a channel it can subscribe to.
### Channel patterns
Patterns accept exact channel names, `*`, and wildcard matching as implemented by
`ConnectionCapabilities`: one `*` may appear at the end (`private-ai-user-42-*`), at the start
(`*-sess-01J`), or in the middle. Matching is case-sensitive. Keep user tokens narrow:
```json
{
"private-ai-user-42-*": [
"subscribe",
"publish",
"history",
"message_append_own",
"message_update_own",
"annotation-subscribe"
],
"presence-ai-user-42-*": ["subscribe", "presence"]
}
```
Put the user (and tenant) in the channel name, as above, so a single trailing wildcard cannot reach
another user's sessions.
## Issue tokens from your backend
1. Authenticate the user with your own session (cookie, OAuth, or similar). Never accept a user ID
from the request body.
2. Build the capability map for that user from server-side data.
3. Sign the JWT with the app secret and return it with its expiry.
4. Configure the client to call this endpoint for the first connection and every refresh.
### Node example
```ts title="lib/sockudo-token.ts"
function base64url(value: unknown): string {
return Buffer.from(JSON.stringify(value)).toString("base64url");
}
export function issueSockudoToken({
appKey,
appSecret,
clientId,
capabilities,
ttlSeconds = 3600,
now = Math.floor(Date.now() / 1000),
}: {
appKey: string;
appSecret: string;
clientId: string;
capabilities: Record;
ttlSeconds?: number;
now?: number;
}): { token: string; exp: number } {
const header = { alg: "HS256", typ: "JWT", kid: appKey };
const exp = now + ttlSeconds;
const payload = {
"x-sockudo-client-id": clientId,
"x-sockudo-capability": JSON.stringify(capabilities),
iat: now,
nbf: now,
exp,
jti: crypto.randomUUID(),
};
const signingInput = `${base64url(header)}.${base64url(payload)}`;
const signature = crypto.createHmac("sha256", appSecret).update(signingInput).digest("base64url");
return { token: `${signingInput}.${signature}`, exp };
}
```
```ts title="app/api/sockudo/token/route.ts"
// App code: returns the signed-in user's ID or throws.
export async function POST(req: Request) {
const userId = await requireUserId(req);
const result = issueSockudoToken({
appKey: process.env.SOCKUDO_APP_KEY!,
appSecret: process.env.SOCKUDO_APP_SECRET!,
clientId: userId,
capabilities: {
[`private-ai-${userId}-*`]: ["subscribe", "publish", "history", "message_append_own"],
[`presence-ai-${userId}-*`]: ["subscribe", "presence"],
},
});
return Response.json(result); // { token, exp }
}
```
### Rust example
```rust title="src/sockudo_token.rs"
use jsonwebtoken::{Algorithm, EncodingKey, Header, encode};
use serde::Serialize;
use std::collections::BTreeMap;
#[derive(Serialize)]
struct Claims {
#[serde(rename = "x-sockudo-client-id")]
client_id: String,
#[serde(rename = "x-sockudo-capability")]
capability: BTreeMap>,
iat: i64,
nbf: i64,
exp: i64,
jti: String,
}
fn issue_sockudo_token(
app_key: &str,
app_secret: &str,
client_id: &str,
capabilities: BTreeMap>,
now: i64,
jti: String,
) -> jsonwebtoken::errors::Result {
let mut header = Header::new(Algorithm::HS256);
header.kid = Some(app_key.to_owned());
encode(
&header,
&Claims {
client_id: client_id.to_owned(),
capability: capabilities,
iat: now,
nbf: now,
exp: now + 3600,
jti,
},
&EncodingKey::from_secret(app_secret.as_bytes()),
)
}
```
## Connect with a token
A V2 WebSocket connection carries its first token in the `token` query parameter. With
`@sockudo/client`, set `token` for a static token, or `authUrl` / `authCallback` to fetch one on
connect and on every refresh:
```ts title="src/realtime.ts"
export const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
protocolVersion: 2,
channelAuthorization: { endpoint: "/api/sockudo/channel-auth" },
authCallback: async ({ reason }) => {
// reason is "initial", "reconnect", "refresh", or "expired"
const res = await fetch("/api/sockudo/token", { method: "POST" });
return res.json(); // { token, exp }
},
});
```
Capability tokens require Protocol V2; a V1 connection cannot apply one.
## Refresh and revocation
Refresh a V2 connection with `sockudo:auth`:
```json
{
"event": "sockudo:auth",
"data": { "token": "" }
}
```
Refresh cannot change `client_id`: a token for a different identity is rejected and the existing
token stays in effect.
When a token reaches `exp`, Sockudo sends `sockudo:token_expired` with `code` `40142` and
`reason` `"expired"`. The client then has 30 seconds to send `sockudo:auth` with a fresh token;
otherwise the connection closes with code `4009`. Clients configured with `authUrl` or
`authCallback` refresh automatically.
To revoke tokens, call signed HTTP:
```http
POST /apps/{appId}/revocations
```
with `jti`, `client_id`, or both. Optional fields are `ttl_seconds` or `expires_at` (how long the
revocation is remembered; defaults to 24 hours) and `reason`. The response reports `revoked_jti`,
`revoked_client_id`, and `closed_connections`.
```json title="Request body"
{ "jti": "tok_01J...", "reason": "device lost" }
```
Matching local sockets receive `sockudo:token_expired` with `code` `40160` and `reason`
`"revoked"`, then close immediately with code `4009`. Revocations are stored in the cache, and
every new connection or refresh checks them, so a revoked token cannot be reused.
A `client_id` revocation rejects every token for that identity, including tokens you issue after
the revocation, until the revocation's TTL expires. Revoke by `jti` to end one session; revoke by
`client_id` only to lock out a user entirely, and choose `ttl_seconds` deliberately.
## Agents and server workers
Agent/server workers should use trusted app-key HTTP or a deliberately scoped server-issued token.
Agent events (`ai-output` and the `ai-run-*` lifecycle) are accepted only through trusted app-key
HTTP. HTTP mutations sent with a `socket_id` are evaluated against that socket's identity and
capabilities; mutations sent without `socket_id` are privileged server-side operations.
## Next steps
- [AI Transport overview](/docs/server/ai-transport-overview): See where tokens fit in the run lifecycle.
- [Conventions](/docs/server/ai-transport-conventions): Identity headers and the anti-spoof rule.
- [Security](/docs/server/security): Secrets, origin policy, and client message controls.
- [Troubleshooting](/docs/server/ai-transport-troubleshooting): Diagnose token loops and permission errors.
---
# Token streaming and rollup (https://sockudo.io/docs/server/token-streaming-rollup)
> AI Transport append rollup settings for high-rate token streams.
Append rollup coalesces bursts of `message.append` operations into fewer WebSocket frames, so a
model streaming hundreds of tokens per second does not send hundreds of frames per second to every
subscriber. This page explains what rollup changes, what it deliberately does not change, and how
to tune it. You need it when you stream fast models to many devices, or when you are diagnosing
added latency or low fanout savings.
## Why rollup exists
An agent typically appends one small fragment per model chunk. At 200 tokens per second with five
subscribed devices, that is 1,000 tiny frames per second for a single answer, most of which a UI
renders faster than a person can read. Rollup holds appends for a short window and delivers them as
one frame. At the default 40 ms window, a steady stream is capped near 25 deliveries per second per
stream, and no text is lost.
## What rollup never changes
Sockudo persists every versioned `message.append` operation exactly as received. Append rollup only
changes WebSocket egress: the version store, durable history, recovery, and latest-message reads
still observe the unrolled mutation log.
```mermaid
flowchart LR
A[Agent append] --> V[Version store]
V --> H[Durable history]
V --> W[Webhooks and push]
V --> R[Rollup engine]
R --> C[Live subscribers]
```
Rollup sits at the very end of the pipeline, after everything durable has already happened. That
placement is what makes it safe:
- **Correctness is the reduced state.** A client that folds every original append and a client that
received coalesced frames end with the identical message, because rollup only merges adjacent
fragments of one stream.
- **Recovery never depends on a buffer.** A reconnecting or late-joining subscriber rebuilds state
from history and the version store, never from in-memory rollup state. A node can drop its rollup
buffers (restart, crash) without losing data.
- **Accounting stays honest.** Billing, rate limits, durable history, version storage, webhooks,
and push accounting count original create/update/delete/append requests, so coalescing cannot hide
ingress load.
## Configure rollup
Enable the `ai-transport` Cargo feature and runtime AI Transport config before using rollup:
```toml title="config.toml"
[ai_transport]
enabled = true
[[ai_transport.channels]]
prefix = "private-ai-"
[ai_transport.rollup]
enabled = true
default_window_ms = 40
min_window_ms = 0
max_window_ms = 500
orphan_ttl_ms = 60000
wheel_tick_ms = 5
shards = 64
```
| Key | Default | Meaning |
| --- | --- | --- |
| `enabled` | `true` | Turns egress coalescing on for AI channels. |
| `default_window_ms` | `40` | Coalescing window applied to channel fanout. |
| `min_window_ms` | `0` | Smallest window a client may request. |
| `max_window_ms` | `500` | Largest window a client may request. |
| `orphan_ttl_ms` | `60000` | How long a `streaming` message may go without a terminal update before the orphan janitor cancels it. |
| `wheel_tick_ms` | `5` | Scheduler tick interval. |
| `shards` | `64` | Number of scheduler shards. |
## Choose a window
The WebSocket query parameter `append_rollup_window` is accepted only for Protocol V2 and must be
one of the locked values below, and within `[min_window_ms, max_window_ms]`. Any other value fails
the WebSocket upgrade with HTTP `400`.
| `append_rollup_window` | Behavior |
| --- | --- |
| `0` | Disable coalescing; every append fan-out counts individually. |
| `20` | Short coalescing window for lower added latency. |
| `40` | Default; caps a steady stream near 25 deliveries per second. |
| `100` | Heavier coalescing for overloaded subscribers. |
| `500` | Maximum supported coalescing window. |
Current server v1 semantics use server-wide per `(app_id, channel, message_serial)` coalescing at
`[ai_transport.rollup].default_window_ms`. The query parameter is validated for SDK compatibility
but does not allocate per-subscriber rollup state. Tune latency and savings with
`default_window_ms`.
With the JavaScript SDK, pass the window and append delivery mode when you create the client:
```ts title="src/realtime.ts"
export const realtime = await createSockudoRealtimeClient("app-key", {
clientOptions: { wsHost: "realtime.example.com", forceTLS: true },
appendRollupWindow: 40,
appendMode: "delta",
});
```
`append_mode=delta` (the V2 default) delivers live `sockudo:message.append` frames whose `data` is
only the new fragment. `append_mode=full` delivers the accumulated message on every live append.
Both modes reduce to the same final message.
## How coalescing works
1. **The first append is immediate.** A new stream's first append is delivered without waiting, so
the user sees output as soon as the model produces it.
2. **Later appends inside the window are held.** Subsequent appends to the same
`(app_id, channel, message_serial)` are merged: the delivered frame carries the latest append's
envelope and the concatenation of every held fragment, so delta subscribers receive all text.
3. **The window closes and one frame goes out.** When the deadline passes, the merged append is
fanned out to subscribers.
4. **Terminal operations flush first.** An append whose `extras.ai.transport.status` is `complete`
or `cancelled`, and any `message.update` or `message.delete`, flushes pending append state before
delivering the terminal operation. A terminal operation can never overtake text that preceded it.
## Scheduler internals
The scheduler keeps sharded flush and orphan deadline heaps. A tick peeks one heap per shard and
only pops due entries; it does not scan active streams. Generation tokens make rescheduled and
terminally removed deadlines harmless. The first append retains timing metadata only after its
immediate fanout. Pending payload state is allocated when a later append actually needs
coalescing.
Deadline discovery and delivery are two phases. The worker enters the existing per-channel publish
ordering gate before claiming a generation token, so a concurrent terminal mutation either flushes
the pending append itself or waits behind the claimed delivery. Deferred context retains the app,
channel, excluded socket/echo decision, full-message versus delta choice, original envelope, and
continuity position.
The worker processes at most 4,096 due or retry entries per tick and at most 16 channels
concurrently. Messages within one channel remain sequential. App lookup and ordering-gate
backpressure use a bounded 4,096-entry retry queue. Once a due delivery is claimed, its three
fanout attempts stay inside the same channel ordering permit so a terminal mutation cannot
overtake a retry. Exhaustion is logged with `outcome="reset_required"`; it is never treated as a
successful delivery. Scheduler deadlines use a monotonic runtime clock and are unaffected by
wall-clock corrections. Graceful shutdown stops deadline intake, drains pending content in bounded
chunks, finishes bounded retries, and only then allows connection teardown.
## Metrics
Prometheus exposes low-cardinality rollup metrics per app:
| Metric | What it tells you |
| --- | --- |
| `sockudo_appends_received_total` | Original appends received by rollup (ingress). |
| `sockudo_appends_delivered_total` | Append deliveries emitted after coalescing (egress). |
| `sockudo_rollup_ratio` | Input appends represented by one rollup output. Higher means more savings. |
| `sockudo_active_streams` | Streams currently tracked by the rollup engine. |
| `sockudo_flush_latency` | Rollup flush latency in milliseconds. Compare it to the configured window. |
`sockudo_active_streams` counts rollup streams; `sockudo_ai_active_streams` counts AI lifecycle
streams from the observability tracker. Both should return to baseline once runs finish.
Rollup metrics count both original append receipt and coalesced egress delivery, so operators can
see the reduction ratio without hiding ingress load. Active-stream gauges are changed by exact
insertion/removal deltas. Publishing a gauge does not lock or scan scheduler shards.
If `sockudo_flush_latency` is close to your latency budget, lower `default_window_ms`; if
`sockudo_rollup_ratio` stays near `1` under heavy streams, raise it.
## Edge cases
- A terminal append flushes pending state before delivering the terminal operation.
- `message.update` and `message.delete` flush pending append state first.
- A late subscriber reconstructs state from history/version storage, not from rollup buffers.
- A node that sees a stale stream after `orphan_ttl_ms` claims it through shared cache and appends a
normal cancellation update; the latest version is re-read before mutation to avoid cancelling a
stream that advanced on another node.
- `append_rollup_window=0` disables egress coalescing but does not change persistence or rate-limit
behavior.
## Benchmarks and load tests
The scheduler exposes `RollupEngine::tracking_overhead_bytes_per_stream()` for the inline tracking
footprint, excluding key and payload allocations. The permanent Criterion matrix measures 2,000
and 50,000 independent streams for empty tick, one-percent due, all due, and terminal-storm
workloads:
```bash
cargo bench -p sockudo-ai-transport --bench rollup_engine -- --noplot
```
`rollup_scheduler::at_least_999_per_mille_due_by_window_plus_five_ms_under_documented_load`
locks the deadline boundary at a 40 ms window, 5 ms tick allowance, and 2,000 independent streams.
Run `scripts/ai-rollup-load-test.mjs` against a local server to exercise a synthetic `200 tok/s`
stream and inspect delivery rate, final content, and append mutation latency.
## Next steps
- [AI Transport overview](/docs/server/ai-transport-overview): The run lifecycle and the session channel model.
- [Production operations](/docs/server/ai-transport-production-ops): Alert thresholds for flush latency and rollup ratio.
- [Limits and quotas](/docs/server/ai-transport-limits-quotas): Append counts, accumulated bytes, and open-stream caps.
- [Mutable messages](/docs/server/mutable-messages): How appends are aggregated and stored.
---
# Agent Presence (https://sockudo.io/docs/server/agent-presence)
> Track AI agent status with Protocol V2 presence updates and optional disconnect grace.
Agent presence lets every device in a session see what the agent is doing right now (idle,
thinking, or streaming) and whether it is still connected. It uses ordinary presence channels plus
one V2 frame, `sockudo:presence_update`, which changes a member's data in place instead of making
it leave and re-join. Use it to drive typing indicators, "agent is thinking" spinners, and
handoff UIs.
## How it works
A presence channel tracks who is subscribed and carries a small JSON payload per member. With
Protocol V1, the only way to change that payload is to leave and re-enter the channel, which every
other member sees as a departure followed by an arrival. `sockudo:presence_update` replaces the
payload for an existing member without that flap.
```mermaid
sequenceDiagram
participant A as Agent worker
participant S as Sockudo
participant U as User devices
A->>S: subscribe presence-ai-session-123
S-->>U: member_added
A->>S: sockudo:presence_update (thinking)
S-->>U: presence update (thinking)
A->>S: sockudo:presence_update (streaming)
S-->>U: presence update (streaming)
Note over S,U: late joiners get latest data in the snapshot
```
## Status convention
Sockudo does not interpret member data. The SDKs and examples use a `status` field:
```json
{ "status": "idle" }
{ "status": "thinking" }
{ "status": "streaming" }
```
Keep the payload small. Put run identifiers or progress details in channel events, not in presence
data.
## Send an update
1. Connect with Protocol V2 and subscribe to a `presence-` channel. Capability-token connections
need the `presence` operation for that channel.
2. Wait until the subscription succeeds; the connection must already be a member of the presence
channel before it can update its data.
3. Send the update frame with the channel name and the new member data:
```json
{
"event": "sockudo:presence_update",
"data": {
"channel": "presence-ai-session-123",
"data": { "status": "thinking" }
}
}
```
Other V2 members receive a `sockudo_internal:presence_update` event whose data contains `user_id`
and the new `user_info`. If presence history is enabled for the channel, the transition is
recorded there too, and apps subscribed to the `member_updated` webhook are notified.
Late joiners see the latest member data in the presence snapshot returned with
`subscription_succeeded`.
## Limits and access
| Rule | Behavior when violated |
| --- | --- |
| Protocol V2 only. | Rejected with `104009` `ai_presence_update_invalid`. |
| The target must be a `presence-` channel, and the connection must already be a member. | Rejected with `104009`. |
| Capability-token connections need `presence` permission for the channel. | Rejected as a capability failure. |
| Member `data` is capped at 1 KiB (1024 bytes, JSON-encoded). | Rejected with `40009` `payload_too_large`. |
| Updates are rate limited per app/channel/member by `[presence].update_rate_limit_per_member_per_second`, default `10`. | Rejected with the client-event rate-limit error. |
V1/Pusher clients do not receive update events.
## Disconnect grace
Agents and mobile clients drop connections for a few seconds all the time. Presence can hold a
member during a short grace window so a brief network loss does not look like the agent left.
| Setting | Default | Applies to |
| --- | --- | --- |
| `[presence].ungraceful_timeout_seconds` | `0` | Protocol V1 abnormal disconnects. `0` preserves legacy immediate `member_removed` timing. |
| `[presence].v2_ungraceful_timeout_seconds` | `15` | Protocol V2 abnormal disconnects, so transient transport loss does not produce a leave/enter flap. |
Clean unsubscribe and clean WebSocket close still remove immediately. A V2 client that resumes
within the window keeps its membership without a flap.
```toml title="config.toml"
[presence]
update_rate_limit_per_member_per_second = 10
ungraceful_timeout_seconds = 0
v2_ungraceful_timeout_seconds = 15
```
Presence update ordering is independent from channel message ordering. Treat presence status as
latest-known member state, not as a causal marker for `ai-output` or mutable-message delivery. To
know whether a reply is finished, read the output message's `status` or the `ai-run-end` event.
## Next steps
- [AI Transport overview](/docs/server/ai-transport-overview): Where presence fits in the session channel model.
- [Presence history](/docs/clients/presence-history): Read historical join, leave, and update transitions.
- [Authentication](/docs/server/ai-transport-authentication): Grant the presence capability to the right members.
- [Troubleshooting](/docs/server/ai-transport-troubleshooting): Fix presence flaps and rejected updates.
---
# Push notifications for AI Transport (https://sockudo.io/docs/server/ai-transport-push-notifications)
> Wake users when an AI agent finishes while they are away.
Long agent runs often outlive the user's attention: they ask a question, lock their phone, and come
back later. This page shows two ways to send a push notification when the answer is ready, both
built on Sockudo's existing push pipeline. You need it when runs take longer than a user will keep
the app open.
AI Transport uses normal Sockudo channel events for lifecycle and output. Push rules let those same
channel publishes wake devices through the existing push notification pipeline.
## Choose a recipe
| Recipe | How it triggers | Choose it when |
| --- | --- | --- |
| Channel push rule | The agent publishes an ordinary event (for example `agent-complete`) to a notification channel; a `[[push_rules]]` entry turns it into a push. | You want zero backend code and static notification text. |
| Webhook recipe | The `ai_run_ended` webhook reaches your backend, which calls the push publish API. | You need enrichment, quiet hours, or a database lookup before notifying. |
Both recipes deliver through the same durable push status, fanout, queue, retry, and feedback
transitions as any other push publish.
```mermaid
sequenceDiagram
participant A as Agent worker
participant S as Sockudo
participant P as Push provider
participant D as User device
A->>S: ai-run-end (run-reason=complete)
A->>S: agent-complete on notifications channel
S->>S: push rule matches channel and event
S->>P: provider request (FCM, APNs, ...)
P-->>D: notification
D->>S: open session, read history
```
## Channel rule recipe
1. Build Sockudo with the `push` feature and configure at least one provider. See
[Push notifications](/docs/server/push-notifications) for provider credentials.
2. Add a rule that watches each user's notification channel:
```toml title="config.toml"
[[push_rules]]
enabled = true
channel_pattern = "notifications:*"
event_filter = ["agent-complete"]
rate_limit_per_second = 100
[push_rules.payload_mapping]
title_field = "title"
body_field = "body"
template_data_field = "data"
include_remaining_fields = true
```
3. Register each device and subscribe it to the same channel the app is allowed to read:
```http
POST /apps/{appId}/push/channelSubscriptions
```
4. Have the long-running agent publish a normal event when the answer is ready:
```json
{
"name": "agent-complete",
"channel": "notifications:user-123",
"data": {
"title": "Agent finished",
"body": "Your answer is ready",
"sessionId": "sess_123"
}
}
```
Sockudo maps `title` and `body` into the push notification and copies the remaining fields into
provider data under `data`. The push target is `Channel { channel: "notifications:user-123" }`, so
existing channel subscriptions determine which devices receive FCM, APNs, Web Push, HMS, or WNS
delivery.
Rule details:
- `channel_pattern` accepts exact names, `*`, or a single trailing wildcard such as
`notifications:*`.
- `event_filter` is a non-empty allowlist of at most 32 event names, each at most 200 bytes.
- `rate_limit_per_second` defaults to `100` per rule.
- Rules are validated at startup, and a publish that matches no rule does no push work.
A channel such as `notifications:user-123` belongs to the `notifications` namespace. Sockudo
rejects colon-delimited channel names unless the app defines that namespace; see
[Channel namespaces](/docs/reference/configuration). Alternatively use a hyphenated name such as
`private-notifications-user-123` with a matching `channel_pattern`.
When the user taps the notification, the client should load `sessionId`, attach to the realtime
session channel, and read history/recovery state for the authoritative transcript. The notification
is a pointer, not the answer.
## Webhook recipe
For teams that keep push orchestration in their backend, enable the `ai_run_ended` webhook. Its
payload carries `channel`, `run_id`, `reason`, and an optional `error_code`. On
`reason = "complete"`, the backend can call the existing push publish API
(`POST /apps/{appId}/push/publish`):
```json
{
"recipients": [
{ "type": "channel", "channel": "notifications:user-123" }
],
"payload": {
"title": "Agent finished",
"body": "Your answer is ready",
"templateData": {
"data": { "sessionId": "sess_123" }
}
},
"sync": false
}
```
With the Node server SDK, the same handler looks like this:
```ts title="app/api/sockudo/webhooks/route.ts"
// App code: maps a session channel to the user who owns it.
const sockudo = new Sockudo({
appId: process.env.SOCKUDO_APP_ID!,
key: process.env.SOCKUDO_APP_KEY!,
secret: process.env.SOCKUDO_APP_SECRET!,
host: "realtime.example.com",
useTLS: true,
});
export async function POST(req: Request) {
const rawBody = await req.text();
const webhook = sockudo.webhook({
headers: Object.fromEntries(req.headers),
rawBody,
});
if (!webhook.isValid()) return new Response("invalid signature", { status: 401 });
for (const event of webhook.getEvents()) {
if (event.name !== "ai_run_ended" || event.reason !== "complete") continue;
const userId = await ownerOfChannel(event.channel);
await sockudo.publishPush({
recipients: [{ type: "channel", channel: `notifications:${userId}` }],
payload: {
title: "Agent finished",
body: "Your answer is ready",
templateData: { data: { sessionId: event.channel } },
},
});
}
return new Response(null, { status: 200 });
}
```
This path is useful when the backend enriches notifications, applies product-specific quiet hours,
or joins run metadata with an application database before sending push. Webhooks are delivered at
least once, so make the handler idempotent (for example by keying on `run_id`).
## Authorization
Use capability patterns so clients can subscribe only to their own notification channels, for
example `notifications:user-123`, while trusted backends or agent workers publish `agent-complete`.
Do not accept `client_id` from the message body when choosing a notification channel; derive the
channel from verified auth state or server-side session ownership.
Anyone who can publish `agent-complete` to a user's notification channel can send that user a push.
Grant `publish` on notification channels only to the agent's app-key HTTP path, never in a client
capability token.
## Verification surface
The recipe relies on existing push device registration, channel subscriptions, channel publish
targets, scheduled/status handling, delivery feedback, retries, and dead-letter behavior. Push
rules add only the channel-to-push trigger; accepted work still goes through the same durable
status, fanout, queue, retry, and feedback transitions as explicit push publishes.
## Next steps
- [Push notifications](/docs/server/push-notifications): Providers, device registration, publish status, and retries.
- [Webhooks](/docs/server/webhooks): Configure endpoints and validate ai_run_ended deliveries.
- [Authentication](/docs/server/ai-transport-authentication): Scope push-subscribe and publish capabilities per user.
- [AI Transport overview](/docs/server/ai-transport-overview): Reopen the session channel after the notification tap.
---
# Limits and quotas (https://sockudo.io/docs/server/ai-transport-limits-quotas)
> Canonical AI Transport, mutable-message, history, presence-history, push, and capability-token limits.
This is the canonical server-side limit table for AI Transport and the features it builds on. Every
row lists the config key (or a fixed constant), the default, and what a client sees when it hits
the limit. Use it when you size a deployment, pick retention windows, or decode a rejection.
Limits fall into three groups:
- **Hard rejections** stop a publish, mutation, or connection with an error code. The client must
change what it sends.
- **Clamps** silently reduce a request, such as a page size, to the allowed maximum.
- **Retention windows** decide how far back history, versions, and recovery can reach. Exceeding
them shows up later as a recovery or history gap, not as an immediate error.
## AI Transport
| Limit | Key or constant | Default | When exceeded |
| --- | --- | --- | --- |
| Accumulated content per output message | `ai_transport.max_accumulated_message_bytes` | `1048576` (1 MiB) | Append rejected with `40009` `payload_too_large`. Start a new message. |
| Appends per message | `ai_transport.max_appends_per_message` | `4096` | Append rejected with `40009`. |
| Open streaming messages per channel | `ai_transport.max_open_streaming_messages_per_channel` | `1024` | New streaming create rejected with `40009`. |
| Keys per `transport` or `codec` tier | constant | `32` | Publish rejected with `104001` `ai_header_too_large`. |
| Header key length | constant | `64` bytes | Publish rejected with `104001`. |
| Transport value length | `ai_transport.max_transport_value_bytes` | `256` bytes | Publish rejected with `104001`. Can only be raised. |
| `steer-codec-message-ids` value length | `ai_transport.max_steer_codec_message_ids_bytes` | `2048` bytes | Publish rejected with `104001`. Can only be raised. |
| Ordinary codec value length | constant | `256` bytes | Publish rejected with `104001`. |
| `extras.ai.codec.providerMetadata` length | constant | `8` KiB | Publish rejected with `104001`. |
| `message_id` on AI channels | constant | `64` bytes | HTTP publish rejected with `104001`; an empty value is rejected with `104000`. |
| Rollup windows | `append_rollup_window` | `0`, `20`, `40`, `100`, `500` | Any other value fails the V2 WebSocket upgrade with HTTP `400`. |
| Rollup default | `ai_transport.rollup.default_window_ms` | `40` | Node-local egress coalescing window. |
| Rollup orphan TTL | `ai_transport.rollup.orphan_ttl_ms` | `60000` | Stale stream claim/cancel threshold. The stream is closed with `status=cancelled` and `error-code=orphan_timeout`. |
The three stream caps are checked inside the same version-store transaction as the mutation, so
they hold across nodes and cannot be bypassed by racing appends.
## Events, history, and versions
| Limit | Key | Default | When exceeded |
| --- | --- | --- | --- |
| HTTP request body | `http_api.request_limit_in_mb` | `10` | Whole request cap; the request is rejected. |
| Event payload | `event_limits.max_payload_in_kb` | `100` | Publish rejected. App policy can lower the effective limit. |
| Event batch | `event_limits.max_batch_size` | `10` | Batch publish rejected. |
| Version history page | `versioned_messages.max_page_size` | `100` | Clamped; the server caps version pages. |
| Version retention | `versioned_messages.retention_window_seconds` | `0` | `0` means no expiry. Purged chains fall back to the raw history payload. |
| Version purge batch | `versioned_messages.purge_batch_size` | `1000` | SQL/memory purge worker bound. |
| History page | `history.max_page_size` | `100` | Clamped. Client history over the WebSocket is additionally capped at `1000`. |
| History retention | `history.retention_window_seconds` | `86400` | Older positions fail recovery with `position_expired`. Per-app/namespace policy can override. |
| History writer queue | `history.writer_queue_capacity` | `4096` | Bounded writer queue. |
| Presence history page | `presence_history.max_page_size` | `100` | Presence transition history, not current members. |
| Presence history retention | `presence_history.retention_window_seconds` | `86400` | Durable or memory retention. |
## Presence
| Limit | Key or constant | Default | When exceeded |
| --- | --- | --- | --- |
| `sockudo:presence_update` member data | constant | `1024` bytes | Update rejected with `40009`. |
| Presence updates per member | `presence.update_rate_limit_per_member_per_second` | `10` | Update rejected by the client-event rate limiter. |
| V2 disconnect grace | `presence.v2_ungraceful_timeout_seconds` | `15` | After the grace window, the member is removed. |
| V1 disconnect grace | `presence.ungraceful_timeout_seconds` | `0` | Legacy immediate removal. |
## Capability tokens
| Limit | Key or constant | Default | When exceeded |
| --- | --- | --- | --- |
| Capability token size | constant | `8192` bytes | Authentication fails. HS256 JWT only. |
| Capability patterns per token | constant | `100` | Authentication fails. |
| Capability token lifetime | constant | `24h` | `iat` to `exp`; longer tokens fail authentication. |
| Capability `client_id` | constant | `128` bytes | Required; authentication fails outside `1..=128`. |
| Capability `jti` | constant | `128` bytes | Required; authentication fails outside `1..=128`. |
| Clock skew | constant | `30` seconds | Tolerance applied to `exp`, `nbf`, and `iat`. |
| Expiry grace | constant | `30` seconds | After `sockudo:token_expired`, the connection closes with `4009` unless refreshed. |
| Revocation memory | `ttl_seconds` or `expires_at` on the request | `24h` | Revoked identities stay blocked for this long. |
## Push
| Limit | Key | Default | Notes |
| --- | --- | --- | --- |
| Push fanout fast path | `push.fanout_fast_threshold` | `10000` | Planning threshold. |
| Push shard size | `push.fanout_shard_size` | `100000` | Channel fanout shard size. |
| Push acceptance quota | `push.default_quotas.acceptance_rps` | `100` | Per-app default quota. |
| Push inflight quota | `push.default_quotas.inflight_max` | `1000` | Per-app default. |
| Push status retention | `push.publish_status_ttl_days` | `30` | Support/audit retention. |
| Push rule event filter | `push_rules.event_filter` | max `32` names | Each event name at most 200 bytes. Violations fail startup validation. |
| Push rule rate | `push_rules.rate_limit_per_second` | `100` | Per rule. |
Recovery and rewind can only reach what history still retains. Set `history.retention_window_seconds`
to cover your longest expected offline gap plus support investigations, and keep
`versioned_messages.retention_window_seconds` at least as long so history can still materialize the
latest aggregate of each output message.
The machine-checked default block is in [Configuration reference](/docs/reference/configuration).
## Next steps
- [Configuration reference](/docs/reference/configuration): Every key with its machine-checked default.
- [Error reference](/docs/reference/errors): HTTP and WebSocket error codes in full.
- [Troubleshooting](/docs/server/ai-transport-troubleshooting): What to do when a limit rejects traffic.
- [Production operations](/docs/server/ai-transport-production-ops): Capacity formulas and alert thresholds.
---
# Production checklist (https://sockudo.io/docs/server/ai-transport-production-checklist)
> Preflight checklist for AI Transport deployments.
Use this checklist before enabling AI Transport for production traffic. It is grouped by area so
each owner (platform, security, operations, client teams) can sign off their section. Every item
links a requirement to something you can verify; if an item does not apply, write down why.
## Build and config
- [ ] Build with `v2`, `ai-transport`, the selected shared adapter/cache/store features, and `push`
when push recipes are used. For example:
`cargo build -p sockudo --features "v2,ai-transport,redis,postgres,push"`.
- [ ] Build Docker images both with and without `ai-transport` enabled, and confirm the default
profile carries no AI Transport feature.
- [ ] Enable `[history]`, `[versioned_messages]`, and `[ai_transport]`. Startup rejects AI Transport
without the first two.
- [ ] Use shared history, version store, and cache for horizontal deployments. Startup rejects
memory history, a memory version store, or a `memory`/`none` cache with any adapter other than
`local`.
- [ ] Scope `[[ai_transport.channels]]` to narrow prefixes, and confirm every session channel name
actually starts with one (prefix matching is literal).
- [ ] Set history and version retention to cover reconnect, rewind, and support windows. Keep
version retention at least as long as history retention so history can materialize each output's
latest aggregate.
- [ ] Review `[ai_transport]` stream caps (`max_accumulated_message_bytes`,
`max_appends_per_message`, `max_open_streaming_messages_per_channel`) against your longest
expected answer.
- [ ] Choose `ai_transport.rollup.orphan_ttl_ms` longer than your longest legitimate pause inside a
stream. `config/ai-transport.example.toml` is the release config example.
## Auth
- [ ] Issue short-lived V2 capability tokens from a backend only.
- [ ] Keep capability patterns exact or tenant/user scoped.
- [ ] Use app-key HTTP only from trusted server workers. Agent events are accepted only there.
- [ ] Revoke by `jti` or `client_id` for incident response, and know that a `client_id` revocation
also blocks new tokens until its TTL expires.
- [ ] Do not accept client-asserted identity fields. Derive the user in your `/api/chat` route from
your own session, not from the `clientId` in the request body.
- [ ] Grant `publish` on push notification channels only to server-side publishers.
## Operations
- [ ] Scrape Prometheus metrics from every node.
- [ ] Alert on history degraded/reset-required channels, recovery failures, rollup flush latency,
active stream leaks, horizontal transport drops, and push queue/provider failures. Starting
thresholds are in [production operations](/docs/server/ai-transport-production-ops).
- [ ] Subscribe a webhook endpoint to `ai_run_ended` and `ai_stream_orphaned` if you persist
transcripts or monitor orphaned streams.
- [ ] Execute `tests/load` and `tools/chaos` profiles on production-like hardware before headline
claims.
- [ ] Keep `[ai_transport]` runtime-disabled during a mixed-version rolling upgrade on shared Redis.
Enable it only after every node runs the new release, then run an AI stream and recovery smoke
test.
## Release evidence
1. Run the AIT-S conformance suite before release:
```bash
AIT_CONFORMANCE_OFFLINE=1 scripts/ai-conformance-node.sh
```
2. Run the benchmark guard and review budgets:
```bash
scripts/ai-transport-bench-guard.sh
```
3. Run the deterministic gate on every release-candidate branch:
```bash
scripts/ai-transport-ga-gate.sh ci-evidence
```
4. Generate release evidence with `scripts/ai-transport-s14-release-evidence.sh`,
`scripts/ai-transport-rolling-upgrade-redis.mjs --execute`, and
`scripts/sdk-compat-full-matrix.mjs --execute`.
5. Run `scripts/ai-transport-ga-gate.sh release-evidence` before any GA tag; it fails until the
external S14 scale/chaos, rolling-upgrade, and full SDK compatibility manifests are committed.
Capacity claims must come from the runner output with hardware notes, Prometheus snapshots,
transcript audits, serial audits, and leak evidence. The committed `*.not-run.json` manifests under
`docs/specs/ai-transport-results` are explicit placeholders, not results.
## Client behavior
- [ ] Treat Sockudo history as authoritative after reconnect or notification open.
- [ ] Use `until_attach` for late-join history to avoid gaps.
- [ ] Reduce mutable-message events by `message_serial` and `version.serial`.
- [ ] Send terminal stream status for completed/cancelled/error runs, and call `run.end(...)` on
every code path, including errors.
- [ ] Refresh tokens before expiry and back off on auth errors. Configure `authCallback` or
`authUrl` so refresh after `sockudo:token_expired` is automatic.
- [ ] Handle `position_expired` and `stream_reset` by resubscribing and reloading history.
## GA readiness
The canonical readiness record is
[`docs/specs/ai-transport-ga-readiness.md`](https://github.com/sockudo/sockudo/blob/master/docs/specs/ai-transport-ga-readiness.md).
Server releases ship AI Transport defaults off before SDK releases that depend on the feature. Full
product parity still depends on the SDK plans and enablement work listed in that record.
## Next steps
- [Production operations](/docs/server/ai-transport-production-ops): Capacity model, required runs, and alert thresholds.
- [Limits and quotas](/docs/server/ai-transport-limits-quotas): Every limit and default in one table.
- [Authentication](/docs/server/ai-transport-authentication): Token issuance, refresh, and revocation.
- [Troubleshooting](/docs/server/ai-transport-troubleshooting): Symptom, cause, and fix for common failures.
---
# AI Transport production operations (https://sockudo.io/docs/server/ai-transport-production-ops)
> Runbooks, capacity formulas, alerts, chaos playbooks, and troubleshooting for AI Transport at scale.
This runbook is for the people who operate AI Transport: how to size a fleet, which load and chaos
runs prove it works, which signals to alert on, and what to do when they fire. It covers the S14
scale, soak, and chaos validation path for AI Transport deployments. Use it with the profiles in
`tests/load/profiles`, the runner in `tests/load/ai-scale-runner.mjs`, and the chaos harness in
`tools/chaos`.
## Capacity model
Start with these formulas, then replace the inputs with measured numbers from your hardware:
| Dimension | Formula |
| --- | --- |
| Connections per node | `target_connections / node_count`, with 20 percent headroom reserved |
| Active streams per node | `active_streams / node_count`, with shard imbalance budgeted at p95 |
| Append ingress rate | `active_streams * tokens_per_second` |
| Fanout egress rate | `append_ingress_rate * mean_subscribers_per_session` |
| Rollup memory ceiling | `active_streams_per_node * 4 KiB`, excluding payload buffers |
| History bytes per response | `mean_response_tokens * mean_token_bytes * retention_multiplier` |
| History store write rate | `append_ingress_rate + create/update/delete/summary_rate` |
Two consequences of the design shape these numbers:
- **Ingress is never reduced by rollup.** Every append is persisted, so the history store write rate
scales with tokens per second, not with deliveries.
- **Egress is.** Rollup caps a steady stream near 25 deliveries per second at the default 40 ms
window, so fanout egress grows with subscribers, not with model speed.
For example, 10,000 active streams at 100 tokens per second is 1,000,000 appends per second of
ingress, and the history store must absorb at least that write rate.
The headline target is 1M concurrent connections, 50k active streams, 100 tok/s, and one to five
subscriber devices per session for a 30-minute run. The 24-hour soak target is 20 percent of that
load.
## Required runs
1. Start the five-node local stack:
```bash
docker compose -f docker-compose.ai-transport.yml -f tests/load/docker-compose.ai-transport-scale.yml up -d --build
```
2. Run the smoke profile against it:
```bash
node tests/load/ai-scale-runner.mjs --profile tests/load/profiles/smoke.json
```
3. Generate a production fleet plan:
```bash
node tests/load/ai-scale-runner.mjs --profile tests/load/profiles/headline-1m.json --plan
```
4. Execute only on prepared hardware:
```bash
node tests/load/ai-scale-runner.mjs \
--profile tests/load/profiles/headline-1m.json \
--execute \
--urls "$SOCKUDO_NODE_URLS" \
--metricsUrls "$SOCKUDO_METRICS_URLS" \
--output docs/specs/ai-transport-results/headline-1m.json
```
Profiles with `planOnly: true` are protected from accidental laptop execution; passing `--execute`
is the explicit operator action that starts the workload. The other committed profiles are
`reconnect-storm.json`, `cancel-storm.json`, and `soak-20pct.json`.
The required pass criteria are zero sampled transcript mismatches, clean serial monotonicity,
p99 append-to-delivery under 25 ms intra-region, flat memory after warm-up, and no file descriptor
or Tokio task leaks.
Store raw result JSON under `docs/specs/ai-transport-results` only when it comes from the runner,
with hardware notes, Prometheus snapshots, transcript and serial audit summaries, and leak evidence.
Never replace the committed `*.not-run.json` placeholders with synthetic numbers.
## Chaos playbooks
Run all local chaos scenarios:
```bash
tools/chaos/ai-chaos-runner.sh all
```
The harness writes JSON artifacts to `target/ai-chaos` by default; set `RESULT_DIR` to keep them
elsewhere.
### Node kill mid-stream
Confirm post-heal transcript audits pass, `sockudo_ai_active_streams` returns to the expected
level, and janitor cleanup leaves no unbounded active rollup stream growth.
### Redis restart or failover
Expect a bounded stall, no versioned-message corruption, and no durable history reset-required
channels. Watch `sockudo_horizontal_transport_reconnections_total`,
`sockudo_horizontal_transport_messages_dropped_total`, `sockudo_history_degraded_channels`, and
`sockudo_history_reset_required_channels`.
### Inter-node partition
Verify S8 duplicate/delay semantics, then confirm healed nodes converge through sampled latest
reads and transcript audits.
### Slow subscribers
Validate that the existing slow-consumer policy protects hot channels and that
`sockudo_broadcast_latency_ms` recovers after the pressure phase.
### Clock skew
The local script verifies signed client timestamp skew does not affect serial monotonicity. True
node clock mutation must run on privileged production-like infrastructure; serial reservation is
u64 store-backed and must not depend on wall-clock ordering.
## Alert thresholds
Use these as starting alerts and tune with measured baselines:
| Signal | Alert |
| --- | --- |
| `sockudo_connected` | above planned per-node budget for 5 minutes |
| `sockudo_rate_limit_triggered_total` | reconnect storm lane does not trigger shaping, or triggers remain elevated after 10 minutes |
| `sockudo_broadcast_latency_ms` | p99 above 25 ms during headline run or above 10 ms during single-node fanout benchmark |
| `sockudo_ai_active_streams` | does not return to baseline after stream TTL plus janitor window |
| `sockudo_appends_received_total` vs `sockudo_appends_delivered_total` | delivered ratio drops outside configured rollup expectations |
| `sockudo_flush_latency` | 99.9 percent exceeds rollup window plus 5 ms |
| `sockudo_history_recovery_failures_total` | any sustained non-auth failure during reconnect tests |
| `sockudo_history_degraded_channels` | nonzero for more than one scrape interval |
| `sockudo_history_reset_required_channels` | any nonzero value pages |
| `sockudo_horizontal_transport_messages_dropped_total` | any increase outside explicit partition tests |
| `sockudo_horizontal_transport_queue_depth` | sustained growth across three scrape intervals |
| `sockudo_tokio_active_tasks` | positive slope after warm-up in soak |
A reset-required channel means Sockudo can no longer prove history continuity and will fail
recovery closed. That is user-visible and warrants a page. A briefly degraded channel usually
recovers on its own and is a ticket unless it persists.
## Troubleshooting
| Symptom | Likely cause | Operator action |
| --- | --- | --- |
| Mutations rejected as not permitted | App/channel capability lacks the create/update/delete/append action | Inspect token capability and server app policy; client-supplied identity is not authoritative |
| History retention too short | Rewind or recovery request points before retained durable history | Increase retention count/time or reduce expected rewind window; watch retained message/byte gauges |
| Run never ends | Missing terminal append/update/summary or rollup orphan TTL too long | Check `sockudo_ai_active_streams`, terminal event rates, and client run lifecycle |
| Suspended-state publishes | Producer continued after suspend/continuation boundary | Reject or route through the continuation flow; audit `op_id`/`message_id` idempotency |
| Reconnect loop on bad tokens | Clients retrying auth failures without refreshing credentials | Rate-limit, inspect auth error codes, and fix token refresh logic |
| Rollup mis-tuning | Window too high for latency budget or too low for fanout reduction | Compare `sockudo_flush_latency` and `sockudo_rollup_ratio`; tune `[ai_transport.rollup].default_window_ms` |
The full symptom, cause, and fix list is in
[AI Transport troubleshooting](/docs/server/ai-transport-troubleshooting).
## Next steps
- [Production checklist](/docs/server/ai-transport-production-checklist): Preflight sign-off grouped by area.
- [Observability](/docs/server/observability): Every AI Transport metric and the starter Grafana dashboard.
- [Scaling](/docs/server/scaling): Adapters, load balancing, and rolling deploys.
- [Capacity planning](/docs/deployment/capacity-planning): Size nodes, file descriptors, and memory.
---
# AI Transport troubleshooting (https://sockudo.io/docs/server/ai-transport-troubleshooting)
> Common AI Transport production issues and operator actions.
Each entry below starts from what you observe (an error code, a log line, a UI symptom), names the
most likely cause, and gives the fix. Start with the rejection code if you have one: most AI
Transport failures are explicit, validated rejections rather than silent drops, and
`sockudo_ai_messages_rejected_total{code}` shows which codes your traffic is hitting.
## Quick code lookup
| Code | Name | Usual cause |
| --- | --- | --- |
| `104000` | `ai_invalid_transport_header` | Unknown `extras.ai.transport` key, invalid value, or empty `message_id`. |
| `104001` | `ai_header_too_large` | Too many keys, or a key or value over its byte limit. |
| `104002` | `ai_client_id_spoof` | A `*-client-id` header does not match the verified identity. |
| `104003` | `ai_event_not_permitted` | Channel is not an AI channel, the connection is V1, or a client published an agent event. |
| `104009` | `ai_presence_update_invalid` | Malformed `sockudo:presence_update`, or the sender is not a V2 presence member. |
| `40009` | `payload_too_large` | Stream content, append count, open-stream cap, or presence data exceeded. |
| `93002` | `mutable_not_permitted` | Mutable messages are disabled for the channel. |
| `40142` | token expired | Capability token reached `exp`. |
| `40160` | token revoked | Token `jti` or `client_id` was revoked. |
## Startup and setup
### Sockudo refuses to start with an `ai_transport` error
**Cause:** the configuration violates the deployment matrix. Startup messages include
`ai_transport.enabled requires history.enabled`, `ai_transport.enabled requires
versioned_messages.enabled`, and, for horizontal adapters, `require a shared history backend`,
`require a shared version_store driver`, or `require a shared cache driver for orphan ownership`.
**Fix:** enable `[history]` and `[versioned_messages]`. On any adapter other than `local`, use a
non-memory history backend and version store and a Redis or Redis Cluster cache. See the
[deployment matrix](/docs/server/ai-transport-conventions).
### The SDK throws `104007` when creating a client session
**Cause:** the server advertised its feature flags and `ai-transport` was not among them. The binary
was built without the `ai-transport` Cargo feature.
**Fix:** rebuild with `--features "v2,ai-transport,..."` and confirm `[ai_transport].enabled = true`.
### `104003`: "AI Transport is not enabled for this channel"
**Cause:** the channel name does not start with any `[[ai_transport.channels]]` prefix. Prefixes are
plain string prefixes, so `private-ai:user-42` does not match `private-ai-`.
**Fix:** rename the channel to start with a configured prefix, or add the prefix you use. Remember
`presence-` channels need their own prefix entry.
### "Unknown channel namespace"
**Cause:** the channel name contains `:` after its type prefix (for example `private-ai:user-42`),
and the app defines no matching channel namespace.
**Fix:** use hyphenated names such as `private-ai-user-42-sess-01J`, or define the namespace in the
app policy. See [Configuration reference](/docs/reference/configuration).
## Publishing and permissions
### `104003`: "agent AI events require trusted app-key publish"
**Cause:** an agent event (`ai-output`, `ai-run-*`, `ai-step-*`) was published over a WebSocket
connection.
**Fix:** publish agent events through signed app-key HTTP. With the JS SDK, pass
`createAgentSession` a client whose `publish` and mutation methods use HTTP, as the demo's
`realtimeClient()` does.
### "AI client events require an authenticated client_id"
**Cause:** the WebSocket connection has no verified identity. Channel authorization alone does not
establish a `client_id`.
**Fix:** connect with a capability token (or a signed-in user) so Sockudo knows who is publishing.
See [Authentication](/docs/server/ai-transport-authentication).
### `104002` `ai_client_id_spoof`
**Cause:** a `run-client-id`, `input-client-id`, or `step-client-id` header differs from the
connection's verified identity, or an untrusted client sent an empty sentinel.
**Fix:** let the SDK fill identity headers from the connection, and make the token's
`x-sockudo-client-id` the same ID your app uses as `clientId`.
### `104000` or `104001` on publish
**Cause:** an unknown transport key, a value outside its domain (for example `status=done`), an
empty `message_id`, more than 32 keys in a tier, or a key or value over its byte limit.
**Fix:** check the [header registry](/docs/server/ai-transport-conventions). Move provider-specific
data into `extras.ai.codec`; it is bounded but not interpreted.
### Mutations rejected as not permitted
**Cause:** the capability lacks `message_*_own` or `message_*_any`, or the actor identity does not
match the original creator for an own-scoped mutation.
**Fix:** inspect the verified connection identity and token capability map; do not rely on body
`client_id`. HTTP mutations without `socket_id` are server-privileged and bypass this check.
### `93002` "mutations not permitted on this channel"
**Cause:** message mutations are disabled for the channel by server or app policy, so `ai-output`
cannot be created or appended. In the UI this usually shows as an empty assistant message.
**Fix:** enable versioned messages for the AI channel's policy.
## Streaming and runs
### Appends fail with `40009` mid-answer
**Cause:** the output message reached `max_accumulated_message_bytes` (1 MiB),
`max_appends_per_message` (4096), or the channel already has
`max_open_streaming_messages_per_channel` (1024) streams open.
**Fix:** finish the message and continue in a new one, append larger fragments, or raise the cap
after checking storage impact. Check that previous streams on the channel reached a terminal status.
See [Limits and quotas](/docs/server/ai-transport-limits-quotas).
### Run never ends
**Cause:** missing terminal append/update/summary or orphan TTL too high. The agent crashed, or did
not call `end()` on every path.
**Fix:** call `run.end(...)` in a `finally` block and always send a final `status=complete` or
`status=cancelled`. Check `sockudo_ai_active_streams`, terminal event rates, and
`ai_transport.rollup.orphan_ttl_ms`.
### Streams end with `error-code=orphan_timeout`
**Cause:** a message stayed in `status=streaming` without progress for longer than
`ai_transport.rollup.orphan_ttl_ms` (default 60 seconds), so the orphan janitor cancelled it. This is
expected after a worker crash, and unexpected when a long tool call pauses output.
**Fix:** keep long pauses out of an open stream (end the step or suspend the run), or raise
`orphan_ttl_ms`. Subscribe to the `ai_stream_orphaned` webhook to observe these cancellations.
### Suspended-state publishes
**Cause:** the producer continued after the suspend/continuation boundary.
**Fix:** reject or route through the continuation flow; audit `message_id` and `op_id`. A
continuation uses the same `run-id` with a new `invocation-id`, and client inputs that re-enter a
run carry `run-continue=true`.
### Rollup latency too high
**Cause:** the window is too high, or flushes are slow under load.
**Fix:** compare `sockudo_flush_latency` to the configured window and lower `default_window_ms`.
### Rollup savings too low
**Cause:** the window is too low, or terminal flushes dominate.
**Fix:** compare `sockudo_appends_received_total`, `sockudo_appends_delivered_total`, and
`sockudo_rollup_ratio`, then raise `default_window_ms` if latency allows.
### WebSocket upgrade fails with HTTP `400`
**Cause:** a V2 connection sent an `append_rollup_window` that is not `0`, `20`, `40`, `100`, or
`500` (or is outside `min_window_ms`/`max_window_ms`), or an `append_mode` other than `delta` or
`full`.
**Fix:** send an allowed value or omit the parameter.
## Tokens and auth
### Reconnect loop on bad tokens
**Cause:** the client retries an expired, revoked, or malformed JWT without fetching a new one.
**Fix:** fix token refresh, rate-limit retry storms, and revoke compromised `jti` or `client_id`.
Use `authUrl` or `authCallback` so the client fetches a token on every reconnect.
### Connection closes with `4009` after `sockudo:token_expired`
**Cause:** the token expired (`40142`) and no `sockudo:auth` refresh arrived within 30 seconds, or
the token was revoked (`40160`), which closes immediately.
**Fix:** configure `authCallback` or `authUrl` so refresh is automatic; for revocations, issue a new
token only if the revocation was intended to end a single session.
### Token refresh is rejected
**Cause:** the new token carries a different `x-sockudo-client-id`. Refresh cannot change identity.
**Fix:** reconnect to switch users; refresh only with a token for the same identity.
### New tokens for a user are rejected after a revocation
**Cause:** the user's `client_id` was revoked. A `client_id` revocation blocks every token for that
identity until its TTL expires (24 hours by default).
**Fix:** revoke by `jti` to end one session. When revoking by `client_id`, set `ttl_seconds` to the
lockout you actually want.
## Recovery and history
### History retention too short
**Cause:** rewind or recovery points before retained durable history.
**Fix:** increase `[history]` retention or lower the expected rewind/recovery window; watch
retained gauges.
### Resume fails with `position_expired` or `stream_reset`
**Cause:** `position_expired` means the requested position is older than retained history.
`stream_reset` means the channel's stream identity changed (for example after a history reset), so
continuity cannot be proven.
**Fix:** resubscribe and load history with `until_attach: true` to rebuild a gap-free view from the
new attach point. The SDKs surface this as a continuity-lost event.
### Recovery fails after node loss
**Cause:** durable state is degraded or reset-required, or the shared cache or store is unavailable.
**Fix:** inspect `sockudo_history_recovery_failures_total`, the degraded/reset-required gauges, and
store health.
## Presence and push
### Presence flaps during transient loss
**Cause:** the ungraceful timeout is zero or too low.
**Fix:** raise `[presence].v2_ungraceful_timeout_seconds` for Protocol V2 agent presence channels.
### `sockudo:presence_update` rejected
**Cause:** `104009` when the connection is V1, the channel is not a `presence-` channel, or the
sender is not yet a member; `40009` when member data exceeds 1024 bytes; a rate-limit error above
`update_rate_limit_per_member_per_second`.
**Fix:** update only after `subscription_succeeded`, keep the payload small, and debounce status
changes.
### Push channel misses a user
**Cause:** the device is not registered, or it is not subscribed to the notification channel.
**Fix:** inspect push channel subscriptions and provider token state.
For scale and chaos-specific playbooks, see
[AI Transport production operations](/docs/server/ai-transport-production-ops).
## Next steps
- [Production operations](/docs/server/ai-transport-production-ops): Alert thresholds and chaos playbooks.
- [Limits and quotas](/docs/server/ai-transport-limits-quotas): Every cap and what happens when it is exceeded.
- [Error reference](/docs/reference/errors): Full HTTP and WebSocket error code tables.
- [Observability](/docs/server/observability): AI Transport metrics, dashboards, and logs.
---
# Realtime clients (https://sockudo.io/docs/clients)
> Choose and operate an official Sockudo realtime client for web, mobile, desktop, Python, and backend WebSocket consumers.
A realtime client SDK holds a long-lived WebSocket connection to Sockudo. Over that one connection it
subscribes to **channels** (named streams of events such as `orders` or `presence-lobby`), delivers
each incoming event to the handlers you bind, reconnects after network failures, and asks your
backend to authorize channels that carry protected data. On Protocol V2 it also exposes Sockudo's
native features: recovery, rewind, filters, deltas, and versioned messages.
Install client SDKs from the package names below. The old per-SDK repositories should be treated as
archived/legacy mirrors.
## Official clients
| SDK | Package | Runtime | Default protocol |
| --- | --- | --- | --- |
| [JavaScript / TypeScript](/docs/clients/javascript) | `@sockudo/client` | Web, Node, worker, React, Vue, React Native, NativeScript | Protocol V1 compatibility |
| [Swift](/docs/clients/swift) | `SockudoSwift` via SwiftPM | iOS, macOS, tvOS, watchOS, visionOS | Protocol V1 compatibility |
| [Kotlin](/docs/clients/kotlin) | `io.sockudo:sockudo-kotlin` | Android and JVM | Protocol V2 |
| [Flutter / Dart](/docs/clients/flutter) | `sockudo_flutter` | Flutter | Protocol V1 compatibility |
| [.NET realtime](/docs/clients/dotnet) | `Sockudo.Client` | .NET apps | Protocol V2 |
| [Python realtime](/docs/clients/python) | `sockudo-python` | Python 3.10+ and asyncio | Protocol V2 |
Every SDK page follows the same order: install, connect, subscribe, protected channels, Protocol V2
features, connection lifecycle, error handling, and platform notes.
## Choose a protocol
Sockudo serves two protocols on the same port, chosen per connection:
- **Protocol V1** is byte-compatible with Pusher. Use it when migrating existing Pusher or Laravel
Echo clients without code changes.
- **Protocol V2** is Sockudo-native. Use it for recovery, rewind, delta compression, tag filters,
mutable messages, annotations, and native Sockudo metadata.
A backend publishes once; Sockudo strips V2-only fields from deliveries to V1 subscribers. Read
[Protocol V2](/docs/clients/protocol-v2) for the details.
## Clients and servers have different trust levels
The realtime and HTTP SDKs have different trust boundaries. Realtime clients hold only the public
app key and maintain WebSocket subscriptions. Server SDKs hold the app secret, publish over HTTP,
authorize protected subscriptions, and validate webhooks. For Python and .NET in particular, make
sure you choose the package whose role matches your process.
| Language | Realtime package | Trusted HTTP package |
| --- | --- | --- |
| JavaScript / TypeScript | `@sockudo/client` | `sockudo` |
| Python | `sockudo-python` | `sockudo-http-python` |
| .NET | `Sockudo.Client` | `SockudoServer` |
| Swift | `SockudoSwift` | `Sockudo` server package |
### Client responsibilities
Client SDKs should:
- connect with public app key only
- call your backend for private, presence, encrypted, push, and history proxy operations
- keep app secrets out of the client bundle
- bind events and update UI state
- store recovery positions when V2 recovery is enabled
- register devices for push through trusted backend endpoints
A client must never contain the app secret, provider credentials, an encryption master key, or an
unrestricted capability token. Anything in a browser bundle or mobile binary can be extracted.
### Server responsibilities
Server SDKs should:
- publish events over the HTTP API
- sign channel and user authentication responses
- proxy presence history and versioned message reads for clients
- validate webhooks
- manage push registration, credentials, channel push subscriptions, and publish workflows
## Common V2 setup
```ts title="realtime.ts"
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
protocolVersion: 2,
connectionRecovery: true,
});
```
Use this sequence in every client:
1. Construct one long-lived client with your public key and public WebSocket endpoint.
2. Bind connection diagnostics and event handlers.
3. Subscribe to channels, using a backend auth endpoint for protected names.
4. Connect and let the SDK resubscribe after transient failures.
5. If recovery fails, replace derived state from an authoritative snapshot.
6. Unbind or unsubscribe when a screen or worker no longer needs a channel.
7. Disconnect during application shutdown.
## Choosing features
| Requirement | Client setting or API | Backend requirement |
| --- | --- | --- |
| Pusher migration | Protocol V1 | Pusher-compatible app config |
| Resume after a short outage | Protocol V2 recovery | Durable or hot recovery configured |
| Initial backlog | Subscription rewind | History enabled with suitable retention |
| Gap-free late join | History with `until_attach` | Durable history enabled for the channel |
| Lower high-volume bandwidth | Tag filters and delta compression | V2 filtering/delta enabled |
| Private data | Private channel auth | Session-aware auth endpoint |
| End-to-end content encryption | `private-encrypted-*` | Shared-secret auth and key management |
| User-targeted events | User sign-in | User auth endpoint |
| Mobile notifications | Provider token collection | Push registration and publish backend |
## Connection and state model
A successful WebSocket connection does not mean every channel is subscribed. Wait for each
channel's subscription-success event before treating its state as live. During reconnect, keep the
last rendered state but mark it stale. After a successful resume, continue from the recovered
serial; after a failed resume, discard derived state and fetch a fresh snapshot.
Event handlers should be idempotent. Network reconnects, application retries, and upstream
publishing workflows can all repeat logical work even when the client deduplicates messages by
`message_id`.
## Protected-channel auth flow
Channels named `private-*`, `presence-*`, or `private-encrypted-*` carry data that not every
holder of the public app key may see. Sockudo cannot know who your users are, so it delegates that
decision to your backend: the client asks your auth endpoint to sign a specific socket and channel,
and Sockudo verifies the signature with the app secret before completing the subscription.
```mermaid
sequenceDiagram
participant C as Client SDK
participant B as Your backend
participant S as Sockudo
C->>S: Open WebSocket with public app key
S-->>C: socket_id
C->>B: socket_id and requested channel
Note over B: Authenticate session and authorize that exact channel
B-->>C: Short-lived signed auth response
C->>S: Subscribe with auth response
Note over S: Verify signature with the app secret
S-->>C: subscription_succeeded
```
Do not implement an auth endpoint that signs every requested channel. Validate tenant ownership,
resource access, and presence identity before signing. For encrypted channels, return only the
derived channel shared secret, not the encryption master key.
Protocol V2 can also authorize the whole connection with a scoped capability token instead of
per-channel signatures. See [Authentication](/docs/getting-started/authentication) for both flows
end to end.
## Production checklist
- Terminate TLS at Sockudo or a load balancer and use `wss://`.
- Keep a single client per app or process unless isolation is intentional.
- Observe connected, unavailable, failed, and reconnecting states.
- Set auth and proxy request timeouts; handle `401`, `403`, and `429` separately from transport
failures.
- Bound rewind and history page sizes.
- Test a rolling restart, network interruption, auth expiry, and resume failure.
- Unsubscribe and unbind handlers when views or jobs end.
## Push from clients
Mobile and browser clients can collect provider tokens, but they should not call Sockudo push admin
APIs directly. Send provider tokens to your backend, bind them to the authenticated user, and let
the backend call Sockudo.
```ts title="register-device.ts"
await fetch("/api/push/devices", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
device_id: "browser-device-1",
platform: "webpush",
provider_token: subscription,
}),
});
```
See [Push notifications](/docs/server/push-notifications) for the backend side.
## Next steps
- [JavaScript](/docs/clients/javascript): Browser, Node, React, Vue, worker, React Native, and NativeScript examples.
- [Swift](/docs/clients/swift): Apple platforms with auth, recovery, filters, deltas, and push proxy notes.
- [Kotlin](/docs/clients/kotlin): Android and JVM WebSocket client examples.
- [Flutter](/docs/clients/flutter): Flutter client examples with auth, recovery, and push helpers.
- [.NET realtime](/docs/clients/dotnet): Protocol V2 by default for .NET applications.
- [Python realtime](/docs/clients/python): Asyncio client for services, agents, workers, and CLIs.
- [Protocol V2](/docs/clients/protocol-v2): Recovery, history, and native metadata.
- [Filters and delta compression](/docs/clients/filters-delta): Reduce delivered traffic per subscription.
---
# JavaScript and TypeScript (https://sockudo.io/docs/clients/javascript)
> Use @sockudo/client in browsers, Node.js, workers, React, Vue, React Native, and NativeScript.
`@sockudo/client` is the official JavaScript and TypeScript realtime SDK. It keeps the Pusher
`subscribe` / `bind` programming model, so existing `pusher-js` code ports with few changes, and
adds Sockudo-native features on top: connection recovery, rewind, server-side filters, delta
compression, binary wire formats, and versioned messages.
One package covers every JavaScript runtime. Runtime-specific entrypoints keep bundlers from
pulling in transports the target cannot use:
| Runtime | Entrypoint |
| --- | --- |
| Browser and bundlers (default) | `@sockudo/client` |
| Node.js 22+ | `@sockudo/client` (resolved through the `node` export condition) |
| Web Worker | `@sockudo/client/worker` |
| React hooks | `@sockudo/client/react` |
| Vue composables | `@sockudo/client/vue` |
| React Native | `@sockudo/client/react-native` |
| NativeScript | `@sockudo/client/nativescript` |
| End-to-end encrypted channels | `@sockudo/client/with-encryption`, `@sockudo/client/worker/with-encryption` |
| Filter builder | `@sockudo/client/filter` |
The client speaks Protocol V1 (Pusher-compatible, `protocol=7`) by default. Set
`protocolVersion: 2` to opt into Sockudo-native recovery, rewind, filters, deltas, binary wire
formats, and versioned-message metadata. See [Protocol V2](/docs/clients/protocol-v2) for what
changes on the wire.
## Install
Install the published package from npm:
```bash
npm install @sockudo/client
# or: bun add @sockudo/client
# or: pnpm add @sockudo/client
# or: yarn add @sockudo/client
```
Framework integrations use peer dependencies that your application already has:
```bash
npm install @sockudo/client react react-dom # React hooks
npm install @sockudo/client vue # Vue composables
npm install @sockudo/client @valor/nativescript-websockets # NativeScript
```
`@react-native-community/netinfo` is an optional peer dependency for React Native network
reachability.
### Runtime imports
```ts title="imports.ts"
```
The package also sets the `main`, `browser`, `react-native`, and `nativescript` fields in
`package.json`, so most bundlers resolve the right build from the root import alone.
## Connect
A **client** (`new Sockudo(...)`) owns one WebSocket connection to Sockudo, including its
heartbeats and reconnect policy. A **channel** is a named stream of events multiplexed over that
connection. You subscribe to the channels you care about and bind handlers to event names.
The client needs only your **public app key** and the address of your Sockudo server. The app
secret never ships to a browser or device.
```ts title="realtime.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
enabledTransports: ["ws"],
protocolVersion: 2,
connectionRecovery: true,
});
client.connection.bind("state_change", ({ previous, current }) => {
console.log(`connection: ${previous} -> ${current}`);
});
const channel = client.subscribe("public-updates");
channel.bind("price-updated", (payload) => {
console.log("price-updated", payload);
});
```
Publish a `price-updated` event to `public-updates` from your backend (see the
[Node.js server SDK](/docs/server-sdks/node)) and the console shows:
```text
connection: connecting -> connected
price-updated { symbol: 'BTC', price: 67210 }
```
The client connects automatically once the runtime is ready. Calling `client.connect()` yourself
is safe: it does nothing while a connection is already open or being established.
Passing only `cluster: "eu"` resolves the host to `ws-eu.sockudo.io`. For a self-hosted server,
always set `wsHost` to your own ingress. The constructor throws if neither `cluster` nor `wsHost`
is provided.
For self-hosted production, `wsHost` should be the public ingress or load balancer name, `wssPort`
is normally `443`, and `forceTLS` should be `true`. Specify `enabledTransports: ["ws"]` only when
debugging a direct local connection; allowing both secure and fallback transports is safer across
browser and mobile runtimes.
### Common options
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `wsHost` | `string` | `ws-.sockudo.io` when `cluster` is set | WebSocket host. Set this for any self-hosted server. |
| `cluster` | `string` | none | Hosted cluster name. Required when `wsHost` is not set. |
| `wsPort` | `number` | `80` | Port used for unencrypted `ws://` connections. |
| `wssPort` | `number` | `443` | Port used for `wss://` connections. |
| `wsPath` | `string` | `""` | Path prefix in front of `/app/`, for servers mounted behind a path. |
| `forceTLS` | `boolean` | TLS on | TLS is used unless this is `false`. Pages served over `https:` always use TLS. |
| `enabledTransports` | `Transport[]` | all supported | Restrict transports, for example `["ws", "wss"]`. |
| `protocolVersion` | `number` | `7` (Protocol V1) | Set to `2` for Protocol V2. |
| `connectionRecovery` | `boolean` | `false` | V2: track per-channel positions and resume after reconnect. |
| `wireFormat` | `"json"`, `"messagepack"`, `"protobuf"` | `"json"` | V2 wire encoding. `"msgpack"` and `"proto"` are accepted aliases. |
| `messageDeduplication` | `boolean` | `true` | Drop repeated deliveries with the same `message_id`. |
| `messageDeduplicationCapacity` | `number` | `1000` | How many recent `message_id` values to remember. |
| `echoMessages` | `boolean` | server default | V2: set `false` to stop receiving your own publishes. |
| `appendMode` | `"delta"` or `"full"` | server default (`delta`) | V2: how mutable-message appends are delivered. |
| `deltaCompression` | `DeltaOptions` | not set | Enables client-side delta reconstruction. See [Delta compression](#delta-compression). |
| `channelAuthorization` | `object` | endpoint `/sockudo/auth`, transport `fetch` | Private and presence channel auth. |
| `userAuthentication` | `object` | endpoint `/sockudo/user-auth`, transport `fetch` | User sign-in auth. |
| `token`, `authUrl`, `authCallback` | see below | none | V2 capability-token authentication. |
| `presenceHistory` | `{ endpoint, headers?, headersProvider? }` | not set | Backend proxy for presence history helpers. |
| `versionedMessages` | `{ endpoint, headers?, headersProvider? }` | not set | Backend proxy for history, versions, and annotations. |
| `activityTimeout` | `number` (ms) | `120000` | Idle time before the client sends a ping. |
| `pongTimeout` | `number` (ms) | `30000` | Time to wait for a pong before reconnecting. |
| `unavailableTimeout` | `number` (ms) | `10000` | Time in `connecting` before the state becomes `unavailable`. |
| `maxReconnectAttempts` | `number` or `null` | `6` | Retry limit; `null` retries forever. |
| `maxReconnectGapInSeconds` | `number` | `120` | Cap on the quadratic backoff delay. |
| `reconnectJitter` | `number` (0 to 1) | `0` | Fraction of each backoff delay to randomize. |
| `retryJitterMs` | `number` (ms) | `0` | Random delay before immediate protocol retries such as close code `4200`. |
## Subscribe and bind events
`subscribe()` returns the channel immediately, even before the server confirms it. Bind handlers
right away; they start firing once the subscription succeeds.
```ts title="orders.ts"
const channel = client.subscribe("orders");
const onOrder = (order: unknown) => console.log("order.created", order);
channel.bind("order.created", onOrder);
// Observe every event on the channel, including system events.
channel.bind_global((eventName, data) => console.log(eventName, data));
```
### Subscription state
A connected socket can still have a pending or unauthorized channel. Gate channel-dependent UI on
the channel's own success event, not only the global `connected` state.
```ts title="subscription-state.ts"
channel.bind("sockudo:subscription_succeeded", () => {
console.log("orders is live");
});
channel.bind("sockudo:subscription_error", (error) => {
// { type: "AuthError", error: "...", status: 403 }
console.error("orders subscription failed", error);
});
```
Under Protocol V2, system events use the `sockudo:` prefix (`sockudo:subscription_succeeded`,
`sockudo:member_added`). Under the default Protocol V1 they keep the Pusher names
(`pusher:subscription_succeeded`, `pusher:member_added`). Bind the names that match the
`protocolVersion` you configured.
### Unbind and unsubscribe
Keep a reference to the handler when you need to remove exactly one binding. Remove the whole
subscription when its owning component or job ends:
```ts title="cleanup.ts"
channel.unbind("order.created", onOrder);
client.unsubscribe("orders");
client.disconnect();
```
`unsubscribe()` also discards the channel's recovery position and delta state, so a later
`subscribe()` starts fresh. Framework hooks clean up their own bindings when components unmount;
when using the core client directly, lifecycle cleanup is your responsibility.
### Client events
Private and presence channels can carry peer-to-peer client events. Their names must start with
`client-`; any other name throws `BadEventName`.
```ts title="typing.ts"
const room = client.subscribe("private-room-42");
room.bind("sockudo:subscription_succeeded", () => {
room.trigger("client-typing", { userId: "u_123" });
});
```
## Private and presence channels
Anyone with your public app key can open a connection, so Sockudo cannot decide on its own who may
read a protected channel. Channels prefixed `private-`, `presence-`, or `private-encrypted-`
therefore require a signature from **your backend**, which knows who the user is. The SDK asks
your auth endpoint for that signature during every subscribe.
1. The client connects and receives a `socket_id`.
2. On `subscribe("private-orders")`, the SDK sends a form-encoded `POST` with `socket_id` and
`channel_name` to `channelAuthorization.endpoint`.
3. Your backend authenticates the session, checks that this user may access that exact channel,
and returns a signed JSON response (`auth`, plus `channel_data` for presence).
4. The SDK forwards the signature to Sockudo, which verifies it with the app secret and confirms
the subscription.
```ts title="realtime.ts"
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
channelAuthorization: {
endpoint: "/sockudo/auth",
headersProvider: () => ({ Authorization: `Bearer ${getSessionToken()}` }),
},
userAuthentication: {
endpoint: "/sockudo/user-auth",
},
});
```
Both auth option objects accept `endpoint`, `transport` (`"fetch"` or `"ajax"`), static `params`
and `headers`, the dynamic `paramsProvider` and `headersProvider`, and a `customHandler` that
replaces the HTTP request entirely. Any non-`200` response surfaces as a
`subscription_error` with the HTTP `status`.
A minimal backend endpoint with the [Node.js server SDK](/docs/server-sdks/node):
```ts title="server/auth.ts"
const sockudo = new Sockudo({
appId: process.env.SOCKUDO_APP_ID!,
key: process.env.SOCKUDO_KEY!,
secret: process.env.SOCKUDO_SECRET!,
host: "realtime.example.com",
useTLS: true,
});
const app = express();
app.use(express.urlencoded({ extended: false }));
app.post("/sockudo/auth", (req, res) => {
const user = requireSessionUser(req); // your session lookup
const { socket_id, channel_name } = req.body;
if (!canAccessChannel(user, channel_name)) {
return res.status(403).end();
}
const presence = channel_name.startsWith("presence-")
? { user_id: user.id, user_info: { name: user.name } }
: undefined;
res.json(sockudo.authorizeChannel(socket_id, channel_name, presence));
});
```
Never sign every channel name the client asks for. Validate the current session and the exact
`channel_name` before signing. For presence channels, derive `user_id` on the server; do not trust
an identity supplied by the client.
### Presence
Presence channels add a live member list. `subscription_succeeded` delivers the members collection
and member events deliver single members:
```ts title="lobby.ts"
const presence = client.subscribe("presence-lobby");
presence.bind("sockudo:subscription_succeeded", (members) => {
console.log("online:", members.count);
});
presence.bind("sockudo:member_added", (member) => {
console.log("joined", member.id, member.info);
});
presence.bind("sockudo:member_removed", (member) => {
console.log("left", member.id);
});
```
### User sign-in
Configure `userAuthentication.endpoint` and call `client.signin()` when using user-targeted events
or watchlists. The backend signs the current `socket_id` and the authenticated user data (for
example with `sockudo.authenticateUser(socketId, { id: user.id })`). A presence auth response
without `channel_data` falls back to the signed-in user's `id`.
### Capability-token authentication
Protocol V2 can authorize the whole WebSocket with a scoped capability token instead of signing
each channel:
```ts title="token-auth.ts"
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
protocolVersion: 2,
token: initialToken,
authCallback: async ({ socketId, reason }) =>
fetchCapabilityToken({ socketId, reason }),
});
```
The optional static `token` is used on the first connection. The callback receives a `reason` of
`initial`, `reconnect`, `refresh`, or `expired`. It is called before reconnects and at 80% of a
JWT's lifetime, then the client sends `sockudo:auth` without dropping the socket. `authUrl` is
also supported for a token endpoint: it receives a JSON `POST` with `socket_id` and `reason`, and
returns either a token string or an object with `token` and optional expiry fields.
Expiry code `40142` refreshes only through a provider; a static token is never resent in a loop.
Revocation code `40160` surfaces `TokenRevokedError` without an in-place retry. Token auth
configured outside Protocol V2 is rejected when the client is constructed.
### Encrypted channels
Use the encryption entrypoint for `private-encrypted-*` channels:
```ts title="encrypted.ts"
const encryptedClient = new SockudoEncrypted("app-key", {
wsHost: "realtime.example.com",
forceTLS: true,
channelAuthorization: { endpoint: "/sockudo/auth" },
});
encryptedClient
.subscribe("private-encrypted-documents")
.bind("doc.updated", (payload) => console.log(payload));
```
Your auth response supplies the derived channel `shared_secret`. Keep the encryption master key on
the trusted backend. End-to-end encryption complements TLS and authorization; it does not replace
either.
## Protocol V2 features
Everything in this section requires `protocolVersion: 2` on the client and the matching feature
enabled on the server. V1 clients connected to the same server keep receiving plain
Pusher-compatible frames.
### Recovery and rewind
With `connectionRecovery: true`, the client records the `stream_id` and `serial` of the last
message on each channel. After an unexpected reconnect it sends those positions in a
`sockudo:resume` frame, and Sockudo replays what the client missed if it can prove continuity.
```ts title="recovery.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
protocolVersion: 2,
connectionRecovery: true,
});
const channel = client.subscribe("market:BTC", {
rewind: { seconds: 30 }, // or { count: 50 }, or a bare number of messages
});
channel.bind("sockudo:rewind_complete", (payload) => {
console.log(payload.historical_count, payload.complete);
});
client.bind("sockudo:resume_success", (payload) => {
// payload.recovered: [{ channel, source, replayed, position }]
console.log(payload.recovered, payload.failed);
});
client.bind("sockudo:resume_failed", (payload) => {
// payload.code is for example "position_expired" or "continuity_unverifiable"
console.warn("rebuild", payload.channel, payload.code);
});
```
On `resume_failed` the SDK drops that channel's position and resubscribes it automatically. Your
application must discard derived state for the channel and reload an authoritative snapshot.
Read or restore positions with `getRecoveryPosition(channel)`, `getRecoveryPositions()`,
`setRecoveryPosition(channel, position)`, and `setRecoveryPositions(positions)`, for example to
carry positions across a page reload.
### Channel history and late joins
The `versionedMessages` proxy exposes paged channel history through
`channel.channelHistory({ limit, direction, cursor, start_serial, end_serial, start_time_ms, end_time_ms })`.
Each page has `hasNext()` and `next()`.
For a gap-free late join, use history bounded at the subscription's attach point: first attach,
then load history ending at the server-provided `attach_serial`, then apply buffered live events.
The SDK does not wrap the WebSocket `channel_history` request; send it directly and bind the
response on the channel:
```ts title="late-join.ts"
const chat = client.subscribe("chat:room-1");
chat.bind("sockudo:channel_history", (page) => {
// page.items, page.has_more, page.next_cursor, page.bounds.attach_serial
renderBackfill(page.items);
});
chat.bind("sockudo:subscription_succeeded", () => {
client.send_event("sockudo:channel_history", {
channel: "chat:room-1",
limit: 100,
direction: "newest_first",
until_attach: true,
});
});
```
The request and response shapes are documented in [Protocol V2](/docs/clients/protocol-v2#client-history).
### Filters
Server-side filters stop irrelevant messages before they reach the client. A subscription can
combine event names, a tag filter, and a bounded JMESPath expression; every supplied component must
match:
```ts title="filters.ts"
client.subscribe("orders.*", {
events: ["order.updated"],
filter: Filter.and(Filter.eq("region", "eu"), Filter.gte("total", "100")),
expression: 'data.total >= `100` && headers.priority == `"high"`',
});
```
The full operator list is in [Filters and delta compression](/docs/clients/filters-delta).
### Delta compression
Delta compression sends only the difference between consecutive similar messages. Enable
reconstruction on the client with `deltaCompression`, then optionally choose an algorithm or opt
out per subscription:
```ts title="deltas.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
protocolVersion: 2,
deltaCompression: { enabled: true, algorithms: ["fossil", "xdelta3"] },
});
client.subscribe("market:btc", {
delta: { enabled: true, algorithm: "xdelta3" },
});
client.subscribe("audit-log", { delta: { enabled: false } });
console.log(client.getDeltaStats()?.bandwidthSavedPercent);
```
A subscription `delta` setting asks the server to send deltas, but only a client constructed with
`deltaCompression` can decode them. Without it, delta frames reach the channel as raw
`sockudo:delta` events and your normal handlers do not fire.
`DeltaOptions` also accepts `debug`, `onStats`, and `onError`. `resetDeltaStats()` clears the
counters.
### Binary wire formats
Protocol V2 can negotiate JSON, MessagePack, or Protobuf with `wireFormat`. MessagePack and
Protobuf preserve `Uint8Array` message data as native binary values without a JSON or base64
intermediate.
```ts title="binary.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
protocolVersion: 2,
wireFormat: "messagepack",
});
const channel = client.subscribe("private-binary");
channel.trigger("client-binary", new Uint8Array([0xde, 0xad, 0xbe, 0xef]));
```
Native MessagePack uses the additive tagged data variant `["binary", ]`. Existing string,
structured, and JSON variants retain their previous representation.
### Versioned messages
Mutable messages arrive as explicit action events: `sockudo:message.update`,
`sockudo:message.delete`, and `sockudo:message.append`.
- `message.update`: replace local state with the full payload in the event.
- `message.delete`: treat the event as the latest visible version; the payload may be `null`.
- `message.append`: concatenate the incoming string fragment onto your current string state. If
you do not have a string base yet, fetch the latest visible message first.
The `isMutableMessageEvent` and `reduceMutableMessageEvent` helpers read the action and serials
from the frame's `extras.headers`. Channel handlers receive only `data`, so feed the helpers full
frames from the connection-level `message` stream:
```ts title="mutable.ts"
getMutableMessageInfo,
isMutableMessageEvent,
reduceMutableMessageEvent,
type MutableMessageState,
} from "@sockudo/client";
const messages = new Map();
client.connection.bind("message", (event) => {
if (event.channel !== "chat:room-1" || !isMutableMessageEvent(event)) return;
const info = getMutableMessageInfo(event)!;
const current = messages.get(info.messageSerial) ?? null;
if (current?.versionSerial && current.versionSerial === info.versionSerial) return;
const next = reduceMutableMessageEvent(current, event);
messages.set(info.messageSerial, next);
console.log(next.messageSerial, next.action, next.data);
});
```
The reducer tracks one logical message and throws if it receives a different `messageSerial`, so
keep one state per serial as shown.
`client.connection` emits every decoded frame before `message_id` deduplication runs. Compare
`versionSerial` before applying a frame, as above, so that a repeated append is not applied twice.
Configure a backend proxy for REST reads. The endpoint receives a JSON `POST` of the form
`{ channel, messageSerial?, params?, action }` and must sign the upstream Sockudo HTTP request with
the app secret:
```ts title="versions.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
protocolVersion: 2,
versionedMessages: {
endpoint: "/sockudo/versioned",
},
});
const channel = client.subscribe("chat:room-1");
const latest = await channel.getMessage("42");
const versions = await channel.getMessageVersions("42", {
limit: 20,
direction: "oldest_first",
});
```
The proxy must authenticate the caller and authorize access to the channel. A helper called
without `versionedMessages.endpoint` throws.
### Annotations
Annotations attach reactions, labels, or counters to a message serial. The same proxy powers
`channel.publishAnnotation(messageSerial, { type, name, data, ... })`,
`channel.deleteAnnotation(messageSerial, annotationSerial)`, and
`channel.listAnnotations(messageSerial, { type, limit })`.
By default a subscriber receives only `message.summary` projections. Subscribe with
`annotationSubscribe: true` to also receive raw `annotation.create` and `annotation.delete`
events:
```ts title="annotations.ts"
const chat = client.subscribe("chat:room-1", { annotationSubscribe: true });
chat.bind("annotation.create", (annotation) => console.log(annotation.type, annotation.name));
chat.bind("message.summary", (summary) => console.log(summary));
```
## Framework integrations
### React
Create the client once, outside render functions, and provide it to the tree. `useChannel`
subscribes on mount, unsubscribes on unmount, and returns a state snapshot (`subscribed`,
`subscriptionPending`, `members`, `lastEventName`, `lastEventData`, and more).
```tsx title="App.tsx"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
});
function Orders() {
const { subscribed, lastEventName, lastEventData } = useChannel("orders", {
onEvent: (eventName, data) => console.log(eventName, data),
});
return
;
}
export function App() {
return (
);
}
```
React exports: `SockudoProvider`, `useSockudo`, `useSockudoEvent`, `useChannel`, and
`usePresenceChannel`. Hook options include `enabled`, `filter`, `delta`, `events`, `onEvent`,
`onSubscriptionSucceeded`, `onSubscriptionError`, `onMemberAdded`, and `onMemberRemoved`.
### Vue
The Vue composables return the same state as refs, plus a `stop()` function:
```ts title="main.ts"
const client = new Sockudo("app-key", {
wsHost: "127.0.0.1",
wsPort: 6001,
forceTLS: false,
});
const Orders = defineComponent({
setup() {
const { subscribed, lastEventData } = useChannel("orders", {
onEvent: (eventName, data) => console.log(eventName, data),
});
return () => h("pre", JSON.stringify({ subscribed: subscribed.value, last: lastEventData.value }));
},
});
createApp(Orders).use(createSockudoPlugin(client)).mount("#app");
```
Vue exports: `createSockudoPlugin`, `provideSockudo`, `useSockudo`, `useSockudoEvent`,
`useChannel`, and `usePresenceChannel`.
## Connection lifecycle and reconnection
`client.connection` reports one of these states:
| State | Meaning |
| --- | --- |
| `initialized` | The client was created and has not started connecting. |
| `connecting` | The first connection attempt is in progress. |
| `reconnecting` | A retry after an unexpected disconnect is in progress. |
| `connected` | The socket is open; `client.connection.socket_id` is set. |
| `unavailable` | No connection within `unavailableTimeout`, or the network is down. Retries continue. |
| `disconnected` | Closed by `disconnect()`, by a refusing close code, or after `maxReconnectAttempts`. |
| `failed` | The runtime cannot open any supported transport, or a token provider failed before connecting. |
Each state is emitted as its own event and as `state_change` with `{ previous, current }`. Before a
delayed retry the manager also emits `connecting_in` with the delay in seconds.
```ts title="lifecycle.ts"
client.connection.bind("state_change", ({ previous, current }) => {
console.log(`connection: ${previous} -> ${current}`);
});
client.connection.bind("connecting_in", (seconds) => {
console.log(`retrying in ${seconds}s`);
});
```
Unexpected disconnects emit `reconnecting` and use quadratic backoff (0s, 1s, 4s, 9s) capped at
120 seconds. Configure `maxReconnectAttempts` (default `6`, `null` for unlimited) and
`maxReconnectGapInSeconds`. `reconnectJitter` (0 to 1) randomizes that fraction of each backoff
delay. Protocol retry close codes, such as `4200` during a server shutdown, reconnect immediately
unless `retryJitterMs` is set, in which case each client waits a random 0 to `retryJitterMs`
milliseconds first. TLS-upgrade retries remain immediate.
```ts title="reconnect.ts"
const client = new Sockudo("app-key", {
wsHost: "realtime.example.com",
maxReconnectAttempts: 10,
maxReconnectGapInSeconds: 60,
reconnectJitter: 0.5,
retryJitterMs: 2000,
});
client.connection.bind("reconnecting", () => console.log("reconnecting"));
```
The attempt counter resets after a successful connection and on explicit `connect()` or
`disconnect()` calls. Close codes map to these actions:
| Close code | Action |
| --- | --- |
| `4000` | Reconnect immediately over TLS. |
| `4001` to `4099` | Refused: the client disconnects and does not retry. |
| `4100` to `4199` | Reconnect with backoff. |
| `4200` to `4299` | Reconnect immediately (subject to `retryJitterMs`). |
| `4300` and above | Refused: the client disconnects and does not retry. |
Without jitter, every client dropped by the same event retries on the same schedule. In large
deployments set `reconnectJitter` (for example `0.5`) and `retryJitterMs` (for example `2000`) so
a rolling restart does not land all clients on the remaining nodes at once.
Heartbeats are automatic. Protocol V2 prefers native WebSocket ping/pong frames; browser-like
runtimes that cannot send native pings may use lightweight `sockudo:ping` / `sockudo:pong`
fallback messages, which carry no `message_id`, `serial`, or `stream_id`.
## Error handling
Errors surface on three channels. Bind all of them in production:
```ts title="errors.ts"
// Transport and server errors: { type: "SockudoError" | "WebSocketError" | ..., data | error }
client.connection.bind("error", (error) => {
console.error("Sockudo connection error", error);
});
// Capability-token failures (TokenAuthError, TokenExpiredError, TokenRevokedError)
client.bind("error", (error) => {
if (error instanceof TokenRevokedError) signOut();
});
// Per-channel auth failures: { type: "AuthError", error, status }
channel.bind("sockudo:subscription_error", (error) => {
if (error.status === 401 || error.status === 403) showAccessDenied();
});
```
| Error | Where | Typical cause |
| --- | --- | --- |
| `SockudoError` | `client.connection` `error` | Server `sockudo:error` frame or a close code; `data.code` carries the code. |
| `WebSocketError` | `client.connection` `error` | Transport failure while connecting. |
| `MessageParseError` | `client.connection` `error` | A frame could not be decoded. |
| `AuthError` | channel `subscription_error` | The auth endpoint returned non-`200` or invalid JSON. |
| `TokenExpiredError`, `TokenRevokedError` | `client` `error` | Capability token expired without a provider (`40142`) or revoked (`40160`). |
| `BadEventName` | thrown by `trigger()` | Client event name does not start with `client-`. |
- Treat auth `401` and `403` as an application-permission problem, not a reconnect loop.
- On `sockudo:resume_failed`, clear derived state and fetch an authoritative snapshot before
applying new events.
- Make handlers idempotent and validate unknown payloads at the application boundary.
## Platform notes
### Browser and Node.js
- Reuse one client per browser tab, worker, or Node process.
- Create the client outside React render functions and Vue component setup.
- Use the runtime-specific entrypoint so bundlers do not pull in incompatible browser or Node
transports. The Node build requires Node.js 22 or newer.
- Unbind handlers, unsubscribe unused channels, and disconnect during application shutdown.
- Never expose the app secret, push credentials, or encryption master key in a browser or mobile
bundle.
### React Native and NativeScript
- React Native resolves through the root `react-native` field or the explicit
`@sockudo/client/react-native` subpath.
- Import `@sockudo/client/nativescript` to load the `@valor/nativescript-websockets` polyfill
before the SDK.
### Push proxy helpers
Browser push registration should flow through your backend. Keep app secrets and provider
credentials server-side.
```ts title="push-device.ts"
await fetch("/sockudo/push/devices", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
device_id: "browser-device-1",
platform: "webpush",
provider_token: pushSubscription,
}),
});
```
`SockudoPushRegistration` is available from the root package for authenticated application
proxies. It covers device registration, channel push subscriptions, publish, scheduling, and
status, and includes typed direct and broadcast Apple Live Activity recipients and payloads:
```ts title="live-activity.ts"
const push = new SockudoPushRegistration({
endpoint: "/sockudo/push",
headers: async () => ({ Authorization: `Bearer ${await sessionToken()}` }),
});
await push.publishLiveActivity({
publishId: "ride-184-update-42",
recipients: [{
type: "recipient",
recipient: {
transportType: "apnsLiveActivity",
activityToken: currentActivityToken,
},
}],
liveActivity: {
event: "update",
timestamp: 1725000042,
contentState: { status: "arriving", etaMinutes: 1 },
priority: "conservePower",
},
});
```
React Native and NativeScript must obtain rotating ActivityKit tokens through the application's
native iOS bridge because the attributes type belongs to the app's Widget Extension. Upload every
rotation immediately. Use `apnsLiveActivityBroadcast` only with a server-created channel and its
immutable storage policy. See [Apple Live Activities](/docs/server/apple-live-activities).
The proxy must authenticate the caller and authorize the exact activity or channel. Use a server
SDK for unrestricted administrative publishing; never expose the app secret, APNs provider key, or
a blanket `push-admin` capability to client code.
## Next steps
- [Protocol V2](/docs/clients/protocol-v2): Wire metadata, recovery contract, and client history.
- [Filters and delta compression](/docs/clients/filters-delta): Tag filters, compound predicates, and delta tuning.
- [Presence history](/docs/clients/presence-history): Current members, transition history, and snapshots.
- [Authentication](/docs/getting-started/authentication): Channel auth, user auth, and capability tokens end to end.
- [Node.js server SDK](/docs/server-sdks/node): Publish events and sign auth responses from your backend.
- [History and recovery](/docs/server/history-recovery): Server-side recovery tiers, rewind, and durable history.
---
# Swift (https://sockudo.io/docs/clients/swift)
> Use SockudoSwift on iOS, macOS, tvOS, watchOS, and visionOS.
`SockudoSwift` is the official realtime client for Apple platforms. It opens a
WebSocket to Sockudo, subscribes to channels, and delivers events to your
callbacks. It supports public, private, presence, and encrypted channels, user
sign-in, client events, and the Protocol V2 feature set: connection recovery,
rewind, tag filters, delta compression, mutable messages, and proxy-backed
history helpers.
Supported deployment targets are iOS 13+, macOS 10.15+, tvOS 13+, watchOS
6+, and visionOS 1+. The package uses Swift 6.2 concurrency and isolates the
client, channels, and callbacks on `@SockudoActor`.
`SockudoSwift` starts in Pusher-compatible Protocol V1 mode (`protocol=7`). Set
`protocolVersion: 2` explicitly for new Sockudo applications that need
continuity metadata, recovery, filters, deltas, or any other V2 feature.
## Install
Install through Swift Package Manager from the SockudoSwift mirror. SwiftPM resolves this from
the `v3.0.0` Git tag and the mirror repository's root `Package.swift` manifest.
1. Add the package dependency to `Package.swift`:
```swift title="Package.swift"
.package(url: "https://github.com/sockudo/sockudo-swift", from: "3.0.0")
```
2. Depend on the `SockudoSwift` product from your target:
```swift title="Package.swift"
.target(
name: "YourApp",
dependencies: [
.product(name: "SockudoSwift", package: "sockudo-swift"),
]
)
```
3. In an Xcode project, add the same URL through **File > Add Package Dependencies** and select
the `SockudoSwift` library.
For local development against the monorepo, point SwiftPM at the package
directory instead:
```swift title="Package.swift"
.package(path: "../sockudo/client-sdks/sockudo-swift")
```
The package depends on `swift-sodium` for encrypted channels and vendors
xdelta3 for delta decoding; both resolve automatically.
## Connect
A `SockudoClient` owns one WebSocket connection for your app. Over that
connection you subscribe to **channels**: named streams such as
`public-updates` that your backend publishes events to. The client holds only
the public app key; the app secret stays on your server.
```swift title="Realtime.swift"
@SockudoActor
func startRealtime() throws -> SockudoClient {
let client = try SockudoClient(
"app-key",
options: .init(
cluster: "local",
protocolVersion: 2,
forceTLS: false,
enabledTransports: [.ws],
wsHost: "127.0.0.1",
wsPort: 6001,
wssPort: 6001,
connectionRecovery: true
)
)
let channel = client.subscribe("public-updates")
channel.bind("price-updated") { data, _ in
print(data ?? "")
}
client.connect()
return client
}
```
The function is annotated `@SockudoActor` because every client and channel
method is isolated to that actor. From other contexts, call it with `await`.
For production, point `wsHost` at the public ingress or load balancer
hostname, set `forceTLS: true`, and remove `enabledTransports: [.ws]` so the
client uses the secure transport.
`SockudoClient.Options` uses a memberwise-style initializer, so Swift requires
labelled arguments in the order they are declared. `protocolVersion` comes
directly after `cluster`, and `connectionRecovery` comes after the auth, history,
and delta options. The table below lists options in declaration order.
### Connection options
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `cluster` | `String` | required | Cluster name. Must be non-empty. |
| `protocolVersion` | `Int` | `7` | `7` is Protocol V1 (Pusher-compatible); `2` enables Protocol V2. |
| `activityTimeout` | `TimeInterval` | `120` | Idle seconds before the client checks liveness. |
| `forceTLS` | `Bool?` | `nil` | Force `wss://`. Use `true` in production. |
| `enabledTransports` | `[Transport]?` | `nil` | Restrict to `.ws` and/or `.wss`. |
| `wsHost` | `String?` | `nil` | Sockudo hostname. |
| `wsPort` | `Int` | `80` | Port for `ws://`. |
| `wssPort` | `Int` | `443` | Port for `wss://`. |
| `wsPath` | `String` | `""` | Path prefix for the WebSocket URL. |
| `pongTimeout` | `TimeInterval` | `30` | Seconds to wait for a pong. |
| `unavailableTimeout` | `TimeInterval` | `10` | Seconds before a stalled connection is reported `unavailable`. |
| `channelAuthorization` | `ChannelAuthorizationOptions` | endpoint `/sockudo/auth` | Private, presence, and encrypted channel auth. |
| `userAuthentication` | `UserAuthenticationOptions` | endpoint `/sockudo/user-auth` | User sign-in auth. |
| `presenceHistory` | `PresenceHistoryOptions?` | `nil` | Backend proxy for presence history and snapshots. |
| `versionedMessages` | `VersionedMessagesOptions?` | `nil` | Backend proxy for channel history and mutable-message helpers. |
| `capabilityToken` | `CapabilityTokenOptions?` | `nil` | Protocol V2 capability-token auth. |
| `deltaCompression` | `DeltaOptions?` | `nil` | Connection-level delta negotiation. |
| `messageDeduplication` | `Bool` | `true` | Drop repeated `message_id` values. |
| `messageDeduplicationCapacity` | `Int` | `1000` | Size of the deduplication window. |
| `connectionRecovery` | `Bool` | `false` | Track positions and resume after reconnect (V2). |
| `echoMessages` | `Bool` | `true` | Receive your own published messages. |
| `wireFormat` | `SockudoWireFormat` | `.json` | `.json`, `.messagepack`, or `.protobuf` (V2). |
| `appendMode` | `SockudoAppendMode` | `.delta` | `.delta` or `.full` delivery of message appends. |
| `appendRollupWindow` | `Int?` | `nil` | Append rollup window in milliseconds: `0`, `20`, `40`, `100`, or `500`. |
| `maxReconnectAttempts` | `Int?` | `6` | Reconnect attempts before giving up; `nil` is unlimited. |
| `maxReconnectGapInSeconds` | `Double` | `120` | Upper bound on reconnect delay. |
| `reconnectJitter` | `Double` | `0` | Fraction (0 to 1) of each reconnect delay to randomize. |
## Subscribe and bind events
`subscribe(_:options:)` returns a `Channel` immediately; the subscription
completes asynchronously once the socket is connected (and, for protected
channels, once your auth endpoint responds). `bind` attaches a callback for one
event name and returns an `EventBindingToken`.
```swift title="PriceFeed.swift"
@SockudoActor
final class PriceFeed {
private let client: SockudoClient
private var channel: Channel?
private var token: EventBindingToken?
init(client: SockudoClient) {
self.client = client
}
func start() {
let channel = client.subscribe("public-updates")
channel.bind("sockudo:subscription_succeeded") { _, _ in
print("public-updates is live")
}
token = channel.bind("price-updated") { data, _ in
print("price:", data ?? "")
}
self.channel = channel
}
func stop() {
if let token {
channel?.unbind(eventName: "price-updated", token: token)
}
client.unsubscribe("public-updates")
}
}
```
System events use the protocol prefix: `sockudo:` on Protocol V2 and `pusher:`
on Protocol V1 (for example `pusher:subscription_succeeded`). Use
`bindGlobal` to observe every event name on a channel or client, and
`unbindAll()` only when the owner is being torn down.
### Callbacks run on SockudoActor
Event callbacks execute on the dedicated Sockudo actor, not the main thread,
and in binding order. Hop to `MainActor` for UIKit or SwiftUI state:
```swift
channel.bind("price-updated") { data, _ in
Task { @MainActor in
viewModel.latestPrice = String(describing: data)
}
}
```
From code outside `@SockudoActor`, access client methods with `await`:
```swift
func startRealtime() async {
await client.connect()
}
func stopRealtime() async {
await client.unsubscribe("public-updates")
await client.disconnect()
}
```
### Client events
Private and presence channels can send `client-` prefixed events directly to
other subscribers. `trigger(event:data:)` throws `SockudoError.badEventName`
for any other name:
```swift
_ = try channel.trigger(event: "client-typing", data: ["user": "alice"])
```
Client events triggered while the channel is not yet subscribed are buffered,
up to 50 per channel (oldest dropped on overflow), and replayed after the
subscription succeeds. Unsubscribing clears that buffer.
## Private and presence channels
Anyone with your public app key can open a socket, so Sockudo cannot decide by
itself who may join `private-*`, `presence-*`, or `private-encrypted-*`
channels. Instead, the client sends its `socket_id` and the requested channel
name to **your** backend, which checks the user's session and returns a
signature made with the app secret. Sockudo verifies that signature before
completing the subscription.
```mermaid
sequenceDiagram
participant App as iOS app
participant API as Your backend
participant S as Sockudo
App->>S: connect with public app key
S-->>App: socket_id
App->>API: POST /sockudo/auth (socket_id, channel_name)
API-->>App: signed auth response
App->>S: subscribe with auth
S-->>App: subscription_succeeded
```
```swift title="Realtime.swift"
let client = try SockudoClient(
"app-key",
options: .init(
cluster: "local",
forceTLS: false,
wsHost: "127.0.0.1",
wsPort: 6001,
channelAuthorization: .init(
endpoint: "https://api.example.com/sockudo/auth",
headersProvider: { ["Authorization": "Bearer \(SessionStore.token)"] }
)
)
)
```
`ChannelAuthorizationOptions` accepts `endpoint` (default `/sockudo/auth`),
static `headers` and `params`, `headersProvider` and `paramsProvider` closures
evaluated per request, and a `customHandler` when you need to perform the
request yourself. `userAuthentication` has the same shape (default endpoint
`/sockudo/user-auth`) and is used by `client.signIn()` for user-targeted events
and watchlists.
The backend must authenticate the current user and authorize the exact
`channel_name` before signing. Presence identity must come from the server-side
session, not a value supplied by the app. Never ship the app secret in the
app bundle.
### Presence
Subscribe as `PresenceChannel` to access members and V2 in-place presence
updates:
```swift title="Lobby.swift"
let presence = client.subscribe("presence-agent:session-123") as! PresenceChannel
presence.bind("sockudo:subscription_succeeded") { _, _ in
print("members:", presence.members.count, "me:", presence.members.myID ?? "")
}
presence.bind("sockudo:member_added") { member, _ in
print("joined:", member as Any)
}
presence.bind("sockudo:member_removed") { member, _ in
print("left:", member as Any)
}
presence.bind("sockudo:presence_update") { member, _ in
print("updated:", member as Any)
}
_ = try presence.update(data: ["status": "thinking"])
```
Wait for the subscription-success event before treating the initial membership
set as authoritative. `presence.members` exposes `count`, `myID`, `me`,
`member(id:)`, and `forEach`. `update(data:)` changes your member data without a
leave/rejoin cycle and throws `SockudoError.unsupportedFeature` on Protocol V1.
### Capability tokens
Protocol V2 also supports scoped capability tokens instead of per-channel
signatures:
```swift title="Realtime.swift"
let client = try SockudoClient(
"app-key",
options: .init(
cluster: "local",
protocolVersion: 2,
forceTLS: true,
wsHost: "realtime.example.com",
capabilityToken: .init(asyncProvider: {
try await tokenService.fetchSockudoToken()
})
)
)
```
The initial token is sent as the WebSocket `token` query parameter. JWTs with
expiry metadata refresh proactively at 80% of their lifetime, and providers are
called before reconnects. On `sockudo:token_expired`, only expiry code `40142`
refreshes in place through `sockudo:auth`; revocation code `40160` and static
tokens are not resent. Configuring token auth outside Protocol V2 throws
`SockudoError.invalidOptions`.
## Protocol V2 features
The features below require `protocolVersion: 2`. See
[Protocol V2](/docs/clients/protocol-v2) for the wire-level contract.
### Recovery and rewind
With `connectionRecovery: true`, the client records a `RecoveryPosition` per
channel and asks Sockudo to resume from it after a reconnect. `rewind` replays
recent history when a subscription starts.
```swift title="Market.swift"
let channel = client.subscribe(
"market:BTC",
options: .init(rewind: .seconds(30))
)
channel.bind("trade") { _, _ in
print(client.recoveryPosition(for: "market:BTC") as Any)
}
channel.bind("sockudo:rewind_complete") { data, _ in
print(data as Any)
}
client.bind("sockudo:resume_success") { data, _ in
print(data as Any)
}
client.bind("sockudo:resume_failed") { data, _ in
print("rebuild state:", data as Any)
}
```
`SubscriptionRewind` is `.count(n)` or `.seconds(n)`. `recoveryPositions()`
and `setRecoveryPositions(_:)` let you persist positions across app launches.
On `sockudo:resume_failed` the client drops the stored position and
resubscribes; your app must discard derived state and reload an authoritative
snapshot.
### Channel history and late join
`channel.attachSerial` holds the history head captured when the subscription
attached. Load history up to that serial for a gap-free late join. Channel
history is served through the `versionedMessages` backend proxy:
```swift
print(channel.attachSerial as Any)
channel.channelHistory(.init(limit: 50, untilAttach: true)) { result in
print(result)
}
```
### Filters and deltas
Tag filters, event-name filters, and JMESPath expressions are evaluated on the
server, so non-matching messages never reach the device. Delta compression
sends only the difference against the previous message.
```swift title="Market.swift"
let channel = client.subscribe(
"price:btc",
options: .init(
filter: Filter.eq("market", "spot"),
delta: .init(enabled: true, algorithm: .xdelta3),
events: ["price.updated"],
expression: .source("data.price >= `100`")
)
)
```
`Filter` provides `eq`, `neq`, `in`, `nin`, `exists`, `notExists`,
`startsWith`, `endsWith`, `contains`, `gt`, `gte`, `lt`, `lte`, `and`, `or`,
and `not`. Delta algorithms are `.fossil` and `.xdelta3`. See
[Filters and delta compression](/docs/clients/filters-delta).
### Mutable messages
Protocol V2 mutable messages arrive as update, delete, and append actions for
an existing `message_serial`. Apply them in serial order:
- an update replaces local content with the full event payload
- a delete is the latest visible version and may carry `nil` data
- an append concatenates onto the current local string state
If an append arrives before you have a string base, fetch the latest visible
message first and seed local state before applying more appends.
Configure `versionedMessages.endpoint` to use proxy-backed write and read
helpers without exposing Sockudo app secrets in the app. The proxy accepts
JSON `{ channel, messageSerial?, action, payload }` and calls Sockudo's signed
REST API server-side. The Swift SDK does not send WebSocket mutation frames.
```swift title="Chat.swift"
let client = try SockudoClient(
"app-key",
options: .init(
cluster: "local",
protocolVersion: 2,
forceTLS: false,
wsHost: "127.0.0.1",
wsPort: 6001,
versionedMessages: .init(
endpoint: "https://api.example.com/sockudo/versioned"
)
)
)
let channel = client.subscribe("chat:room-1")
client.connect()
channel.createMessage(
.init(name: "chat.message", data: ["text": "hello"], messageID: "msg-1")
) { result in
guard case .success(let ack) = result else { return }
Task { @SockudoActor in
channel.appendMessage(
ack.messageSerial,
request: .init(data: " world", opID: "append-1")
) { result in
print(result)
}
}
}
```
Each helper returns a `VersionedMessageAck` with `messageSerial`,
`versionSerial`, `historySerial`, `deliverySerial`, and `status`.
`updateMessage`, `deleteMessage`, `getMessage`, and `getMessageVersions` follow
the same pattern. Proxy requests time out after 10 seconds.
### Presence history
Presence history and snapshots are also proxy-backed, because a mobile client
must not sign Sockudo REST requests:
```swift title="Lobby.swift"
let client = try SockudoClient(
"app-key",
options: .init(
cluster: "local",
forceTLS: false,
wsHost: "127.0.0.1",
wsPort: 6001,
presenceHistory: .init(
endpoint: "https://api.example.com/sockudo/presence-history"
)
)
)
let channel = client.subscribe("presence-lobby") as! PresenceChannel
channel.history(.init(direction: "newest_first", limit: 50)) { result in
print(result)
}
channel.snapshot(.init(atSerial: 4)) { result in
print(result)
}
```
The proxy accepts `{channel, params, action}`. Authorize the caller and channel
on every request, bound page sizes, and forward opaque cursors unchanged. See
[Presence history](/docs/clients/presence-history).
### Binary wire formats
Set `wireFormat: .messagepack` or `.protobuf` to negotiate a binary encoding.
Both preserve `Data` message payloads as native binary. The MessagePack
representation uses the additive `["binary", ]` tagged value; existing
string and JSON variants are unchanged.
### Encrypted channels
`private-encrypted-*` channels decrypt automatically when the protected-channel
auth response contains a derived `shared_secret`:
```swift
let encrypted = client.subscribe("private-encrypted-documents")
encrypted.bind("doc-updated") { payload, _ in
print(payload as Any)
}
```
Keep the encryption master key on the backend. Continue to use TLS and normal
channel authorization.
## Connection lifecycle and reconnection
The client moves through the `ConnectionState` values `initialized`,
`connecting`, `connected`, `reconnecting`, `disconnected`, `unavailable`, and
`failed`. Every transition emits `state_change` with `previous` and `current`
keys, followed by an event named after the new state:
```swift
client.bind("state_change") { change, _ in
print("connection:", change as Any)
}
client.bind("reconnecting") { _, _ in
print("reconnecting")
}
```
A connected socket can still have a pending or rejected channel. Gate
channel-dependent UI on the channel's `subscription_succeeded` event, not on
the global `connected` state.
Unexpected disconnects reconnect with quadratic backoff: `attempt²` seconds
(0s, 1s, 4s, 9s, 16s, ...) capped by `maxReconnectGapInSeconds` (default 120),
for up to `maxReconnectAttempts` (default 6, `nil` for unlimited). The attempt
counter resets after a successful connection and on explicit `connect()` or
`disconnect()` calls.
```swift title="Realtime.swift"
let client = try SockudoClient(
"app-key",
options: .init(
cluster: "local",
maxReconnectAttempts: 10,
maxReconnectGapInSeconds: 60,
reconnectJitter: 0.5
)
)
```
Jitter is off by default. Set `reconnectJitter` (0 to 1) to randomize each
delay by that fraction so clients dropped by one event do not retry in
lockstep. Close codes `4200`–`4299` and the TLS-upgrade code `4000` reconnect
immediately; codes `4001`–`4099` are refused and are not retried.
Apply an app-specific reconnect limit for backgrounded mobile processes, and
call `disconnect()` on explicit sign-out rather than letting the retry budget
run out.
## Error handling
- **Construction:** `SockudoClient.init` throws `SockudoError.invalidAppKey`
for an empty key and `SockudoError.invalidOptions` for an empty cluster,
token auth without Protocol V2, or an unsupported `appendRollupWindow`.
- **Connection:** bind `"error"` on the client. Payloads include
`SockudoError` values such as `connectionUnavailable` as well as server
error events.
- **Subscription:** a failed auth request emits `sockudo:subscription_error`
(`pusher:subscription_error` on V1) on the channel with `type` and `error`
keys. Treat it as a permission problem, not a reason to reconnect.
- **Recovery:** on `sockudo:resume_failed`, reload authoritative application
state.
- **Helpers:** history, versioned-message, and push helpers deliver
`Result` values; a missing proxy endpoint fails with
`SockudoError.unsupportedFeature`.
```swift
client.bind("error") { error, _ in
print("Sockudo error:", error as Any)
}
channel.bind("sockudo:subscription_error") { error, _ in
print("subscription failed:", error as Any)
}
```
See [Errors](/docs/reference/errors) for server error and close codes.
## Platform notes
### Threading model
Message decoding, deduplication, and delta reconstruction run on a background
`MessageProcessor` actor before events reach `@SockudoActor` for routing and
callback delivery. Neither stage blocks the main thread. Synchronous reads of
client state such as `connectionState` and `socketID` require `await` from
outside `@SockudoActor`.
### Heartbeats
On Protocol V2 the client uses `URLSessionWebSocketTask.sendPing` for native
liveness checks. Lightweight `sockudo:ping` / `sockudo:pong` fallback messages
remain reserved for compatibility and diagnostics and do not carry
`message_id`, `serial`, or `stream_id`.
### Push notifications
Use APNs to obtain a device token, then send it to your backend. Your backend
registers the device with Sockudo and keeps APNs credentials server-side.
```swift title="AppDelegate.swift"
func application(
_ application: UIApplication,
didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data
) {
let token = deviceToken.map { String(format: "%02x", $0) }.joined()
Task {
await registerDeviceWithBackend(
deviceId: UIDevice.current.identifierForVendor?.uuidString ?? token,
platform: "apns",
providerToken: token
)
}
}
```
`SockudoPushRegistration` wraps device registration, channel subscription, and
publish calls against **your** authenticated proxy. `publish` and
`publishBatch` always send `sync = false` and expect `202 Accepted` with a
`publish_id`; list helpers use `limit` and `cursor`.
```swift title="Push.swift"
let push = SockudoPushRegistration(
options: .init(
endpoint: "https://api.example.com/sockudo/push",
headers: ["Authorization": "Bearer session-token"]
)
)
push.publish(
[
"recipients": [["type": "channel", "channel": "orders"]],
"payload": ["title": "Order updated", "body": "Ready for pickup"],
]
) { result in
if case .success(let payload) = result {
print(payload["publish_id"] as Any)
}
}
```
### Live Activities
ActivityKit can rotate Live Activity tokens. Observe the token sequence for
the full lifetime of the activity and upload every replacement:
```swift title="LiveActivity.swift"
let tokenTask = SockudoLiveActivityTokens.observePushTokens(for: activity) {
activityID, token in
await backend.replaceLiveActivityToken(activityID: activityID, token: token)
}
push.publishLiveActivity(
activityToken: currentToken,
payload: .init(
event: .update,
contentState: ["etaMinutes": 2],
priority: .conservePower
)
) { result in
print(result)
}
```
`observePushTokens` requires iOS 16.1; `observePushToStartTokens` requires
iOS 17.2. For iOS 18 shared-content audiences, use
`createLiveActivityBroadcastChannel` and `publishLiveActivityBroadcast`.
Broadcast channel management belongs on the trusted backend. See
[Apple Live Activities](/docs/server/apple-live-activities).
Do not embed APNs private keys, Sockudo app secrets, or push-admin credentials
in the app. Every push helper must point at your authenticated backend.
### Production checklist
- Use one long-lived client for the app session and disconnect it on explicit
sign-out.
- Select Protocol V2 deliberately; do not assume the default changed during a
Pusher migration.
- Move UI mutations from `@SockudoActor` to `MainActor`.
- Apply an app-specific reconnect limit for backgrounded mobile processes.
- Treat failed recovery as a signal to reload authoritative application state.
- Keep auth, history, versioned-message, and push proxy credentials on the
backend.
- Unbind view-owned callbacks to prevent retained views and duplicate updates.
- Test offline/online transitions, token expiry, app suspension, and a rolling
Sockudo restart.
## Next steps
- [Protocol V2](/docs/clients/protocol-v2): Recovery, rewind, history, and native event semantics.
- [Filters and delta compression](/docs/clients/filters-delta): Cut delivered traffic with server-side filters and deltas.
- [Authentication](/docs/getting-started/authentication): Build the backend auth endpoint for private and presence channels.
- [Push notifications](/docs/server/push-notifications): Register devices and publish through your backend.
---
# Kotlin (https://sockudo.io/docs/clients/kotlin)
> Use sockudo-kotlin on Android and JVM with auth, recovery, rewind, filters, deltas, history proxies, and push registration.
`sockudo-kotlin` is the official Android and JVM realtime SDK. It opens a
WebSocket to Sockudo through OkHttp, subscribes to channels, and delivers
events to your callbacks using the same subscribe/bind model as other Sockudo
clients. It supports public, private, presence, and encrypted channels, user
sign-in, client events, and the Protocol V2 feature set: connection recovery,
rewind, tag filters, delta compression, mutable messages, and proxy-backed
history helpers.
The current package targets JVM 17 and supports Android API 26+ (Android
8.0+); the build fails on any JDK API missing from that level.
Unlike the JavaScript and Swift clients, `sockudo-kotlin` uses Protocol V2 by
default. Set `protocolVersion = 1` only when strict Pusher Protocol V1
compatibility is required.
## Install
Install the published package from Maven Central. Requirements: Java 17 or
higher, Kotlin 1.9+, and Android API 26+ for Android apps.
```kotlin title="build.gradle.kts"
dependencies {
implementation("io.sockudo:sockudo-kotlin:2.3.1")
}
```
For Maven projects:
```xml title="pom.xml"
io.sockudosockudo-kotlin2.3.1
```
For local monorepo development, run `./gradlew :lib:publishToMavenLocal` from
`client-sdks/sockudo-kotlin`.
## Connect
A `SockudoClient` owns one WebSocket connection for your process. Over that
connection you subscribe to **channels**: named streams such as
`public-updates` that your backend publishes events to. The client holds only
the public app key; the app secret stays on your server.
```kotlin title="Realtime.kt"
val client =
SockudoClient(
"app-key",
SockudoOptions(
cluster = "local",
forceTls = false,
enabledTransports = listOf(SockudoTransport.ws),
wsHost = "127.0.0.1",
wsPort = 6001,
wssPort = 6001,
protocolVersion = 2,
connectionRecovery = true,
),
)
val channel = client.subscribe("public-updates")
channel.bind("price-updated") { data, _ ->
println(data)
}
client.connect()
```
For production, point `wsHost` at the public load balancer or ingress, set
`forceTls = true`, use the secure WebSocket port, and drop the
`enabledTransports` restriction. Keep one long-lived client per application
process or signed-in mobile session.
### Connection options
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `cluster` | `String` | required | Cluster name. Must be non-blank. |
| `protocolVersion` | `Int` | `2` | `2` is Protocol V2; `1` is Pusher-compatible V1. |
| `forceTls` | `Boolean?` | `null` | Force `wss://`. Use `true` in production. |
| `enabledTransports` | `List?` | `null` | Restrict to `SockudoTransport.ws` and/or `wss`. |
| `wsHost` | `String?` | `null` | Sockudo hostname. |
| `wsPort` | `Int` | `80` | Port for `ws://`. |
| `wssPort` | `Int` | `443` | Port for `wss://`. |
| `wsPath` | `String` | `""` | Path prefix for the WebSocket URL. |
| `activityTimeout` | `Duration` | `120.seconds` | Idle time before the client checks liveness. |
| `pongTimeout` | `Duration` | `30.seconds` | Time to wait for a pong. |
| `unavailableTimeout` | `Duration` | `10.seconds` | Time before a stalled connection is reported `UNAVAILABLE`. |
| `channelAuthorization` | `ChannelAuthorizationOptions` | endpoint `/sockudo/auth` | Private, presence, and encrypted channel auth. |
| `userAuthentication` | `UserAuthenticationOptions` | endpoint `/sockudo/user-auth` | User sign-in auth. |
| `presenceHistory` | `PresenceHistoryOptions?` | `null` | Backend proxy for presence history and snapshots. |
| `versionedMessages` | `VersionedMessagesOptions?` | `null` | Backend proxy for channel history, annotations, and mutable-message helpers. |
| `deltaCompression` | `DeltaOptions?` | `null` | Connection-level delta negotiation. |
| `messageDeduplication` | `Boolean` | `true` | Drop repeated `message_id` values. |
| `messageDeduplicationCapacity` | `Int` | `1000` | Size of the deduplication window. |
| `connectionRecovery` | `Boolean` | `false` | Track positions and resume after reconnect. |
| `echoMessages` | `Boolean` | `true` | Receive your own published messages. |
| `wireFormat` | `SockudoWireFormat` | `json` | `json`, `messagepack`, or `protobuf`. |
| `appendMode` | `SockudoAppendMode` | `delta` | `delta` or `full` delivery of message appends. |
| `appendRollupWindow` | `Int?` | `null` | Append rollup window in milliseconds: `0`, `20`, `40`, `100`, or `500`. |
| `maxReconnectAttempts` | `Int?` | `6` | Reconnect attempts before giving up; `null` is unlimited. |
| `maxReconnectGapInSeconds` | `Double` | `120.0` | Upper bound on reconnect delay. |
| `reconnectJitter` | `Double` | `0.0` | Fraction (0 to 1) of each reconnect delay to randomize. |
| `authToken` | `String?` | `null` | Static Protocol V2 capability token. |
| `authTokenProvider` | `ClientAuthTokenProvider?` | `null` | Refreshing Protocol V2 capability-token provider. |
`SockudoOptions` validates itself on construction and throws
`SockudoException.InvalidOptions` for token auth below Protocol V2 or an
unsupported `appendRollupWindow`.
## Subscribe and bind events
`subscribe(channelName, options)` returns a `SockudoChannel` immediately; the
subscription completes asynchronously once the socket is connected (and, for
protected channels, once your auth endpoint responds). `bind` attaches a
callback for one event name and returns an `EventBindingToken`:
```kotlin title="PriceFeed.kt"
val stateToken =
client.bind("state_change") { change, _ ->
println("connection: $change")
}
val readyToken =
channel.bind("sockudo:subscription_succeeded") { _, _ ->
println("public-updates is live")
}
val eventToken =
channel.bind("price-updated") { data, _ ->
println(data)
}
// Later, when the owner no longer needs updates:
channel.unbind("price-updated", eventToken)
channel.unbind("sockudo:subscription_succeeded", readyToken)
client.unbind("state_change", stateToken)
client.unsubscribe("public-updates")
```
System events use the protocol prefix: `sockudo:` on Protocol V2 and `pusher:`
on Protocol V1. Use `bindGlobal` to observe every event name, and
`unbindAll()` only when the owner is being torn down.
### Typed listeners
As an alternative to string event names, bind a typed channel listener. It
receives the full `SockudoEvent`, including `serial`, `messageId`, and
`extras`:
```kotlin title="PriceFeed.kt"
val listenerToken =
channel.bindChannelListener(
object : SockudoChannelEventListener {
override fun onSubscriptionSucceeded(channel: SockudoChannel) {
println("${channel.name} is live")
}
override fun onAuthenticationFailure(channel: SockudoChannel, error: SockudoAuthError) {
println("auth failed: ${error.message}")
}
override fun onEvent(channel: SockudoChannel, event: SockudoEvent) {
println("${event.event}: ${event.data}")
}
},
)
channel.unbindChannelListener(listenerToken)
```
### Client events
Private and presence channels can send `client-` prefixed events directly to
other subscribers. `trigger` throws `SockudoException.BadEventName` for any
other name:
```kotlin
channel.trigger("client-typing", mapOf("user" to "alice"))
```
## Private and presence channels
Anyone with your public app key can open a socket, so Sockudo cannot decide by
itself who may join `private-*`, `presence-*`, or `private-encrypted-*`
channels. Instead, the client sends its `socket_id` and the requested channel
name to **your** backend, which checks the user's session and returns a
signature made with the app secret. Sockudo verifies that signature before
completing the subscription.
```kotlin title="Realtime.kt"
val client =
SockudoClient(
"app-key",
SockudoOptions(
cluster = "local",
forceTls = false,
wsHost = "127.0.0.1",
wsPort = 6001,
channelAuthorization =
ChannelAuthorizationOptions(
endpoint = "https://api.example.com/sockudo/auth",
headersProvider = { mapOf("Authorization" to "Bearer ${session.token}") },
),
),
)
```
`ChannelAuthorizationOptions` accepts `endpoint` (default `/sockudo/auth`),
static `headers` and `params`, `headersProvider` and `paramsProvider` lambdas
evaluated per request, and a `customHandler` when you need to perform the
request yourself:
```kotlin
ChannelAuthorizationOptions(
customHandler =
ChannelAuthorizationHandler { request ->
// request.socketId and request.channelName; call your backend here
backend.authorizeChannel(request.socketId, request.channelName)
},
)
```
The handler is a `suspend` function that returns `ChannelAuthorizationData`
(`auth`, optional `channelData`, optional `sharedSecret`).
`userAuthentication` has the same shape (default endpoint `/sockudo/user-auth`)
and is used by `client.signIn()` for user-targeted events and watchlists.
Your auth endpoint must validate the application session and authorize the
exact requested channel. For presence, derive `user_id` from the server-side
identity. Never ship the app secret or an encryption master key in the APK.
### Presence
```kotlin title="Lobby.kt"
val presence = client.subscribe("presence-lobby") as PresenceChannel
presence.bind("sockudo:subscription_succeeded") { _, _ ->
println("members: ${presence.members.count}, me: ${presence.members.myId}")
}
presence.bind("sockudo:member_added") { member, _ -> println("joined: $member") }
presence.bind("sockudo:member_removed") { member, _ -> println("left: $member") }
presence.bind("sockudo:presence_update") { member, _ -> println("updated: $member") }
presence.update(mapOf("status" to "typing"))
```
Wait for the subscription-success event before treating the initial membership
set as authoritative. `presence.members` exposes `count`, `myId`, `me`,
`member(id)`, and `forEach`. `update` changes V2 presence member data without a
leave/rejoin cycle and throws `SockudoException.UnsupportedFeature` on
Protocol V1.
### Capability tokens
Protocol V2 capability tokens can be supplied with a refresh provider:
```kotlin title="Realtime.kt"
val client =
SockudoClient(
"app-key",
SockudoOptions(
cluster = "local",
protocolVersion = 2,
authTokenProvider =
ClientAuthTokenProvider { request ->
fetchCapabilityToken(
reason = request.reason,
socketId = request.socketId,
)
},
),
)
```
`ClientAuthTokenProvider.token` is a `suspend` function. `request.reason` is
one of `INITIAL`, `RECONNECT`, `REFRESH`, or `EXPIRED`. For JWTs, the client
reads `iat` / `exp` without validating the signature and schedules a proactive
refresh at 80% of the token lifetime. The provider is called before every
reconnect, including for opaque tokens. Only expiry code `40142` refreshes in
place through `sockudo:auth`; revocation code `40160` and static `authToken`
values are not resent. Token auth is rejected outside Protocol V2.
## Protocol V2 features
See [Protocol V2](/docs/clients/protocol-v2) for the wire-level contract
behind these features.
### Recovery and rewind
With `connectionRecovery = true`, the client records a `RecoveryPosition` per
channel and asks Sockudo to resume from it after a reconnect. `rewind` replays
recent history when a subscription starts.
```kotlin title="Market.kt"
val channel =
client.subscribe(
"market:BTC",
SubscriptionOptions(rewind = SubscriptionRewind.Seconds(30)),
)
channel.bind("trade") { _, _ ->
println(client.getRecoveryPosition("market:BTC"))
}
channel.bind("sockudo:rewind_complete") { data, _ ->
println(data)
}
client.bind("sockudo:resume_success") { data, _ ->
println(data)
}
client.bind("sockudo:resume_failed") { data, _ ->
println("rebuild state: $data")
}
```
`SubscriptionRewind` is `Count(n)` or `Seconds(n)`. `getRecoveryPositions()`
and `setRecoveryPositions(...)` let you persist positions across process
restarts. On `sockudo:resume_failed`, discard derived state and reload an
authoritative snapshot.
### Channel history and late join
`channel.attachSerial` holds the history head captured when the subscription
attached. History, snapshots, and versioned-message helpers are `suspend`
functions that call your `versionedMessages` proxy:
```kotlin title="Chat.kt"
suspend fun loadBacklog(channel: SockudoChannel) {
println(channel.attachSerial)
var page = channel.channelHistory(ChannelHistoryParams(limit = 50, untilAttach = true))
render(page.items)
while (page.hasNext()) {
page = page.next()
render(page.items)
}
}
```
Use `untilAttach = true` when building a gap-free late join: combine those
items with live events received after attach.
### Filters and deltas
Tag filters, event-name filters, and JMESPath expressions are evaluated on the
server, so non-matching messages never reach the device. Delta compression
sends only the difference against the previous message.
```kotlin title="Market.kt"
val channel =
client.subscribe(
"price:btc",
SubscriptionOptions(
filter = Filter.eq("market", "spot"),
events = listOf("price.updated"),
expression = SubscriptionExpression.Source("data.price >= `100`"),
delta = ChannelDeltaSettings(
enabled = true,
algorithm = DeltaAlgorithm.xdelta3,
),
),
)
```
`Filter` provides `eq`, `neq`, `inside` (`in`), `notIn` (`nin`), `exists`,
`notExists`, `startsWith`, `endsWith`, `contains`, `gt`, `gte`, `lt`, `lte`,
`and`, `or`, and `not`. Delta algorithms are `DeltaAlgorithm.fossil` and
`DeltaAlgorithm.xdelta3`. See
[Filters and delta compression](/docs/clients/filters-delta).
### Mutable messages
Protocol V2 mutable messages arrive as update, delete, and append actions for
an existing `message_serial`. Apply them in serial order:
- an update replaces local data with the full event payload
- a delete becomes the latest visible version and may carry `null` data
- an append concatenates onto a known string base
Reduce them with `reduceMutableMessageEvent` from a typed channel listener,
which receives the full `SockudoEvent` including its `extras` headers:
```kotlin title="Chat.kt"
val states = mutableMapOf()
channel.bindChannelListener(
object : SockudoChannelEventListener {
override fun onEvent(channel: SockudoChannel, event: SockudoEvent) {
val info = getMutableMessageInfo(event) ?: return
try {
states[info.messageSerial] =
reduceMutableMessageEvent(states[info.messageSerial], event)
} catch (e: IllegalStateException) {
println("mutable message reduction failed: ${e.message}")
}
}
},
)
```
If an append arrives before its base, the reducer throws; fetch the latest
message with `channel.getMessage(messageSerial)` before applying subsequent
actions.
Configure `VersionedMessagesOptions.endpoint` for proxy-backed writes. The
client never signs Sockudo REST requests itself:
```kotlin title="Chat.kt"
suspend fun sendAndEdit(chat: SockudoChannel) {
val ack = chat.createMessage("chat.message", mapOf("text" to "hello"))
chat.appendMessage(ack.messageSerial, " world")
chat.updateMessage(ack.messageSerial, mapOf("text" to "edited"))
chat.deleteMessage(ack.messageSerial)
}
```
Each helper accepts optional `extras`, `socketId`, `idempotencyKey`, and
(for mutations) `expectedVersionSerial`, and times out after 10 seconds by
default.
### Presence history
```kotlin title="Lobby.kt"
val client =
SockudoClient(
"app-key",
SockudoOptions(
cluster = "local",
forceTls = false,
wsHost = "127.0.0.1",
wsPort = 6001,
presenceHistory =
PresenceHistoryOptions(
endpoint = "https://api.example.com/sockudo/presence-history",
),
),
)
suspend fun loadPresence() {
val channel = client.subscribe("presence-lobby") as PresenceChannel
val page = channel.history(PresenceHistoryParams(limit = 50, direction = "newest_first"))
if (page.hasNext()) {
val nextPage = page.next()
}
val snapshot = channel.snapshot(PresenceSnapshotParams(atSerial = 4))
}
```
The proxy accepts `{channel, params, action}`, must authenticate the caller,
authorize the channel, bound page sizes, and sign the upstream Sockudo
request. See [Presence history](/docs/clients/presence-history).
### Binary wire formats
Set `wireFormat = SockudoWireFormat.messagepack` or `protobuf` to negotiate a
binary encoding. Both preserve `ByteArray` message data as native binary. The
MessagePack representation uses the additive `["binary", ]` tagged value;
existing string and JSON variants are unchanged.
### Encrypted channels
`private-encrypted-*` subscriptions decrypt automatically when the auth
response includes the derived channel shared secret:
```kotlin
val encrypted = client.subscribe("private-encrypted-documents")
encrypted.bind("doc-updated") { payload, _ -> println(payload) }
```
An auth response without `shared_secret` fails the subscription. Encrypted
channels still require TLS and normal channel authorization.
## Connection lifecycle and reconnection
`client.connectionState` exposes the current `ConnectionState`:
`INITIALIZED`, `CONNECTING`, `CONNECTED`, `RECONNECTING`, `DISCONNECTED`,
`UNAVAILABLE`, or `FAILED`. Observe transitions with a typed listener or the
raw Pusher-compatible `state_change` event:
```kotlin title="Realtime.kt"
val connectionToken =
client.bindConnectionListener(
object : SockudoConnectionEventListener {
override fun onConnectionStateChange(change: ConnectionStateChange) {
println("${change.previous} -> ${change.current}")
}
override fun onError(error: SockudoError) {
println("connection error: ${error.message}")
}
},
)
client.unbindConnectionListener(connectionToken)
```
Every transition also emits an event named after the new state in lower case,
such as `reconnecting` or `connected`.
A connected socket can still have a pending or rejected channel. Wait for the
channel subscription-success event before treating channel state as live.
Unexpected disconnects emit `ConnectionState.RECONNECTING` and use quadratic
backoff (0s, 1s, 4s, 9s) capped by `maxReconnectGapInSeconds` (default 120),
for up to `maxReconnectAttempts` (default `6`, `null` for unlimited). The
attempt counter resets after a successful connection and on explicit
`connect()` or `disconnect()` calls.
```kotlin title="Realtime.kt"
val client =
SockudoClient(
"app-key",
SockudoOptions(
cluster = "local",
maxReconnectAttempts = 10,
maxReconnectGapInSeconds = 60.0,
reconnectJitter = 0.5,
),
)
client.bind("reconnecting") { _, _ -> println("reconnecting") }
```
Because the base delay is identical for every client, clients dropped by a
single event would retry in lockstep. `reconnectJitter` randomizes each delay
by that fraction: `0.5` picks uniformly from 50–100% of the delay, `1.0` from
0–100%. Close codes `4200`–`4299` and the TLS-upgrade code `4000` reconnect
immediately; codes `4001`–`4099` are refused and move to `DISCONNECTED`
without retrying.
### Disconnect versus close
`disconnect()` closes the socket but keeps the client reusable; call
`connect()` to start again. `close()` disconnects and also shuts down the
client's event loop. Use it when the client is discarded for good.
```kotlin
client.unsubscribe("public-updates")
client.disconnect() // reversible: connect() again later
client.close() // permanent shutdown
```
## Error handling
- **Construction:** `SockudoClient` throws `SockudoException.InvalidAppKey`
for a blank key and `SockudoException.InvalidOptions` for a blank cluster or
invalid options.
- **Connection:** `SockudoConnectionEventListener.onError` and the raw
`"error"` event report transport failures and server errors as
`SockudoError` (`message`, optional `code`, optional `cause`).
- **Subscription:** a failed auth request emits `sockudo:subscription_error`
on the channel with `type` and `error` keys, and calls
`onAuthenticationFailure` on typed listeners. Treat it as a permission
failure instead of reconnecting.
- **Overload:** inbound frames are decoded on a bounded queue. If your
callbacks fall behind, frames are dropped and an error with code
`client_overloaded` is reported.
- **Helpers:** `suspend` helpers throw `SockudoException.UnsupportedFeature`
when their proxy endpoint is not configured; call them inside `try`/`catch`
so a failed proxy request does not crash the calling coroutine.
```kotlin
client.bind("error") { error, _ -> println("Sockudo error: $error") }
channel.bind("sockudo:subscription_error") { error, _ ->
println("subscription failed: $error")
}
```
See [Errors](/docs/reference/errors) for server error and close codes.
## Platform notes
### Threading
Decoding and subscriber callbacks run on a dedicated `sockudo-event` thread,
never on OkHttp's reader thread. Connection listeners run synchronously on the
thread performing the transition. In both cases, switch to the main thread
before touching Android views, and keep callbacks short: move blocking work to
your own dispatcher so the bounded inbound queue does not overflow.
### Android lifecycle
Tie subscriptions to an Android lifecycle owner, and unsubscribe and unbind
when it no longer needs updates so configuration changes do not create
duplicate bindings. Disconnect on explicit sign-out. A temporary network loss
should be left to the SDK's reconnect path.
### Heartbeats
Sockudo servers use native WebSocket ping/pong frames for automatic heartbeat
traffic. Kotlin/JVM runtimes may still use lightweight `sockudo:ping` /
`sockudo:pong` fallback messages for client-side activity checks; those are
excluded from V2 recovery metadata such as `message_id`, `serial`, and
`stream_id`.
### Push registration
Use Firebase Messaging or the platform provider to obtain a token, then
register it through your backend:
```kotlin title="PushTokenService.kt"
class PushTokenService : FirebaseMessagingService() {
override fun onNewToken(token: String) {
registerDeviceWithBackend(
deviceId = Settings.Secure.getString(contentResolver, Settings.Secure.ANDROID_ID),
platform = "fcm",
providerToken = token,
)
}
}
```
The backend calls Sockudo push registration APIs with app credentials.
`SockudoPushRegistration` wraps device registration, channel subscription,
and publish calls against **your** authenticated proxy. `publish()` and
`publishBatch()` always send `sync = false` and expect `202 Accepted` with a
`publish_id`; list helpers use `limit` and `cursor`.
```kotlin title="Push.kt"
val push =
SockudoPushRegistration(
PushRegistrationOptions(
endpoint = "https://api.example.com/sockudo/push",
headers = mapOf("Authorization" to "Bearer session-token"),
),
)
suspend fun notifyOrders() {
val publish =
push.publish(
mapOf(
"recipients" to listOf(mapOf("type" to "channel", "channel" to "orders")),
"payload" to mapOf("title" to "Order updated", "body" to "Ready for pickup"),
),
)
println(publish["publish_id"])
val page = push.listChannelSubscriptions(PushSubscriptionParams(deviceId = "device-1", limit = 20))
println(page["next_cursor"])
}
```
The Android app should not hold Sockudo app secrets, provider credentials, or
push-admin credentials. Every push helper must point at your authenticated
backend.
### Production checklist
- Tie subscriptions to an Android lifecycle owner and avoid duplicate bindings
after configuration changes.
- Keep callbacks short; move blocking work off the callback thread.
- Treat auth failures as permission failures instead of endlessly reconnecting.
- On `sockudo:resume_failed`, reload authoritative application state.
- Bound rewind and proxy history requests.
- Refresh FCM tokens through the backend when `onNewToken` fires.
- Test airplane-mode transitions, process recreation, capability-token expiry,
and rolling Sockudo restarts.
## Next steps
- [Protocol V2](/docs/clients/protocol-v2): Recovery, rewind, history, and native event semantics.
- [Filters and delta compression](/docs/clients/filters-delta): Cut delivered traffic with server-side filters and deltas.
- [Authentication](/docs/getting-started/authentication): Build the backend auth endpoint for private and presence channels.
- [Push notifications](/docs/server/push-notifications): Register devices and publish through your backend.
---
# Flutter and Dart (https://sockudo.io/docs/clients/flutter)
> Use sockudo_flutter to add Sockudo realtime channels, presence, and Protocol V2 features to Flutter apps.
`sockudo_flutter` is the official Flutter and Dart realtime SDK for Sockudo. It keeps the Pusher
subscribe/bind model and adds Sockudo-native features: public, private, presence, and encrypted
channels, channel and user auth, Protocol V2 recovery and rewind, tag filters, Fossil and
Xdelta3/VCDIFF delta reconstruction, mutable messages, presence history proxies, and push
registration helpers.
The current package (`2.2.0`) requires Dart `^3.11.3` and Flutter `3.35+`. Protocol V1
compatibility is the default; opt into Protocol V2 for Sockudo-native features.
## Install
Install the published package from `pub.dev`:
```bash
flutter pub add sockudo_flutter
# or, with the Dart CLI inside a Flutter project:
dart pub add sockudo_flutter
```
For local monorepo development, use a path dependency instead:
```yaml title="pubspec.yaml"
dependencies:
sockudo_flutter:
path: ../sockudo/client-sdks/sockudo-flutter
```
Everything is exported from a single library:
```dart
```
## Connect
A `SockudoClient` owns one WebSocket connection to Sockudo. Channels are named streams multiplexed
over that connection: you subscribe to a channel, then bind callbacks to the event names published
on it. The client holds only your public app key; publishing and signing happen on your backend.
```dart title="lib/realtime.dart"
final client = SockudoClient(
'app-key',
const SockudoOptions(
cluster: 'local',
forceTls: false,
enabledTransports: [SockudoTransport.ws],
wsHost: '127.0.0.1',
wsPort: 6001,
wssPort: 6001,
protocolVersion: 2,
connectionRecovery: true,
),
);
void startRealtime() {
client.bind('state_change', (change, _) {
print('connection: ${change['previous']} -> ${change['current']}');
});
final channel = client.subscribe('public-updates');
channel.bind('price-updated', (data, _) {
print(data);
});
client.connect();
}
```
When the server accepts the connection, the state changes print in order:
```text
connection: initialized -> connecting
connection: connecting -> connected
```
`forceTls` defaults to `null`, which the client treats as TLS. A plain `ws://` local server needs
`forceTls: false`. `wsHost` defaults to `ws-.sockudo.com`, so self-hosted deployments must
set `wsHost` to their own ingress.
In production, point `wsHost` at the public load balancer or ingress with `forceTls: true`, and
create the client in an application-scoped service rather than in a widget's `build` method.
### Common options
| Option | Type | Default | Description |
| --- | --- | --- | --- |
| `cluster` | `String` | required | Cluster name. Must be non-empty; also used to derive default hosts. |
| `protocolVersion` | `int` | `7` | `7` is Protocol V1 (Pusher-compatible). Set `2` for Protocol V2. |
| `forceTls` | `bool?` | `null` | Anything other than `false` connects over `wss`. |
| `wsHost` | `String?` | `null` | WebSocket host. Falls back to `ws-.sockudo.com`. |
| `wsPort` / `wssPort` | `int` | `80` / `443` | Ports for `ws` and `wss`. |
| `enabledTransports` | `List?` | `null` | Restrict to `SockudoTransport.ws` and/or `SockudoTransport.wss`. |
| `connectionRecovery` | `bool` | `false` | Track stream positions and send a V2 resume after reconnect. |
| `wireFormat` | `SockudoWireFormat` | `json` | V2 wire format: `json`, `messagepack`, or `protobuf`. |
| `messageDeduplication` | `bool` | `true` | Drop repeated `message_id` values. |
| `messageDeduplicationCapacity` | `int` | `1000` | Size of the deduplication window. |
| `activityTimeout` | `Duration` | `120s` | V1 idle time before the client sends a ping. |
| `pongTimeout` | `Duration` | `30s` | V1 wait for a pong before reconnecting. |
| `unavailableTimeout` | `Duration` | `10s` | Time in `connecting` before the state becomes `unavailable`. |
| `maxReconnectAttempts` | `int?` | `6` | Reconnect attempts before giving up. `null` is unlimited. |
| `maxReconnectGapInSeconds` | `double` | `120.0` | Upper bound on the backoff delay. |
| `reconnectJitter` | `double` | `0.0` | Fraction of each backoff delay to randomize, clamped to 0–1. |
| `appendRollupWindow` | `int?` | `null` | V2 AI append rollup window: `0`, `20`, `40`, `100`, or `500`. |
## Subscribe and bind events
`subscribe` returns the channel immediately and sends the subscription once the socket is
connected. Bind handlers before or after subscribing; each `bind` returns an `EventBindingToken`
you can use to remove exactly that handler later.
```dart title="lib/orders_service.dart"
final orders = client.subscribe('orders');
orders.bind('sockudo:subscription_succeeded', (_, __) {
print('orders is live');
});
orders.bind('sockudo:subscription_error', (error, _) {
print('orders subscription failed: $error');
});
final orderToken = orders.bind('order.created', (data, metadata) {
print('order: $data from user ${metadata?.userId}');
});
// Receive every event on the channel, for logging or reducers.
orders.bindGlobal((eventName, data) => print('$eventName $data'));
```
Remove handlers and subscriptions in the owning service or controller's disposal path:
```dart
orders.unbind(eventName: 'order.created', token: orderToken);
client.unsubscribe('orders');
```
System events use the `sockudo:` prefix on Protocol V2 and `pusher:` on Protocol V1. On the default
V1 connection, bind `pusher:subscription_succeeded` instead of `sockudo:subscription_succeeded`.
A connected socket can still have a pending or rejected channel. Wait for the channel's
subscription-success event before treating its data as live. Client events (`client-*`) can be
sent with `channel.trigger('client-typing', data)` on authorized channels.
## Private and presence channels
Anyone with your public app key can open a socket, so Sockudo will not let a client join a
`private-*`, `presence-*`, or `private-encrypted-*` channel on its own say-so. Instead, the SDK
asks your backend to sign the subscription. Your backend checks the user's session, decides
whether that user may join that exact channel, and returns a signature made with the app secret.
Sockudo verifies the signature before admitting the socket.
The default authorizer sends a form-encoded `POST` with `socket_id` and `channel_name` to
`channelAuthorization.endpoint` (default `/sockudo/auth`) and expects JSON with `auth`, plus
`channel_data` for presence and `shared_secret` for encrypted channels.
```dart title="lib/realtime.dart"
final client = SockudoClient(
'app-key',
SockudoOptions(
cluster: 'local',
forceTls: false,
wsHost: '127.0.0.1',
wsPort: 6001,
protocolVersion: 2,
channelAuthorization: ChannelAuthorizationOptions(
endpoint: 'https://api.example.com/sockudo/auth',
headersProvider: () => {
'Authorization': 'Bearer ${session.accessToken}',
},
),
),
);
```
Use an absolute URL for the endpoint; a Flutter app has no page origin to resolve `/sockudo/auth`
against. To produce the signature yourself, pass `customHandler`, which receives a
`ChannelAuthorizationRequest` and returns `ChannelAuthorizationData`:
```dart
ChannelAuthorizationOptions(
customHandler: (request) async {
final signed = await backend.signChannel(request.socketId, request.channelName);
return ChannelAuthorizationData(
auth: signed.auth,
channelData: signed.channelData,
);
},
)
```
The endpoint must authenticate the user and authorize the exact requested channel. Presence
identity comes from the backend session, not from an untrusted request field. Never sign every
channel a client asks for.
### Presence
Subscribing to a `presence-*` name returns a `PresenceChannel`. Its `members` collection is filled
from the subscription-success payload and kept current as members join, leave, or update.
```dart title="lib/lobby.dart"
final presence = client.subscribe('presence-lobby') as PresenceChannel;
presence.bind('sockudo:subscription_succeeded', (_, __) {
presence.members.forEach((member) => print('here: ${member.id}'));
presence.update({'status': 'editing'});
});
presence.bind('sockudo:member_added', (member, _) => print('joined: $member'));
presence.bind('sockudo:member_removed', (member, _) => print('left: $member'));
presence.bind('sockudo:presence_update', (member, _) {
print((member as PresenceMember).info);
});
```
V2 presence updates modify member data without a leave and rejoin cycle. `update` throws
`SockudoException` on Protocol V1. Inbound `sockudo_internal:presence_update` updates
`channel.members` and emits `sockudo:presence_update` with the updated `PresenceMember`.
### User sign-in
Configure `userAuthentication` (default endpoint `/sockudo/user-auth`, expecting `auth` and
`user_data`) and call `client.signIn()` when the application uses user-targeted events or
watchlists. Bind user events with `client.user.on(...)` and watchlist events with
`client.watchlist.on(...)`.
### Capability tokens
Protocol V2 can authenticate the connection itself with a scoped capability token and an async
refresh callback:
```dart title="lib/realtime.dart"
final client = SockudoClient(
'app-key',
SockudoOptions(
cluster: 'local',
protocolVersion: 2,
wsHost: 'realtime.example.com',
forceTls: true,
token: initialToken,
authCallback: () async => fetchFreshSockudoToken(),
),
);
client.bind('sockudo:auth_success', (data, _) => print('token accepted'));
```
- `token` is sent as the initial WebSocket query parameter.
- When `authCallback` is set and the token is a JWT with an `exp` claim, the client decodes it
(without validating the signature) and refreshes at 80% of its lifetime. If `iat` is absent, the
lifetime starts at the local decode time. Opaque tokens, JWTs without `exp`, and static-only
`token` usage do not schedule proactive refreshes.
- After the optional initial token is used, `authCallback` supplies a fresh token before every
reconnect.
- On `sockudo:token_expired` with code `40142`, the client calls `authCallback` and sends
`sockudo:auth` without dropping the socket. Revocation code `40160` cannot refresh in place; it
is emitted to listeners and left to the server close path.
Setting `token` or `authCallback` with any other `protocolVersion` throws `SockudoException` from
the constructor rather than opening an unauthenticated socket.
## Protocol V2 features
Protocol V2 keeps the same channel model but attaches continuity metadata (`message_id`, `serial`,
`stream_id`) to broadcasts and unlocks server-side filtering, deltas, rewind, and mutable messages.
All examples below assume `protocolVersion: 2`. See [Protocol V2](/docs/clients/protocol-v2) for
the wire-level contract.
### Recovery and rewind
With `connectionRecovery: true`, the client records the latest position for each channel and sends
`sockudo:resume` after reconnecting. Rewind asks the server to replay recent history at subscribe
time, either by count (`SubscriptionRewind.count(n)`) or by age (`SubscriptionRewind.seconds(n)`).
```dart title="lib/market.dart"
final channel = client.subscribe(
'market:BTC',
options: const SubscriptionOptions(
rewind: SubscriptionRewind.seconds(30),
),
);
channel.bind('sockudo:rewind_complete', (data, _) {
final done = data as RewindCompleteData;
print('replayed ${done.historicalCount}, complete: ${done.complete}');
});
channel.bind('message', (_, __) {
print(client.getRecoveryPosition('market:BTC'));
});
client.bind('sockudo:resume_success', (data, _) {
final result = data as ResumeSuccessData;
print('recovered ${result.recovered.length} channels');
});
client.bind('sockudo:resume_failed', (data, _) {
final failed = data as ResumeFailedChannel;
print('rebuild ${failed.channel}: ${failed.code}');
});
```
On `sockudo:resume_failed`, the SDK drops the stored position and resubscribes that channel, but
your derived state may have a gap. Discard it and reload an authoritative snapshot. Persist
positions across process restarts with `getRecoveryPositions()` and `setRecoveryPositions(...)`.
After a V2 subscription succeeds, `channel.attachSerial` holds the server's `attach_serial` when
durable history is enabled. For a gap-free late join, load channel history with
`untilAttach: true`, then apply live events:
```dart
final room = client.subscribe('presence-room') as PresenceChannel;
final history = await room.channelHistory(
const ChannelHistoryParams(untilAttach: true, limit: 100),
);
```
In this SDK, `channelHistory`, `getMessage`, and `getMessageVersions` are defined on
`PresenceChannel`, and they call your `versionedMessages.endpoint` proxy. Configure that option
before using them.
### Filters
Tag filters run on the server, so non-matching messages never reach the device. Build them with the
`Filter` helpers (`eq`, `neq`, `inside`, `notIn`, `exists`, `notExists`, `startsWith`, `endsWith`,
`contains`, `gt`, `gte`, `lt`, `lte`, `and`, `or`, `not`). Event-name lists and a bounded JMESPath
`expression` can be combined with tags; every supplied component must match.
```dart title="lib/prices.dart"
final channel = client.subscribe(
'price:btc',
options: SubscriptionOptions(
filter: Filter.and([
Filter.eq('market', 'spot'),
Filter.gt('spread', '0'),
]),
events: const ['price.updated'],
expression: const SubscriptionExpression('data.price >= `100`'),
),
);
```
### Delta compression
Deltas shrink large, repetitive payloads by sending only the change against the previous message.
The SDK reconstructs Fossil and Xdelta3/VCDIFF deltas in pure Dart.
```dart
final book = client.subscribe(
'orderbook:btc-usd',
options: const SubscriptionOptions(
delta: ChannelDeltaSettings(
enabled: true,
algorithm: DeltaAlgorithm.xdelta3,
),
),
);
```
Set `deltaCompression: DeltaOptions(...)` on `SockudoOptions` to receive `onStats` and `onError`
callbacks, and read totals with `client.getDeltaStats()`. See
[Filters and delta compression](/docs/clients/filters-delta).
### Mutable messages
Protocol V2 mutable messages arrive as `sockudo:message.update`, `sockudo:message.delete`, and
`sockudo:message.append`. An update replaces local content, a delete is the latest visible version
and may carry `null` data, and an append concatenates onto the current string. Reduce them in serial
order:
```dart title="lib/chat.dart"
MutableMessageState? state;
final channel = client.subscribe('chat:room-1');
channel.bindGlobal((eventName, data) {
if (data is! SockudoEvent || !isMutableMessageEvent(data)) return;
try {
state = reduceMutableMessageEvent(state, data);
print('${state?.messageSerial} ${state?.action} ${state?.data}');
} catch (e) {
print('mutable message reduction failed: $e');
}
});
```
If an append arrives before its base, fetch the latest visible message through the proxy first.
Configure `versionedMessages` with a trusted backend endpoint to create, append, update, or delete
messages:
```dart
final created = await channel.createMessage(
const VersionedMessageCreateRequest(data: 'hello'),
);
await channel.appendMessage(created.messageSerial, ' world');
await channel.updateMessage(
created.messageSerial,
const VersionedMessageMutation(data: {'text': 'edited'}),
);
await channel.deleteMessage(created.messageSerial);
```
These helpers send `publish_create`, `message_append`, `message_update`, or `message_delete` to
`versionedMessages.endpoint` and resolve to a `MessageAck` with `messageSerial` and
`historySerial`. The default timeout is 10 seconds (`VersionedMessagesOptions.timeout`). See
[Mutable messages](/docs/server/mutable-messages).
### Presence history
Presence history is proxy-backed. The client does not sign the server REST API; configure
`presenceHistory` with a backend endpoint that accepts `{channel, params, action}` and forwards the
request with server credentials.
```dart title="lib/lobby_history.dart"
final client = SockudoClient(
'app-key',
SockudoOptions(
cluster: 'local',
forceTls: false,
wsHost: '127.0.0.1',
wsPort: 6001,
presenceHistory: const PresenceHistoryOptions(
endpoint: 'https://api.example.com/sockudo/presence-history',
),
),
);
final channel = client.subscribe('presence-lobby') as PresenceChannel;
final page = await channel.history(
const PresenceHistoryParams(limit: 50, direction: 'newest_first'),
);
if (page.hasNext()) {
final nextPage = await page.next();
}
final snapshot = await channel.snapshot(
const PresenceSnapshotParams(atSerial: 4),
);
```
The history proxy owns the app secret, authenticates the caller, checks channel access, and
forwards opaque pagination cursors. See [Presence history](/docs/clients/presence-history).
### Binary wire formats
Set `wireFormat: SockudoWireFormat.messagepack` or `SockudoWireFormat.protobuf` to negotiate a
binary encoding. Both preserve `Uint8List` message data as native binary. MessagePack uses the
additive `["binary", ]` tagged value; existing string and JSON variants are unchanged.
### Encrypted channels
`private-encrypted-*` channels decrypt automatically with libsodium when protected-channel
authorization returns the derived `shared_secret` (surfaced as `ChannelAuthorizationData.sharedSecret`):
```dart
final encrypted = client.subscribe('private-encrypted-documents');
encrypted.bind('doc-updated', (payload, _) => print(payload));
```
Keep the encryption master key on the backend. End-to-end encryption does not replace TLS or
channel authorization.
## Connection lifecycle and reconnection
The connection moves through `ConnectionState` values: `initialized`, `connecting`, `connected`,
`reconnecting`, `unavailable`, `disconnected`, and `failed`. Each change emits `state_change` with
`previous` and `current` names, followed by an event named after the new state. Read the current
value from `client.connectionState` and the socket ID from `client.socketId`.
```dart
final stateToken = client.bind('state_change', (change, _) {
print('connection: ${change['previous']} -> ${change['current']}');
});
client.bind('connected', (meta, _) => print('socket: ${client.socketId}'));
client.bind('reconnecting', (_, __) => print('reconnecting'));
// On sign-out or shutdown:
client.unbind(eventName: 'state_change', token: stateToken);
client.disconnect();
```
Unexpected disconnects emit `ConnectionState.reconnecting` and retry with a bounded quadratic delay
of 0s, 1s, 4s, 9s, up to `maxReconnectGapInSeconds` (120 seconds by default). Protocol retry and
TLS-upgrade close codes reconnect immediately. The default limit is six attempts; `null` enables
unlimited retries. The attempt counter resets after a successful connection and on explicit
`connect()` or `disconnect()` calls. When attempts run out, the state becomes `disconnected`.
```dart
const options = SockudoOptions(
cluster: 'local',
maxReconnectAttempts: 10,
maxReconnectGapInSeconds: 60,
reconnectJitter: 0.5,
);
```
Every client computes the same backoff, so clients dropped by one event retry in lockstep. Set
`reconnectJitter` to randomize each delay by that fraction: `0.5` picks uniformly from 50–100% of
the delay, `1.0` from 0–100%. The default `0.0` keeps delays exact.
Let the SDK handle temporary transport failures and resubscription. `disconnect()` stops
reconnecting and closes the socket; `close()` also closes the underlying HTTP client used for auth
and proxy requests.
On Protocol V2, Sockudo uses native WebSocket ping/pong frames for heartbeats. Dart runtimes may
still use lightweight `sockudo:ping` / `sockudo:pong` fallback messages; those are excluded from
recovery metadata such as `message_id`, `serial`, and `stream_id`.
## Error handling
| Failure | How it surfaces | What to do |
| --- | --- | --- |
| Invalid construction (empty key or cluster, token on V1, bad `appendRollupWindow`) | `SockudoException` thrown by the constructor | Fix configuration. |
| Auth endpoint returns non-200 or invalid JSON | Channel emits `subscription_error` with `type: 'AuthError'` and the error text, which includes the HTTP status | Treat `401`/`403` as a permission problem, not a reconnect loop. |
| Server error frames, decode failures, connection unavailable | Client emits `error` | Log and observe; the SDK keeps reconnecting where possible. |
| Capability token expired (`40142`) | `sockudo:token_expired`, then refresh via `authCallback` | Make sure `authCallback` can mint a fresh token. |
| Capability token revoked (`40160`) | Emitted to listeners; no in-place refresh, the server closes the connection | Sign the user out or obtain new credentials. |
| Recovery gap | `sockudo:resume_failed` with `ResumeFailedChannel` | Discard derived state and reload a snapshot. |
```dart
client.bind('error', (error, _) {
if (error is SockudoException) {
print('sockudo error ${error.statusCode}: ${error.message}');
} else {
print('sockudo error: $error');
}
});
```
Make handlers idempotent: reconnects, retries, and upstream publishing workflows can repeat logical
work even with `message_id` deduplication enabled.
## Platform notes
### Flutter lifecycle
- Scope the client above individual widgets and avoid reconnecting on rebuild.
- Remove bindings and subscriptions when their lifecycle owner is disposed.
- Keep event callbacks fast and move blocking work to an isolate or async service.
- Test background/foreground transitions, offline recovery, token expiry, and application process
recreation.
- The package declares a Flutter SDK dependency and opens sockets through `IOWebSocketChannel`
from `web_socket_channel`.
### Large serials on Dart web
On Dart web, serial values that arrive as JSON numbers may already have been rounded by the
JavaScript runtime. The SDK preserves unsafe serials as strings when decoding raw JSON,
MessagePack, Protobuf, and proxy responses where the original literal is still available. Prefer
string serials in app-level web APIs when values may exceed `2^53 - 1`.
### Push registration
Use a platform push plugin to get the provider token, then send the token to your backend:
```dart
final token = await FirebaseMessaging.instance.getToken();
await http.post(
Uri.parse('https://api.example.com/sockudo/push/devices'),
headers: {'Content-Type': 'application/json'},
body: jsonEncode({
'device_id': deviceId,
'platform': 'fcm',
'provider_token': token,
}),
);
```
The backend registers the device with Sockudo and enforces user ownership.
`SockudoPushRegistration` wraps an authenticated application proxy with typed helpers for device
registrations, channel subscriptions, publishing, scheduling, and status. `publish()` and
`publishBatch()` always send `sync: false` and should expect `202 Accepted` with a `publish_id`;
list helpers use `limit` and `cursor`.
```dart title="lib/push.dart"
final push = SockudoPushRegistration(
const PushRegistrationOptions(
endpoint: 'https://api.example.com/sockudo/push',
headers: {'Authorization': 'Bearer session-token'},
),
);
final publish = await push.publish({
'recipients':