Server

Delta Compression

Reduce bandwidth by sending only message differences with optional entity-based conflation.

Overview

Delta compression reduces bandwidth by transmitting only the differences between consecutive messages instead of sending full payloads every time. This is especially effective for:

Market data feeds with frequent price updates
IoT sensor networks with incremental state changes
Real-time dashboards with repetitive data structures
Gaming applications with player state updates

Typical bandwidth savings:

Basic delta compression: 30-50% reduction
With conflation keys: 80-95% reduction

Quick Start

Global Configuration

Enable delta compression globally in config/config.json:

config/config.json

{
  "delta_compression": {
    "enabled": true,
    "algorithm": "fossil",
    "full_message_interval": 10,
    "min_message_size": 100,
    "max_state_age_secs": 300,
    "max_channel_states_per_socket": 100,
    "omit_delta_algorithm": false
  }
}

Per-Channel Configuration

Override settings for specific channels in your app configuration:

App Config

{
  "id": "app-id",
  "key": "app-key",
  "secret": "app-secret",
  "channel_delta_compression": {
    "market:*": {
      "enabled": true,
      "algorithm": "fossil",
      "max_messages_per_key": 100
    },
    "slow-channel": "disabled"
  }
}

Configuration Options

Global Settings

Option	Type	Default	Description
`enabled`	`boolean`	`true`	Enable delta compression globally
`algorithm`	`string`	`"fossil"`	Compression algorithm: `"fossil"` or `"xdelta3"`
`full_message_interval`	`number`	`10`	Send a full message every N messages (prevents delta chain errors)
`min_message_size`	`number`	`100`	Minimum message size in bytes to apply compression
`max_state_age_secs`	`number`	`300`	Maximum age of cached state before requiring full message
`max_channel_states_per_socket`	`number`	`100`	Maximum channels tracked per WebSocket connection
`max_conflation_states_per_channel`	`number`	`100`	Maximum conflation keys per channel (see Conflation Keys)
`conflation_key_path`	`string?`	`null`	Global JSON path for conflation key extraction
`cluster_coordination`	`boolean`	`false`	Coordinate delta state across cluster nodes
`omit_delta_algorithm`	`boolean`	`false`	Omit algorithm field from messages (saves 20-30 bytes)

Per-Channel Settings

Configure in app's channel_delta_compression object:

{
  "channel_delta_compression": {
    "pattern:*": {
      "enabled": true,
      "algorithm": "fossil",
      "conflation_key": "entity_id",
      "max_messages_per_key": 50,
      "max_conflation_keys": 1000,
      "enable_tags": false
    },
    "disable-this": "disabled"
  }
}

Option	Type	Description
`enabled`	`boolean`	Enable delta for this channel pattern
`algorithm`	`string`	Override global algorithm: `"fossil"` or `"xdelta3"`
`conflation_key`	`string?`	JSON path to extract entity identifier (see Conflation Keys)
`max_messages_per_key`	`number`	Messages to cache per entity (with conflation)
`max_conflation_keys`	`number`	Maximum entities to track
`enable_tags`	`boolean`	Include tags in messages (can disable to save bandwidth)

Algorithms

Fossil (Default)

Best for: General-purpose use, balanced speed and compression
Compression ratio: Good (typically 60-80% reduction)
Speed: Fast encoding and decoding
Use when: You want reliable compression with minimal CPU overhead

XDelta3

Best for: Maximum compression ratio
Compression ratio: Excellent (typically 70-90% reduction)
Speed: Slower than Fossil
Use when: Bandwidth is critical and CPU is available

Choosing an algorithm:

{
  "channel_delta_compression": {
    "high-frequency:*": {
      "algorithm": "fossil"  // Faster, good enough
    },
    "large-payloads:*": {
      "algorithm": "xdelta3"  // Better compression for big messages
    }
  }
}

Conflation keys dramatically improve compression efficiency by grouping messages by entity, ensuring each entity's updates are compressed against its own history rather than mixed with other entities.

The Problem Without Conflation

When multiple entities publish to the same channel:

Message 1: {"symbol": "BTC", "price": 50000}
Message 2: {"symbol": "ETH", "price": 3000}   // Compressed against BTC (poor)
Message 3: {"symbol": "BTC", "price": 50010}  // Compressed against ETH (poor)

Each message compares to a different entity, resulting in poor compression.

The Solution With Conflation

With conflation_key: "symbol":

Message 1: {"symbol": "BTC", "price": 50000}  // BTC history starts
Message 2: {"symbol": "ETH", "price": 3000}   // ETH history starts
Message 3: {"symbol": "BTC", "price": 50010}  // Compressed against BTC message 1 (excellent)

Each symbol maintains its own compression history.

Configuration

Simple path extraction:

{
  "channel_delta_compression": {
    "market:*": {
      "enabled": true,
      "conflation_key": "symbol",
      "max_messages_per_key": 100,
      "max_conflation_keys": 1000
    }
  }
}

Extracts from: {"symbol": "BTC", "price": 50000} → Key: "BTC"

Nested path extraction:

{
  "conflation_key": "data.symbol"
}

Extracts from: {"data": {"symbol": "BTC", "price": 50000}} → Key: "BTC"

Global default:

{
  "delta_compression": {
    "conflation_key_path": "asset"
  }
}

Sets a default for all channels (can be overridden per-channel).

Use Cases

Use Case	Conflation Key	Description
Market data	`"symbol"` or `"asset"`	Each stock/crypto gets its own history
IoT sensors	`"device_id"`	Each device's readings compress independently
User presence	`"user_id"`	Each user's status changes compress together
Gaming	`"player_id"`	Each player's updates compress independently
Multi-tenant logs	`"tenant_id"`	Each tenant's logs maintain separate history

When NOT to Use Conflation

Single entity per channel: No benefit if only one entity publishes
Completely random data: No patterns to exploit
Few messages per entity: Need at least 5-10 messages per entity for benefit
Memory constrained: Each entity requires cache space

Memory Planning

Formula:

Memory per channel = max_conflation_keys × max_messages_per_key × avg_message_size

Example:

1000 keys × 100 messages × 200 bytes = 20 MB per channel

Tuning recommendations:

For high-frequency, many entities:

{
  "conflation_key": "symbol",
  "max_messages_per_key": 50,      // Reduce history
  "max_conflation_keys": 5000      // Track more entities
}

For low-frequency, few entities:

{
  "conflation_key": "device_id",
  "max_messages_per_key": 200,     // Longer history
  "max_conflation_keys": 100       // Fewer entities
}

Protocol Flow

Client Subscription

When a client subscribes with delta compression enabled:

1. Client subscribes:

{
  "event": "pusher:subscribe",
  "data": {
    "channel": "market:live",
    "delta": {
      "enabled": true,
      "algorithms": ["fossil", "xdelta3"]
    }
  }
}

2. Server responds with cache sync:

{
  "event": "pusher:delta_cache_sync",
  "channel": "market:live",
  "data": {
    "enabled": true,
    "algorithm": "fossil",
    "max_messages_per_key": 100,
    "conflation_key": "symbol",
    "states": {
      "BTC": [
        {"content": "{\"symbol\":\"BTC\",\"price\":50000}", "seq": 1},
        {"content": "{\"symbol\":\"BTC\",\"price\":50010}", "seq": 2}
      ],
      "ETH": [
        {"content": "{\"symbol\":\"ETH\",\"price\":3000}", "seq": 1}
      ]
    }
  }
}

Delta Messages

Full message (periodic):

{
  "event": "pusher:message",
  "channel": "market:live",
  "data": {
    "event": "price-update",
    "data": "{\"symbol\":\"BTC\",\"price\":50020}",
    "seq": 3,
    "__full_message": true
  }
}

Delta message:

{
  "event": "pusher:delta",
  "channel": "market:live",
  "data": {
    "event": "price-update",
    "delta": "QkBUQywgIjEwMIg==",
    "seq": 4,
    "base_index": 3,
    "conflation_key": "BTC",
    "algorithm": "fossil"
  }
}

The @sockudo/client library handles this protocol automatically. Standard Pusher clients will ignore delta messages and only receive periodic full messages.

Client-Side Handling

Using @sockudo/client

Enable globally:

import Pusher from "@sockudo/client";

const pusher = new Pusher("app-key", {
  wsHost: "127.0.0.1",
  wsPort: 6001,
  forceTLS: false,
  deltaCompression: {
    enabled: true,
    algorithms: ["fossil", "xdelta3"],
    debug: false,
    onStats: (stats) => {
      console.log(`Bandwidth saved: ${stats.bandwidthSavedPercent}%`);
      console.log(`Messages: ${stats.fullMessages}/${stats.totalMessages}`);
    }
  }
});

Per-channel override:

// Enable delta for this channel only
const channel = pusher.subscribe("market:live", {
  delta: { enabled: true, algorithm: "fossil" }
});

// Disable delta for this channel
const otherChannel = pusher.subscribe("no-delta", {
  delta: { enabled: false }
});

Statistics Tracking

Monitor compression effectiveness:

const pusher = new Pusher("app-key", {
  deltaCompression: {
    enabled: true,
    debug: true,  // Enable detailed logging
    onStats: (stats) => {
      console.log({
        totalMessages: stats.totalMessages,
        deltaMessages: stats.deltaMessages,
        fullMessages: stats.fullMessages,
        bytesReceived: stats.bytesReceived,
        bytesSaved: stats.bytesSaved,
        bandwidthSavedPercent: stats.bandwidthSavedPercent,
        errorCount: stats.errorCount
      });
    },
    onError: (error) => {
      console.error("Delta decode error:", error);
    }
  }
});

Standard Pusher Clients

Clients without delta support (pusher-js, Laravel Echo) will:

Ignore pusher:delta messages
Only receive periodic full messages (every full_message_interval)
Still work correctly, but won't get bandwidth savings

Performance Tuning

Optimize for High Frequency

For channels with hundreds of messages per second:

{
  "channel_delta_compression": {
    "high-freq:*": {
      "enabled": true,
      "algorithm": "fossil",              // Faster than xdelta3
      "full_message_interval": 20,        // Less frequent full messages
      "max_messages_per_key": 50,         // Smaller cache
      "conflation_key": "entity_id",
      "max_conflation_keys": 10000
    }
  }
}

Optimize for Large Payloads

For channels with large JSON documents:

{
  "channel_delta_compression": {
    "big-data:*": {
      "enabled": true,
      "algorithm": "xdelta3",             // Better compression ratio
      "min_message_size": 500,            // Only compress larger messages
      "full_message_interval": 5,         // More frequent full messages (safety)
      "max_messages_per_key": 20
    }
  }
}

Optimize for Many Entities

For channels tracking thousands of entities:

{
  "channel_delta_compression": {
    "iot:*": {
      "enabled": true,
      "conflation_key": "device_id",
      "max_messages_per_key": 30,         // Limit per-entity cache
      "max_conflation_keys": 50000,       // Track many entities
      "max_state_age_secs": 600           // Longer retention
    }
  }
}

Bandwidth vs CPU Trade-off

Maximize bandwidth savings:

{
  "algorithm": "xdelta3",
  "full_message_interval": 50,
  "conflation_key": "entity_id"
}

Minimize CPU usage:

{
  "algorithm": "fossil",
  "full_message_interval": 5,
  "min_message_size": 200
}

Monitoring & Debugging

Server-Side Metrics

Check Prometheus metrics endpoint:

curl http://localhost:9601/metrics | grep delta

Key metrics:

sockudo_delta_compressions_total - Total delta operations
sockudo_delta_compression_ratio - Average compression ratio
sockudo_delta_full_messages_total - Full messages sent
sockudo_delta_errors_total - Compression errors

Client-Side Debugging

Enable debug mode:

const pusher = new Pusher("app-key", {
  deltaCompression: {
    enabled: true,
    debug: true  // Logs all delta operations to console
  }
});

Console output:

[Delta] Cache sync received for market:live (conflation: symbol, 2 entities)
[Delta] Delta message for BTC: seq=3, base=2, size=45 bytes
[Delta] Reconstructed message: 192 bytes (saved 147 bytes, 76%)
[Delta] Stats: 10/12 deltas, 8.5KB received, 32KB saved (79%)

Common Issues

High error rate:

Lower full_message_interval (more safety messages)
Increase max_state_age_secs (keep state longer)
Check network reliability (packet loss causes decode errors)

Poor compression ratio:

Enable conflation keys if multiple entities per channel
Check message structure (highly variable data compresses poorly)
Verify messages are similar (delta works best for incremental updates)

Memory pressure:

Reduce max_conflation_keys
Reduce max_messages_per_key
Increase max_state_age_secs and enable aggressive cleanup

Best Practices

✅ Do

Use conflation keys for multi-entity channels
Monitor bandwidth savings with client stats
Set appropriate full_message_interval (10-20 for safety)
Test with realistic data before production
Use fossil for most cases (good balance)
Set min_message_size to avoid compressing tiny messages

❌ Don't

Don't compress random/encrypted data (no patterns to exploit)
Don't use very long delta chains (errors accumulate)
Don't skip full messages (clients need periodic sync)
Don't over-cache (balance memory vs history depth)
Don't enable for low-frequency channels (overhead not worth it)

Publishing with Delta Control

Override delta behavior per-publish via HTTP API:

Force delta:

curl -X POST http://localhost:6001/apps/app-id/events \
  -H "Content-Type: application/json" \
  -d '{
    "name": "price-update",
    "channel": "market:live",
    "data": "{\"symbol\":\"BTC\",\"price\":50100}",
    "delta": true
  }'

Force full message:

curl -X POST http://localhost:6001/apps/app-id/events \
  -H "Content-Type: application/json" \
  -d '{
    "name": "price-update",
    "channel": "market:live",
    "data": "{\"symbol\":\"BTC\",\"price\":50100}",
    "delta": false
  }'

Cluster Coordination

For multi-node deployments, enable cluster coordination to sync delta state:

{
  "delta_compression": {
    "enabled": true,
    "cluster_coordination": true
  }
}

How it works:

When a client reconnects to a different node, that node can retrieve cached state
Requires adapter (Redis, NATS) to share state
Adds latency but prevents full message cascade after reconnect

When to enable:

Multi-node production clusters
Load balancer without sticky sessions
High reconnection rate

When to skip:

Single node deployments
Sticky sessions enabled
Low reconnection rate (not worth the overhead)