Indestructibility simulator

The Sockudo simulator is a seed-replayable disaster lab for durable Protocol V2 state. It runs inside one process, drives real sockudo-core memory stores, calls the real sockudo-push memory store and accept pipeline for durable push boundaries, injects node, network, IO, queue, and fake-provider faults, and checks a shadow model continuously.

Use it to prove that Sockudo-side durability contracts survive dropped live fanout, duplicated delivery, node crashes, pauses, partitions, stream resets, retention purges, reconnect recovery, storage-level dropped writes, torn multi-record writes, stale or corrupted reads, delayed commit visibility, push queue loss, lost write responses, retryable provider failures, invalid tokens, and repair.

The simulator deliberately does not claim that APNs, FCM, WebPush, HMS, WNS, browsers, mobile OSes, radios, or devices will deliver every notification. External providers are modeled as fallible systems. "Indestructible" means Sockudo does not lose, corrupt, double-apply logically, or make unrecoverable an accepted Sockudo-side operation.

cargo run -p sockudo-simulator --bin sockudo-sim -- --seed 42 --ticks 10000

or through the Makefile wrapper:

make simulator SIM_SEED=42 SIM_TICKS=10000

For a disaster-heavy JSON profile:

make simulator-disaster SIM_SEED=12648430 SIM_TICKS=50000

For VOPR-style randomized distributions and liveness checks:

make simulator-swarm SIM_SEED=3735928559 SIM_TICKS=50000
make simulator-liveness SIM_SEED=3735928559 SIM_TICKS=50000

Every failure prints a replay command with the seed. Keep the seed, tick count, and fault arguments unchanged when reducing a failure.

What It Exercises

The simulator covers these high-value durability surfaces:

It does not replace live multi-node integration or Jepsen-style external testing. It is the fast, deterministic inner loop for the durable primitives those tests rely on.

Outside-In Binary Chaos Harness

Sockudo also has a separate outside-in chaos runner for manual local experiments against actual sockudo server binaries and real client traffic. It is inspired by TigerBeetle Vortex in spirit, but it is not deterministic, not a CI gate, and not part of the simulator's seed-replayable model.

make binary-chaos CHAOS_SEED=42 CHAOS_DURATION_MS=12000

The Make target builds target/debug/sockudo, starts it as a child process with a generated local memory-backed config, connects clients over WebSocket, publishes signed HTTP API events, and injects bounded outside-in faults:

Every run writes an artifact directory under target/outside-in-chaos/<timestamp>-seed-<seed>/ containing:

Push-provider exercise is opt-in because it requires a compatible feature build:

make binary-chaos-push CHAOS_SEED=42 CHAOS_DURATION_MS=12000

For network experiments, the default --network-fault-mode proxy is unprivileged and local-only. Use --network-fault-mode publisher to limit faults to HTTP publisher delay/drop behavior, or --network-fault-mode off to run only process/client/config/push chaos. The replay command in artifact.json reuses the same seed, ports, timings, probabilities, and push-provider settings; outside-in wall-clock scheduling can still vary between runs.

Sockudo's provider dispatch layer rejects private/local provider URLs by design. If that guard prevents the server process from calling the local mock provider, the artifact records the guarded dispatch attempt and the harness performs a direct mock-provider probe so the fake provider outcome is still captured with the same seed.

Keep this harness local/manual only. Do not add CI, scheduled jobs, or GitHub Actions for it.

Real-Code Boundary

The simulator is intentionally not a full server-in-a-process. Its current real-code boundary is:

Still modeled: live socket fanout, horizontal transport timing, replay-buffer hot recovery, operator routing, push worker queues after accept, provider dispatch, provider feedback, and repair timing. Those modeled paths exist to make deterministic faults cheap and replayable; they are checked against the durable stores above rather than replacing them.

Deterministic IO Model

The simulator keeps deterministic IO under crates/sockudo-simulator/src/io.rs. The production server does not use these wrappers; they are simulator-local boundaries around modeled IO and real memory-store calls.

The final JSON report includes a bounded io_trace with the most recent operation, fault, scheduling, and logical-timer decisions. That trace is intentionally compact: it is for replay orientation, not a full event log. Use the seed, tick count, mode, fault flags, and workload weights as the canonical replay input.

To inspect scheduler and timer decisions for a seed:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --seed 42 \
  --ticks 10000 \
  --json | sed '1d' | jq -r '.io_trace[]'

To replay a saved failure capsule and print the same trace window:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file /tmp/sockudo-sim-failure.json \
  --json | jq -r '.[0].io_trace[]'

Current limitations:

VOPR-Style Maturity

The simulator has three layers:

Swarm profiles are deterministic: the seed generates both the simulator run and the profile distribution. A failing swarm seed can be replayed without recording a separate config.

Safety mode answers: did accepted Sockudo-side work remain correct? Liveness mode adds: after faults stop, did durable work become available and drain within a bounded number of ticks?

Use failure capsules when saving CI failures:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --seed 42 \
  --ticks 50000 \
  --swarm \
  --failure-artifact /tmp/sockudo-sim-failure.json

Use shrink mode to reduce the replay before debugging. The shrinker first proves the original config still fails, then deterministically tries smaller tick, operation, and fault prefixes, topology reductions, and workload/fault profile simplifications. It accepts only candidates that still fail and writes a smaller failure capsule:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file /tmp/sockudo-sim-failure.json \
  --shrink-failure \
  --shrink-output /tmp/sockudo-sim-failure.shrunk.json

The shrink command prints the exact replay command, and the artifact stores the same command in replayCommand:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file /tmp/sockudo-sim-failure.shrunk.json

When shrinking a --swarm run directly from seed flags, the artifact also includes seedDerivedProfile. That preserves the original seed-generated topology, workload, and fault profile before any shrink simplifications, while replay continues to use the explicit shrunk config from the capsule. Do not pass --swarm when replaying a failure capsule.

Rolling Upgrade Replay

The simulator has an opt-in upgrade risk profile for production-style rolling changes. It keeps the run deterministic while modeling:

Run the manual local wrapper:

make simulator-upgrade SIM_SEED=48879 SIM_TICKS=1200

Or replay the same profile directly and inspect upgrade coverage:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --seed 48879 \
  --ticks 1200 \
  --upgrade-risk-profile \
  --upgrade-require-coverage \
  --json | sed '1d' | jq '.upgrade, .protocol_oracles, .push'

Use a compressed schedule when reducing a failure or forcing the rollout to happen early in a short manual run:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --seed 43981 \
  --ticks 320 \
  --nodes 4 \
  --upgrade-risk-profile \
  --upgrade-require-coverage \
  --upgrade-schema-prepare-tick 8 \
  --upgrade-start-tick 12 \
  --upgrade-schema-activate-tick 18 \
  --upgrade-restart-duration-ticks 2 \
  --upgrade-interval-ticks 8 \
  --json

To preserve a failing upgrade run for exact replay:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --seed 48879 \
  --ticks 1200 \
  --upgrade-risk-profile \
  --upgrade-require-coverage \
  --failure-artifact /tmp/sockudo-upgrade-failure.json

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file /tmp/sockudo-upgrade-failure.json \
  --json

The replay command printed on invariant failure includes the upgrade timing flags. The full failure artifact remains the canonical replay input when workload weights, fault probabilities, topology, or retention settings were changed in addition to the upgrade flags.

Fault Model

The default workload is a weighted deterministic generator, not an ad hoc random branch in the runner. Each tick samples one action from these weights:

--swarm replaces these fixed defaults with seed-derived distributions. Use fixed weights when reproducing a concrete bug and swarm weights when hunting for new combinations.

Set a weight to 0 to disable that action family. At least one action weight must be non-zero. The final JSON report includes the weights and selected action counts so coverage is visible for a seed.

The default fault model injects:

Rejected operations are part of the model: a workload request may route to a crashed or partitioned node and fail before it reaches durable storage. The shadow only advances after durable Sockudo stores accept the operation.

Storage faults are injected at simulator-local boundaries. Dropped core writes are rejected before serial reservations so the model does not create artificial gaps. Torn writes are limited to multi-record operations where a safe prefix can exist, such as a versioned-message history row committing before its version row. Delayed commits are tracked as per-channel visible prefixes; a recovery read may lag, but it must never expose a gap or invalid cursor. Corrupted reads fail closed and are retried by later recovery probes.

Push Oracles

Push disasters are checked against Sockudo-side invariants:

CI Profile

A good pull-request smoke profile is:

cargo test -p sockudo-simulator
cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --seed 12648430 \
  --ticks 5000 \
  --json

To replay the checked-in seed corpus:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file crates/sockudo-simulator/corpus/disaster-seeds.json \
  --ticks 5000 \
  --json

To replay the storage-fault corpus:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file crates/sockudo-simulator/corpus/storage-faults.json \
  --json

For nightly burn-in, run multiple fixed seeds and keep any new failing seed as a regression test:

for seed in 1 2 3 4 5 12648430 3735928559; do
  cargo run -p sockudo-simulator --bin sockudo-sim -- --seed "$seed" --ticks 50000
done

For nightly swarm burn-in:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file crates/sockudo-simulator/corpus/swarm-seeds.json \
  --ticks 50000 \
  --swarm

For liveness-focused nightly burn-in:

cargo run -p sockudo-simulator --bin sockudo-sim -- \
  --corpus-file crates/sockudo-simulator/corpus/swarm-seeds.json \
  --ticks 50000 \
  --mode liveness \
  --swarm

Reading Failures

Invariant failures include the seed and simulator tick:

sockudo-sim: FAILED - reproduce with --seed 42
simulator invariant failed at tick 812 (seed=42): history serial gap on sim-channel-1: 41 then 43
replay command:
  cargo run -p sockudo-simulator --bin sockudo-sim -- --seed 42 --ticks 5000 --mode safety
config_json:
  {"seed":42,"ticks":5000,...}
recent trace:
  tick=807 push queue produce lost for PublishLog { publish_id: "push-00000000000000000042" }
  tick=811 node 2 restarted

Failure artifacts written with --failure-artifact include the full config and can be replayed directly with --corpus-file. Shrunk artifacts add a shrink block with the original and reduced tick, operation, and fault counts plus each accepted simplification step.

Treat these as product bugs unless the invariant is deliberately stronger than the subsystem's documented contract. If the contract changes, update the simulator shadow and this page in the same change.

JSON Shape

The final JSON includes top-level core counters, a protocol_oracles object with V1/V2 delivery, cursor, identity, monotonicity, rewind, and presence-edge check counts, storage fault counters, restart recovery check counts, a nested push object with durable status/log counts, status transition counts, provider results, queue losses, repair counts, phase counts for upgrade runs, a nested upgrade object with rolling restart/schema gate/oracle coverage counters, recent push trace entries, and top-level io_trace entries for recent deterministic operation/fault decisions. Keep sample outputs with the exact seed and tick count that produced them so they remain replayable. A compact example generated with --seed 12648430 --ticks 16 --json lives at crates/sockudo-simulator/examples/disaster-report.example.json.

Extending It

Add new workloads by following the existing pattern:

1. Generate the operation through the deterministic scheduler or workload generator.
2. Apply it to the real Sockudo store or subsystem first.
3. Advance the shadow only after the real operation succeeds.
4. Add a cheap per-run oracle and a quiesce oracle.
5. Add or pin a seed that fails before the fix and passes after it.