Skip to content

Durable worker pools

Worker pools distribute Logstrm across independent Data Plane instances. Each worker owns its own persistent SQLite outbox, scheduler and destination delivery budgets. There is no fixed application limit of four/five workers or 5 TB/day. Infrastructure limits and measured throughput still apply.

Status: opt-in implementation with local functional tests and chart-render validation. Multi-pod Kubernetes deployment, aggregate throughput, long soak, node-failure recovery and hosted provider qualification remain untested for this candidate. No 1 TB/day or 5 TB/day capacity claim follows from this feature.

Architecture and supported scope

Kafka / Event Hubs partitions, common consumer group
                |
      +---------+---------+
      |         |         |
   worker-0  worker-1  worker-N
   outbox-0  outbox-1  outbox-N   <- independent retained PVCs
      |         |         |
      +---------+---------+
       supported destinations

Use the same ingestor name, broker topic and consumer group across workers. Separate consumer groups each receive the stream rather than sharing its work. Useful source parallelism depends on partition count and distribution. Broker ownership changes do not move deliveries already accepted into an old outbox. That volume must remain recoverable even after its source partitions move.

Initial pool sources: Kafka and Azure Event Hubs. Supported durable outputs keep their existing contracts: Sentinel DCR, Splunk HEC, Azure Blob and HTTP JSON. The HTTP input, Syslog, object-storage inputs and shared legacy DLQ are not enabled in pool mode. Single-instance durable HTTP remains available. Universal connector configuration is not universal durable source support; see capability matrix.

Install in an isolated release

Start from complete tenant/source/output/security values and apply slimstream-helm/examples/worker-pool-values.yaml. Its two workers are an example, not a maximum, minimum production size or throughput recommendation.

Requirements:

  • Kubernetes >=1.32 and an encrypted CSI StorageClass supporting ReadWriteOncePod. Verify exclusion, detach behavior and node fencing before setting dataplane.workerPool.fencingConfirmed=true. RWO alone is insufficient.
  • Persistent retained storage, sufficient per-worker disk capacity, quotas and backups. Loss of an outbox volume can lose already broker-committed deliveries.
  • Explicit network policy, existing API token Secret, verified API client mTLS, actual broker authentication/ACLs and supported emitters/routes.
  • Server certificate SANs for individual pod addresses such as logstrm-dataplane-0.logstrm-workers.logstrm.svc, or an appropriate verified wildcard *.logstrm-workers.logstrm.svc. Retain the aggregate Service SAN for existing metric scrapers. Never disable verification to make pod DNS work.
  • Matching candidate Data Plane/Manager images including retirement and multi-node bootstrap. Old binaries ignore retirement markers and must not own retired PVCs.

Do not change a live singleton Deployment into a StatefulSet in place. The singleton PVC and per-ordinal claims have different names. Use an isolated release and planned cutover: stop/drain or preserve the old owner/PVC; keep a single intended source consumer group; validate pending work, checkpoint/IDs and rollback. Do not copy an active SQLite file or delete old storage to complete migration.

The pool uses OnDelete updates, so changing images/config does not automatically replace current pods. Review updates and recover each original PVC under its stable ordinal. Both scale-down and release deletion retain PVCs. Do not install an HPA with automatic downscale; PDBs do not enforce the drain protocol.

Scale up

After validating partitions, storage, CPU and receiver quotas, change dataplane.replicaCount in tenant Helm values and upgrade the release. This creates new ordinals/PVCs and updates Manager bootstrap inventory. Keep stable DNS and source names/groups. Manager startup upserts each individual endpoint; rollout requests must not use a load-balanced API Service as one node.

For a temporary manual scale, first inspect a read-only plan:

python3 slimstream-helm/scale_worker_pool.py scale \
  --namespace logstrm --name logstrm-dataplane --replicas 8

Add --apply only for the intended change. Then reconcile the same count into Helm values and Manager inventory. A previously retired ordinal stays retired until explicitly reactivated; increasing replicas alone does not resume it. Retired or absent Manager node registrations are not deleted automatically: disable obsolete nodes in Manager before selecting rollout targets.

Scale down without abandoning accepted data

Use private connectivity and the worker pool tool. The token is read from LOGSTRM_WORKER_TOKEN; do not place its value in command arguments or Git.

python3 slimstream-helm/scale_worker_pool.py scale \
  --namespace logstrm --name logstrm-dataplane --replicas 4 \
  --ca /secure/server-ca.crt --cert /secure/client.crt --key /secure/client.key \
  --drain-timeout 300 --apply

The tool retires the highest ordinals through pod-specific loopback port-forwards. TLS verifies each pod DNS name; no insecure override is supported. POST /api/v1/control/worker durably records retirement and stops source intake. The dispatcher keeps running. GET returns worker/retirement identities, retirement_durable, sources_stopped, authoritative stats and safe_to_remove. Both calls require the existing bearer token and chart-required client mTLS.

Removal proof requires joined sources and pending=blocked=0. Disk/stop/auth errors, blocked rows, unavailable pods and drain deadline failure refuse scaling. The tool does not clear deliveries, delete PVCs or retry a failed operation. Its final compare-and-swap refuses concurrent StatefulSet changes.

If scaling fails, retired workers remain retired so they can finish recovery. Do not bypass the failure with direct scale/delete. Diagnose/replay blocked work and rerun the explicit operation when ready. Reconcile Helm count only after successful scale-down. Helm's live lookup also refuses a lower count, but this is not an admission policy: privileged direct Kubernetes changes can bypass it.

Restart, recovery and explicit reactivation

A replacement retired pod starts its dispatcher with sources disabled. It must mount the original PVC/config/fingerprints. The retirement marker lives beside outbox.sqlite as outbox.sqlite.retired and belongs in recovery backups.

Reactivation is deliberately offline: stop the owner and run the matching image with the same PVC/config and -clear-worker-retirement. The command acquires the normal exclusive lock, requires pending=blocked=0, removes/fsyncs the marker and exits. It never starts consumers. Restart the intended ordinal afterwards. Mount under the confirmed CSI exclusion mechanism; never run the command concurrently with the owner or manually delete markers to override pending deliveries.

A failed node requires storage recovery and physical fencing before force-detach or force-deleting pods. Retained PVCs and flock do not replicate data or fence a partitioned machine. Rebalance/retry can duplicate accepted events across stores; receiver deduplication may be required. This remains at-least-once, not exactly-once.

Plan daily volume using measured capacity

For decimal TB/day, average source event bytes B, headroom H and measured sustainable per-worker C:

mean_EPS = TB_per_day * 10^12 / (86400 * B)
target_EPS = mean_EPS * (1 + H)
active_workers = ceil(target_EPS / C)

At 1000B/event and 50% headroom: 1 TB/day needs ~17,361 EPS, 5 TB/day ~86,806 EPS. For an assumed, not production-qualified 3000 EPS/worker, that is 6 or 29 active workers; one extra gives 7 or 30 workers. N+1 also requires measured rebalance, skew and backlog recovery. Fan-out increases output/queue/storage work; compression and filtered bytes do not redefine source-volume accounting.

python3 slimstream-helm/scale_worker_pool.py plan \
  --tb-per-day 5 --event-bytes 1000 --qualified-worker-eps 3000 \
  --headroom-percent 50 --spare-workers 1

This arithmetic always emits capacity_qualified=false. Qualify 1/2/4/... workers on the same workload, separate disk/I/O metrics and fixed limits. Then measure required aggregate source bytes/s, sink confirmations, ID reconciliation, age, backlog slope, resources and long soak, including one-owner failure/recovery, slow destinations and split/fan-out. Source partitions, storage and provider quotas can prevent linear scaling. Keep failed results; no extrapolated 5TB claim.