Durable Sentinel capacity candidate¶
This is an opt-in qualification configuration, not a measured 1 TB/day capacity claim. Existing defaults remain conservative. The integrated candidate combines bounded source group commit, enqueue-triggered scheduler wake-up, concurrent reserved delivery batches and Sentinel DCR request-to-ID accounting in one build. Earlier reports for a different build do not qualify this candidate.
Effective controls¶
| Control | Candidate | Scope |
|---|---|---|
| Source group maximum | 128 records, 1 MiB source / 4 MiB planned deliveries, 5 ms wait | Atomic receipts/deliveries before Kafka offset commit |
| DCR scheduler batch | 500 events / 512 KiB JSON array, 100 ms maximum wait | Whichever size/wait condition triggers first |
| Concurrent DCR batches | 4 | One emitter; bounded by global admission |
| Global admission | 16 tasks / 64 MiB canonical event JSON | Not a Go heap/RSS or wire buffer limit |
| DCR request | 500 events / 512 KiB, gzip | Emitter splits further if necessary |
| Durable payload budget | 32 GiB | Logical outbox payload, not total SQLite/WAL size |
global.performance.worker_pool_size, batch_size and flush_interval remain compatibility fields with no execution effect. Use durable_outbox.group_commit, durable_outbox.dispatcher and the emitter's batch fields. The asynchronous emitter's batch.flush_interval does not flush the durable outbox.
The complete YAML example is slimstream/examples/capacity-sentinel.yaml in the monorepo. Set its environment variables, TLS paths, identities and destination schema before use. Validation does not prove connectivity or authorization. This example routes everything to Sentinel solely for path qualification; production policies must reflect detection and retention requirements.
The Helm overlay is slimstream-helm/examples/capacity-sentinel-values.yaml. It proposes a single worker with 4 CPU, 8 GiB RAM and a 128 GiB persistent disk for reproducible qualification. These are experiment allocations, not established requirements. Merge it with complete tenant values and set the existing sentinel-capacity emitter to batch.max_events: 500, max_bytes: 524288, compression: gzip. Rename the dispatcher override if the configured emitter uses another name. The overlay does not enable persistence, replace secrets or configure cloud endpoints by itself.
Changing emitter configuration changes its fingerprint: do not retarget pending/blocked records through reload. Drain or use a new isolated test database before changing emitter batch settings. Outbox dispatch/source grouping settings are restart-only.
The local enqueue candidate uses bounded SQL chunks for receipt checks/inserts and delivery inserts, with at most 900 bound parameters per statement. These chunks remain inside the same atomic source-group or HTTP-request transaction; they do not change durability, configuration limits or source acknowledgements. This optimization alone does not establish a throughput or 1 TB/day guarantee. Ordered-source controllers also check receipts/checkpoints in bounded groups before routing. This is an advisory read: enqueue still rechecks deduplication inside its write transaction. These changes introduce no new tuning knobs and do not justify increasing worker count, RAM or replica limits.
Delivery and failure boundaries¶
flowchart LR
EH[Event Hubs partitions] --> R[Routing and source group]
R --> O[Private SQLite outbox: WAL / FULL]
O --> S[Bounded scheduler: reserved IDs]
S --> D[Up to 4 DCR batches]
D --> A[HTTP result per request]
A --> P[Atomic success or failure persistence]
P --> O
The scheduler validates the full sequential request-to-ID mapping before acknowledging success. If the second request fails, IDs from the first successful request can be removed atomically, the second request's IDs follow the retry policy, and unsent tail IDs stay pending with unchanged attempt counts. Unattempted work has no successful/failed delivery-attempt metric. A 250 ms emitter-local cooldown prevents immediate hot retries of an unsent tail while other emitters can continue.
Results are persisted by the coordinator after the delivery task returns; this implementation does not stream each request's result during an ongoing multi-request task. Align scheduler and DCR limits to reduce that interval. A crash after remote acceptance but before SQLite success persistence can duplicate delivery. Timeouts have unknown remote outcomes. This is at-least-once, without ordering or exactly-once guarantees.
Reservations and memory admission are finite. Slow destinations can ultimately exhaust the shared logical outbox/disk budget and backpressure intake. Monitor actual filesystem free space, WAL growth, blocked records and source lag as well as logical payload bytes. Additional RAM does not remove a network/quota/disk bottleneck.
Path to 1 TB/day¶
1 TB/day (decimal, uncompressed source bytes) is approximately 11.574 MB/s, or 11,574 events/s at 1000 bytes/event. Routing, fan-out, JSON framing and retention change downstream and disk volumes. At 500 events/request this is about 23.15 requests/s if batches actually fill; 500 is a configured maximum, not an observed average.
First qualify one worker against local DCR-compatible HTTP with controlled latency, then a real source/destination with separately budgeted capacity. Record actual batch histograms, active-window delivery, age/lag trends, CPU, GC, RSS, disk and request errors. Never infer downstream throughput from intake plus an eventual drain.
Before cloud qualification, inspect current namespace ingress/egress limits, partition count, shared traffic, DCR quotas, VM CPU credits and disk IOPS. A shared small Event Hubs namespace or burstable test VM may constrain the experiment. Check current service limits and obtain a resource/cost budget before altering them.
Horizontal scaling is the next architecture milestone, not enabled by this profile. Each worker needs its own stable identity and persistent outbox. A failed worker's retained database must be recovered/drained after partition reassignment: another consumer cannot recover deliveries already checkpointed into the failed worker's private database merely by joining the same group. Keep the chart's durable single-replica restriction until StatefulSet/PVC ownership, recovery fencing and scale-down drain are implemented and tested. Never share one SQLite database between replicas.
For planning only: four workers each independently qualified at 6000 EPS would provide 24,000 EPS, or 18,000 EPS with one unavailable. Neither the per-worker qualification nor the failure-recovery protocol has been demonstrated here. One hour of 1 TB/day source traffic is about 41.7 GB before fan-out, indexes and WAL; size disk and retention from the supported outage window.