Durable connector support¶
Connector configuration support and durable delivery support are separate.
The opt-in durable_outbox path writes transformed per-destination events to
SQLite/WAL with synchronous=FULL, then delivers them independently. It requires
one active owner, persistent storage and verified client mTLS for the API/HTTP.
The legacy asynchronous path remains available with its existing, different
crash-safety limitations.
Current capability matrix¶
| Direction | Connector | Durable mode | Confirmation boundary |
|---|---|---|---|
| Input | Azure Event Hubs | Implemented | Persist source group, commit broker offsets, checkpoint |
| Input | Kafka-compatible broker | Implemented | Same ordered-position contract as Event Hubs |
| Input | HTTP JSON /ingest |
Implemented | Entire bounded request committed before HTTP 202 |
| Input | RabbitMQ, SQS, Pub/Sub | Not enabled | Message ACK/visibility/receipt lifecycle still required |
| Input | Blob/Event Grid, S3, GCS | Not enabled | Object version/position recovery still required |
| Input | Syslog TCP/UDP | Not enabled | Protocol-specific guarantees still required; UDP has no sender ACK |
| Output | Azure Blob | Implemented | Synchronous SDK upload success |
| Output | Sentinel DCR | Implemented | Successful response for each DCR request; indexing is separate |
| Output | Splunk HEC | Implemented | Event POST acceptance; positive indexer ACK when explicitly enabled |
| Output | HTTP JSON / NDJSON | Implemented | Whole-request 2xx acceptance; no per-item response parsing |
| Output | Other configured connector types | Not enabled | Synchronous outcome adapters must be implemented and qualified |
“Implemented” does not mean every provider, throughput or outage duration has been validated. Current multi-source changes have local contract/integration coverage; live provider and capacity qualification are separate activities. Existing Azure test results remain tied to their exact historical binaries.
Kafka source identity and transport¶
Several enabled kafka/azure_eventhub entries may share the outbox. Receipts use
configured ingestor name, consumer group, topic, partition and offset. Preserve
that identity across restart. Changing the upstream cluster/topic meaning under
an old identity can incorrectly suppress records: use a new ingestor name/group
or a separately migrated store. Use distinct ingestor names; http is reserved
while global HTTP ingestion is enabled. Changing source configuration requires
restart. A malformed record can pause its consumer; there is no automatic skip.
Event Hubs retains verified TLS and managed/workload identity authentication.
Ordinary Kafka supports explicit verified TLS, optional client certificates and
SASL PLAIN/SCRAM through Kafka input transport options.
Omitting those options retains legacy plaintext. Do not assume an authenticated
deployment merely because type: kafka validates; qualify the actual broker,
credentials, ACLs and advertised listeners. Current local compatible-broker tests
do not certify a hosted Kafka service or production capacity.
Ordered-source transaction sizing¶
durable_outbox.group_commit.max_records accepts1..1024 (default25).
max_wait remains at most1s (default5ms). Source and serialized byte caps still
apply and can flush earlier; increasing record count does not increase them.
The1024 ceiling is an explicit capacity option, not a recommended default or a
throughput guarantee. Larger groups trade coalescing/processing latency for fewer
transactions. SQL statement chunks share one atomic WAL/FULL transaction.
Kafka polling is bounded by remaining group capacity and defers graceful
revocation through durable enqueue, broker commit and local checkpoint. Failed
or expired members and ambiguous acknowledgments still permit at-least-once
replays; separate owners do not share a global deduplication database.
Optional WAL checkpoint trigger¶
durable_outbox.wal_autocheckpoint_pages is restart-only:0 keeps SQLite's default
(1000 pages in the pinned library),1..16384 selects an explicit trigger. WAL and
synchronous=FULL remain enabled.8192 pages is approximately32MiB with4096B
pages; it is a diagnostic/capacity option, not a new default. A larger trigger can
amortize checkpoint work, trading WAL disk footprint/recovery work for throughput.
It does not reduce per-commit durability. It is NOT a hard WAL size cap: long
readers and large transactions can exceed it. Provision/monitor physical disk
beyond logical outbox max_bytes; slimstream_outbox_file_bytes reports owner
file-stat sizes. The Helm field is dataplane.outbox.walAutoCheckpointPages.
HTTP durable acceptance¶
Enable global.http.listen together with durable_outbox.enabled. On this path:
- POST
/ingestaccepts one object or an array, at most 1000 events, with the configuredglobal.http.max_body_size(10 MiB default) and four decoder slots. - All transformed routed copies for a request must fit both outbox
max_bytesandgroup_commit.max_serialized_bytes(default min(4 MiB, outbox capacity)). Fan-out counts toward this limit. Split larger requests at the sender. - A request is one atomic transaction, not a sequence of accepted prefixes. The broker group-record cap/linger do not split this HTTP transaction.
- HTTP 202 returns
accepted,routed_deliveries,policy_dropped,acceptance: "durable",durable: true,delivered: false. Intentionally filtered events count as policy drops and retain no payload. - HTTP 413 rejects oversized body/event count/transformed payload. HTTP 503 signals
admission/capacity/storage failure and sets
Retry-After. Malformed input is 400. Admission errors reportacceptance: "unconfirmed": a commit error or lost response can leave acceptance uncertain; zero confirmed accepts is not proof that no durable write occurred. - No durable request-idempotency key is implemented. Lost responses, retries, uncertain commit outcomes and remote acceptance before a crash can duplicate. Use stable event IDs and receiver-side deduplication where required.
- Sender success acknowledges local durable routing decisions; it does not prove receipt/indexing by Sentinel, Splunk or another endpoint.
HTTP requests leave no broker receipts/checkpoints. Only pending/blocked routed copies remain until delivered/replayed. Storage loss, source retention expiry and unbounded outages are not protected by a local disk queue.
HTTP JSON destination¶
type: http_json sends NDJSON with application/x-ndjson. Durable sends enforce
batch.max_events and exact serialized bytes including newline framing. Each
request has its own outcome; successful chunks are separated from failed and
unattempted chunks. There are no nested emitter retry/DLQ writes on this path.
Timeout/cancellation after submission can have an unknown remote outcome.
Use only endpoints where a 2xx confirms the entire request. This generic adapter is not a replacement for Elasticsearch bulk, Splunk HEC or an API that embeds per-item failures in a successful HTTP response. Use verified HTTPS for remote endpoints. The Manager does not gain a new HTTP JSON form in this change; author it through YAML.
Multi-instance scope¶
Opt-in durable worker pools use independent retained outboxes with broker partition distribution. The initial pool chart enables only Kafka/Event Hubs inputs, without shared legacy DLQ. Durable HTTP remains single instance in Helm. Pool mechanics do not qualify all connectors, aggregate daily capacity or cloud failure recovery. At-least-once duplication can cross workers on rebalance; broker source ACK does not transfer an old owner's pending deliveries.
Helm opt-in and migration¶
Existing durable releases keep HTTP disabled. To explicitly enable it:
dataplane:
outbox:
enabled: true
httpIngest:
enabled: true
config:
httpListen: ":8080"
networkPolicy:
ingress:
- from:
- ipBlock:
cidr: 10.20.0.0/16
ports:
- protocol: TCP
port: 8080
This fragment is not a complete installation. Retain the existing single-replica Deployment, encrypted persistent PVC, private API token, required verified client certificate, Manager OIDC/client certificate, explicit egress/Manager ingress and valid routes/destinations. The opt-in renders the separate ingestion Service and listener; it does not expose the control API. Sender certificates must be trusted by the configured client CA. Use a candidate image that includes this feature: older binaries reject the expanded source/output configuration. Disabling the opt-in removes the listener/Service; drain or retain the outbox before migration.
Schema remains v2; pending HTTP rows use the existing delivery columns. An older binary can read the database, but cannot necessarily run the new configuration or HTTP JSON pending destinations. Schema readability is not a blanket rollback permission. Preserve the matching configuration/image and rehearse rollback.