Skip to content

Durable connector support

Connector configuration support and durable delivery support are separate. The opt-in durable_outbox path writes transformed per-destination events to SQLite/WAL with synchronous=FULL, then delivers them independently. It requires one active owner, persistent storage and verified client mTLS for the API/HTTP. The legacy asynchronous path remains available with its existing, different crash-safety limitations.

Current capability matrix

Direction Connector Durable mode Confirmation boundary
Input Azure Event Hubs Implemented Persist source group, commit broker offsets, checkpoint
Input Kafka-compatible broker Implemented Same ordered-position contract as Event Hubs
Input HTTP JSON /ingest Implemented Entire bounded request committed before HTTP 202
Input RabbitMQ, SQS, Pub/Sub Not enabled Message ACK/visibility/receipt lifecycle still required
Input Blob/Event Grid, S3, GCS Not enabled Object version/position recovery still required
Input Syslog TCP/UDP Not enabled Protocol-specific guarantees still required; UDP has no sender ACK
Output Azure Blob Implemented Synchronous SDK upload success
Output Sentinel DCR Implemented Successful response for each DCR request; indexing is separate
Output Splunk HEC Implemented Event POST acceptance; positive indexer ACK when explicitly enabled
Output HTTP JSON / NDJSON Implemented Whole-request 2xx acceptance; no per-item response parsing
Output Other configured connector types Not enabled Synchronous outcome adapters must be implemented and qualified

“Implemented” does not mean every provider, throughput or outage duration has been validated. Current multi-source changes have local contract/integration coverage; live provider and capacity qualification are separate activities. Existing Azure test results remain tied to their exact historical binaries.

Kafka source identity and transport

Several enabled kafka/azure_eventhub entries may share the outbox. Receipts use configured ingestor name, consumer group, topic, partition and offset. Preserve that identity across restart. Changing the upstream cluster/topic meaning under an old identity can incorrectly suppress records: use a new ingestor name/group or a separately migrated store. Use distinct ingestor names; http is reserved while global HTTP ingestion is enabled. Changing source configuration requires restart. A malformed record can pause its consumer; there is no automatic skip.

Event Hubs retains verified TLS and managed/workload identity authentication. Ordinary Kafka supports explicit verified TLS, optional client certificates and SASL PLAIN/SCRAM through Kafka input transport options. Omitting those options retains legacy plaintext. Do not assume an authenticated deployment merely because type: kafka validates; qualify the actual broker, credentials, ACLs and advertised listeners. Current local compatible-broker tests do not certify a hosted Kafka service or production capacity.

Ordered-source transaction sizing

durable_outbox.group_commit.max_records accepts1..1024 (default25). max_wait remains at most1s (default5ms). Source and serialized byte caps still apply and can flush earlier; increasing record count does not increase them. The1024 ceiling is an explicit capacity option, not a recommended default or a throughput guarantee. Larger groups trade coalescing/processing latency for fewer transactions. SQL statement chunks share one atomic WAL/FULL transaction. Kafka polling is bounded by remaining group capacity and defers graceful revocation through durable enqueue, broker commit and local checkpoint. Failed or expired members and ambiguous acknowledgments still permit at-least-once replays; separate owners do not share a global deduplication database.

Optional WAL checkpoint trigger

durable_outbox.wal_autocheckpoint_pages is restart-only:0 keeps SQLite's default (1000 pages in the pinned library),1..16384 selects an explicit trigger. WAL and synchronous=FULL remain enabled.8192 pages is approximately32MiB with4096B pages; it is a diagnostic/capacity option, not a new default. A larger trigger can amortize checkpoint work, trading WAL disk footprint/recovery work for throughput. It does not reduce per-commit durability. It is NOT a hard WAL size cap: long readers and large transactions can exceed it. Provision/monitor physical disk beyond logical outbox max_bytes; slimstream_outbox_file_bytes reports owner file-stat sizes. The Helm field is dataplane.outbox.walAutoCheckpointPages.

HTTP durable acceptance

Enable global.http.listen together with durable_outbox.enabled. On this path:

  • POST /ingest accepts one object or an array, at most 1000 events, with the configured global.http.max_body_size (10 MiB default) and four decoder slots.
  • All transformed routed copies for a request must fit both outbox max_bytes and group_commit.max_serialized_bytes (default min(4 MiB, outbox capacity)). Fan-out counts toward this limit. Split larger requests at the sender.
  • A request is one atomic transaction, not a sequence of accepted prefixes. The broker group-record cap/linger do not split this HTTP transaction.
  • HTTP 202 returns accepted, routed_deliveries, policy_dropped, acceptance: "durable", durable: true, delivered: false. Intentionally filtered events count as policy drops and retain no payload.
  • HTTP 413 rejects oversized body/event count/transformed payload. HTTP 503 signals admission/capacity/storage failure and sets Retry-After. Malformed input is 400. Admission errors report acceptance: "unconfirmed": a commit error or lost response can leave acceptance uncertain; zero confirmed accepts is not proof that no durable write occurred.
  • No durable request-idempotency key is implemented. Lost responses, retries, uncertain commit outcomes and remote acceptance before a crash can duplicate. Use stable event IDs and receiver-side deduplication where required.
  • Sender success acknowledges local durable routing decisions; it does not prove receipt/indexing by Sentinel, Splunk or another endpoint.

HTTP requests leave no broker receipts/checkpoints. Only pending/blocked routed copies remain until delivered/replayed. Storage loss, source retention expiry and unbounded outages are not protected by a local disk queue.

HTTP JSON destination

type: http_json sends NDJSON with application/x-ndjson. Durable sends enforce batch.max_events and exact serialized bytes including newline framing. Each request has its own outcome; successful chunks are separated from failed and unattempted chunks. There are no nested emitter retry/DLQ writes on this path. Timeout/cancellation after submission can have an unknown remote outcome.

Use only endpoints where a 2xx confirms the entire request. This generic adapter is not a replacement for Elasticsearch bulk, Splunk HEC or an API that embeds per-item failures in a successful HTTP response. Use verified HTTPS for remote endpoints. The Manager does not gain a new HTTP JSON form in this change; author it through YAML.

Multi-instance scope

Opt-in durable worker pools use independent retained outboxes with broker partition distribution. The initial pool chart enables only Kafka/Event Hubs inputs, without shared legacy DLQ. Durable HTTP remains single instance in Helm. Pool mechanics do not qualify all connectors, aggregate daily capacity or cloud failure recovery. At-least-once duplication can cross workers on rebalance; broker source ACK does not transfer an old owner's pending deliveries.

Helm opt-in and migration

Existing durable releases keep HTTP disabled. To explicitly enable it:

dataplane:
  outbox:
    enabled: true
    httpIngest:
      enabled: true
  config:
    httpListen: ":8080"
networkPolicy:
  ingress:
    - from:
        - ipBlock:
            cidr: 10.20.0.0/16
      ports:
        - protocol: TCP
          port: 8080

This fragment is not a complete installation. Retain the existing single-replica Deployment, encrypted persistent PVC, private API token, required verified client certificate, Manager OIDC/client certificate, explicit egress/Manager ingress and valid routes/destinations. The opt-in renders the separate ingestion Service and listener; it does not expose the control API. Sender certificates must be trusted by the configured client CA. Use a candidate image that includes this feature: older binaries reject the expanded source/output configuration. Disabling the opt-in removes the listener/Service; drain or retain the outbox before migration.

Schema remains v2; pending HTTP rows use the existing delivery columns. An older binary can read the database, but cannot necessarily run the new configuration or HTTP JSON pending destinations. Schema readability is not a blanket rollback permission. Preserve the matching configuration/image and rehearse rollback.