Skip to content

Kafka input transport

Ordinary type: kafka inputs can use verified TLS, optional client certificates, and SASL PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512. These settings work with the legacy consumer and the opt-in durable outbox consumer; acknowledgment guarantees still depend on that path. Event Hubs retains its separate managed/workload identity TLS/OAUTHBEARER transport. These options do not configure Kafka outputs.

Configuration

Use the existing string-valued ingestor options map:

ingestors:
  - name: security-kafka
    type: kafka
    enabled: true
    brokers: ["broker-1.internal:9093", "broker-2.internal:9093"]
    topic: security-logs
    consumer_group: logstrm
    options:
      tls_enabled: "true"
      tls_ca_file: /etc/logstrm/kafka/ca.crt
      # Optional client authentication; configure both or neither.
      tls_client_cert_file: /etc/logstrm/kafka/tls.crt
      tls_client_key_file: /etc/logstrm/kafka/tls.key
      sasl_mechanism: scram-sha-512
      sasl_username: logstrm
      sasl_password_env: LOGSTRM_KAFKA_PASSWORD
Option Meaning
tls_enabled Boolean string; TLS must be explicitly enabled
tls_ca_file Optional PEM CA bundle; if supplied, trust this bundle only
tls_client_cert_file, tls_client_key_file Optional PEM client certificate/key pair
tls_server_name Optional explicit verified certificate identity/SNI; normally omit so each broker hostname is checked
sasl_mechanism plain, scram-sha-256, scram-sha-512 (case-insensitive); omit for TLS-only
sasl_username SASL username; required when a mechanism is selected
sasl_password_env Name of a nonempty password environment variable, not its value or ${ENV:...} expansion

Certificate options require absolute file paths; inline PEM/private-key values are not accepted in the options map. TLS uses Go's default verified certificate handling with a TLS 1.2 minimum. Without tls_ca_file, trust the operating system CA pool. There is no skip-verify option. Each advertised broker must be reachable and have a verifiable identity; bootstrap-only connectivity does not establish that the entire cluster works.

All supported SASL mechanisms require TLS. Invalid booleans, unmatched client certificate/key, absent credential environment variables and unsupported security options fail setup. Common dotted/librdkafka aliases are rejected rather than silently treated as the supported options. A literal sasl_password is not supported. Manager and Data Plane share the structural validator; only Data Plane reads certificate files and resolves the password at startup.

For compatibility, omitting all security settings retains the previous plaintext, unauthenticated Kafka transport. Existing deployments do not become encrypted automatically. Explicitly configure verified TLS before using remote or otherwise untrusted networks. Generic OAuth/GSSAPI/MSK IAM is not implemented by these options, and this is not certification of any hosted Kafka service.

Source options require process restart. Certificate/password rotation also requires restart; no hot credential reload is provided. Preserve the source name/group/topic identity for the same upstream; use a new identity for a different cluster meaning, as described in durable connector support.

Helm

Create the Secrets separately. Keep password/private-key values out of Git, Manager documents, Helm values, rendered ConfigMaps, screenshots and command logs. The chart can mount an existing Secret read-only with ca.crt and optional tls.crt/tls.key at /etc/logstrm/kafka:

dataplane:
  kafkaTLS:
    existingSecret: logstrm-kafka-transport
  extraEnvFrom:
    - secretRef:
        name: logstrm-kafka-password

The password Secret must contain the LOGSTRM_KAFKA_PASSWORD environment key used in the input options. Include the full ingestor configuration under dataplane.config.ingestors. Defaults add no Kafka certificate mount. The chart rejects the known literal sasl_password/sasl.password options. Runtime and Manager validation remain necessary; Helm rendering is not connection validation.

This is a partial values example, not a complete installation. Retain the existing outbox PVC/single-owner, control API mTLS/token and explicit NetworkPolicy setup. Kafka TLS credentials are distinct from the Data Plane control API certificate. The Manager Kafka input form exposes transport/certificate/SASL settings and asks only for the password environment variable name. New Kafka form entries default to verified TLS; plaintext requires an explicit selection. Existing runtime YAML defaults remain unchanged. The structured input model preserves options. Confirm the generated YAML before rollout; backend validation is authoritative.

Validation boundaries

Local tests exercise all three SASL methods over verified mutual TLS against a Kafka-compatible test broker and the production durable runtime adapter. Negative tests cover CA/hostname, client certificate and credential failures. Those native tests alone do not establish real-broker/provider behavior, throughput, a long soak or evidence for 1 TB/day. Independently qualify the chosen broker/release candidate before deployment.

A subsequent independent local Docker qualification used pinned Kafka 4.3.1 with verified client mTLS/SCRAM-SHA-512, Kafka plus HTTP sources, split/fan-out and a controlled sink outage with graceful restart. Finite correctness and reconciliation passed. See the repository engineering report docs/implementation/secure-kafka-mixed-source-qualification-20261010.md for exact candidate hashes, limitations and evidence. This does not qualify other real-broker mechanisms, enterprise ACLs/HA, providers, capacity or 1 TB/day.