Kafka input transport¶
Ordinary type: kafka inputs can use verified TLS, optional client certificates,
and SASL PLAIN, SCRAM-SHA-256 or SCRAM-SHA-512. These settings work with the legacy
consumer and the opt-in durable outbox consumer; acknowledgment guarantees still
depend on that path. Event Hubs retains its separate managed/workload identity
TLS/OAUTHBEARER transport. These options do not configure Kafka outputs.
Configuration¶
Use the existing string-valued ingestor options map:
ingestors:
- name: security-kafka
type: kafka
enabled: true
brokers: ["broker-1.internal:9093", "broker-2.internal:9093"]
topic: security-logs
consumer_group: logstrm
options:
tls_enabled: "true"
tls_ca_file: /etc/logstrm/kafka/ca.crt
# Optional client authentication; configure both or neither.
tls_client_cert_file: /etc/logstrm/kafka/tls.crt
tls_client_key_file: /etc/logstrm/kafka/tls.key
sasl_mechanism: scram-sha-512
sasl_username: logstrm
sasl_password_env: LOGSTRM_KAFKA_PASSWORD
| Option | Meaning |
|---|---|
tls_enabled |
Boolean string; TLS must be explicitly enabled |
tls_ca_file |
Optional PEM CA bundle; if supplied, trust this bundle only |
tls_client_cert_file, tls_client_key_file |
Optional PEM client certificate/key pair |
tls_server_name |
Optional explicit verified certificate identity/SNI; normally omit so each broker hostname is checked |
sasl_mechanism |
plain, scram-sha-256, scram-sha-512 (case-insensitive); omit for TLS-only |
sasl_username |
SASL username; required when a mechanism is selected |
sasl_password_env |
Name of a nonempty password environment variable, not its value or ${ENV:...} expansion |
Certificate options require absolute file paths; inline PEM/private-key values
are not accepted in the options map. TLS uses Go's default verified certificate
handling with a TLS 1.2 minimum.
Without tls_ca_file, trust the operating system CA pool. There is no skip-verify
option. Each advertised broker must be reachable and have a verifiable identity;
bootstrap-only connectivity does not establish that the entire cluster works.
All supported SASL mechanisms require TLS. Invalid booleans, unmatched client
certificate/key, absent credential environment variables and unsupported security
options fail setup. Common dotted/librdkafka aliases are rejected rather than
silently treated as the supported options. A literal sasl_password is not
supported. Manager and Data Plane share the structural validator; only Data Plane
reads certificate files and resolves the password at startup.
For compatibility, omitting all security settings retains the previous plaintext, unauthenticated Kafka transport. Existing deployments do not become encrypted automatically. Explicitly configure verified TLS before using remote or otherwise untrusted networks. Generic OAuth/GSSAPI/MSK IAM is not implemented by these options, and this is not certification of any hosted Kafka service.
Source options require process restart. Certificate/password rotation also requires restart; no hot credential reload is provided. Preserve the source name/group/topic identity for the same upstream; use a new identity for a different cluster meaning, as described in durable connector support.
Helm¶
Create the Secrets separately. Keep password/private-key values out of Git,
Manager documents, Helm values, rendered ConfigMaps, screenshots and command logs.
The chart can mount an existing Secret read-only with ca.crt and optional
tls.crt/tls.key at /etc/logstrm/kafka:
dataplane:
kafkaTLS:
existingSecret: logstrm-kafka-transport
extraEnvFrom:
- secretRef:
name: logstrm-kafka-password
The password Secret must contain the LOGSTRM_KAFKA_PASSWORD environment key
used in the input options. Include the full ingestor configuration under
dataplane.config.ingestors. Defaults add no Kafka certificate mount. The chart
rejects the known literal sasl_password/sasl.password options. Runtime and
Manager validation remain necessary; Helm rendering is not connection validation.
This is a partial values example, not a complete installation. Retain the existing
outbox PVC/single-owner, control API mTLS/token and explicit NetworkPolicy setup.
Kafka TLS credentials are distinct from the Data Plane control API certificate.
The Manager Kafka input form exposes transport/certificate/SASL settings and asks
only for the password environment variable name. New Kafka form entries default
to verified TLS; plaintext requires an explicit selection. Existing runtime YAML
defaults remain unchanged. The structured input model preserves options.
Confirm the generated YAML before rollout; backend validation is authoritative.
Validation boundaries¶
Local tests exercise all three SASL methods over verified mutual TLS against a Kafka-compatible test broker and the production durable runtime adapter. Negative tests cover CA/hostname, client certificate and credential failures. Those native tests alone do not establish real-broker/provider behavior, throughput, a long soak or evidence for 1 TB/day. Independently qualify the chosen broker/release candidate before deployment.
A subsequent independent local Docker qualification used pinned Kafka 4.3.1 with
verified client mTLS/SCRAM-SHA-512, Kafka plus HTTP sources, split/fan-out and a
controlled sink outage with graceful restart. Finite correctness and reconciliation
passed. See the repository engineering report
docs/implementation/secure-kafka-mixed-source-qualification-20261010.md for exact
candidate hashes, limitations and evidence. This does not qualify other real-broker
mechanisms, enterprise ACLs/HA, providers, capacity or 1 TB/day.