Skip to content

Connectors

Logstrm separates ingestors (event inputs) from emitters (event outputs). A configuration can combine multiple inputs, pipelines and destinations; routes decide which processed events are sent to each emitter.

Event inputs

Input Configuration role Notes
Azure Event Hubs via Kafka API Individual cloud ingestor Requires TLS and SASL/OAUTHBEARER with Managed Identity or AKS Workload Identity; configure brokers/topic and a receiver-scoped identity. Set eventhub.resource_logs: true for Azure Monitor resource-log envelopes.
Kafka-compatible brokers Individual ingestor Configure brokers and topic for the source cluster.
Azure Blob / Event Grid Individual cloud ingestor BlobCreated notifications trigger blob download and JSON/JSONL record decoding. This input is separate from the Azure Blob output emitter; raw Azure Monitor resource-log envelopes should use the Event Hubs resource_logs mode.
AWS S3 Individual cloud ingestor The Manager UI requires a bucket; region and object prefix are optional.
AWS SQS Individual cloud ingestor The Manager UI requires a queue URL; region is optional.
GCP GCS Individual cloud ingestor The Manager UI requires a bucket; region is optional.
GCP Pub/Sub Individual cloud ingestor The Manager UI requires a project ID and subscription.
RabbitMQ Individual message-queue ingestor Requires an AMQP/AMQPS URL and queue; deliveries are acknowledged after successful event handling and requeued on handler failure.

HTTP and Syslog are global Data Plane listeners, not individual entries in ingestors: HTTP accepts agent/application requests, while Syslog supports TCP and/or UDP. Their settings are written under global.http and global.syslog. These listeners are independent from cloud/Kafka ingestors, and multiple ingestors plus either or both listeners may be configured together. The separately named benchmark-only HTTP ingestor is disabled by default and is not the global HTTP listener.

Event outputs

Output Configuration role Required configuration
HTTP JSON Generic HTTP emitter Endpoint; optional auth and request settings depend on the destination.
Splunk HEC Security analytics output HEC endpoint and token; optional ACK polling waits for Splunk acknowledgement before marking a batch delivered.
Elasticsearch Search and analytics output Endpoint and index, plus destination authentication as configured.
Microsoft Sentinel DCR Azure security output Logs ingestion endpoint, immutable rule ID, stream name, and either client-secret credentials or Managed Identity.
Azure Blob Storage Object-storage archive Account URL and container; optional object prefix. Uses the Azure SDK default credential chain.
AWS S3 Object-storage output Bucket; region and object prefix are optional.
AWS SQS Queue output Queue URL; region is optional.
GCP GCS Object-storage output Bucket; region is optional.
GCP Pub/Sub Messaging output Project ID and topic configuration.
Kafka Messaging output Broker list and topic; JSON events are produced synchronously with record-level outcomes.
RabbitMQ Messaging output AMQP/AMQPS URL and either queue or exchange; optional routing key, publisher confirms determine acceptance.
SQL Relational database output Driver, DSN, table and parameterized insert query.
JSONL file Local archive or verification output File path.

Connector type names in YAML are implementation identifiers such as http_json, splunk_hec, elasticsearch, sentinel_dcr, azure_blob, aws_s3, aws_sqs, gcp_gcs, gcp_pubsub, kafka, rabbitmq, sql and file. RabbitMQ uses the Go module rabbitmq/amqp091-go for AMQP 0-9-1 connections. Use the YAML specification as the source of truth for the complete configuration shape.

Azure Event Hubs Kafka input uses TLS 1.2+ with server certificate verification and SASL/OAUTHBEARER. On AKS, set ingestors[].eventhub.auth_mode: workload_identity; the credential exchanges the projected token from the Data Plane ServiceAccount for an Event Hubs token scoped to the configured namespace (https://<namespace>.servicebus.windows.net/.default). Set eventhub.resource_logs: true when the Event Hub carries Azure Monitor resource logs: Logstrm expands a records envelope, flattens the nested properties fields for routing, and retains the original properties map. The Manager input form does not yet expose eventhub.auth_mode or eventhub.resource_logs; configure them in the Data Plane YAML/Helm values. Prefer a dedicated user-assigned identity with only Azure Event Hubs Data Receiver at the specific namespace/Event Hub scope. Its federated credential must trust the Data Plane ServiceAccount subject. Configure managed_identity_client_id to select it explicitly when a separate identity is used alongside the Sentinel DCR identity. SAS keys and SASL/PLAIN are not supported by this connector. The input commits Event Hubs offsets according to the current handler contract; see the delivery caveat in Delivery and security contract before treating the connector as lossless.

Local multi-source fan-out validation

The automated integration test TestMultiSourceFanoutIsolatesFailedDestination exercises two actual local inputs (HTTP JSON and Syslog TCP), source-specific routes, and two outputs (a private JSONL archive and a test HTTP destination). It verifies that an unavailable destination is retried and recorded in the DLQ while the archive still receives both events, and that the pipeline allowlist removes sensitive/raw fields before either output. The test uses local in-process test servers and requires no cloud account or external credentials. It validates the runtime routing/failure-isolation path, not a vendor connector, production network policy, sustained throughput, or HA/failover behavior.

Azure pilot plan

The selected next vendor pilot is Azure Event Hubs Kafka API → Sentinel DCR + Splunk HEC. It requires the durable outbox because Emit-only delivery is asynchronous. See Durable Event Hubs delivery outbox. This is implementation and setup guidance, not evidence that a tenant endpoint has been tested.

Manager UI coverage

The Manager Visual Pipeline Builder provides forms for these input/listener types:

  • Azure Blob with Event Grid notifications;
  • Azure Event Hubs through the Kafka-compatible input;
  • Kafka-compatible brokers;
  • RabbitMQ queues;
  • AWS S3 and AWS SQS;
  • GCP GCS and GCP Pub/Sub;
  • global HTTP listener;
  • global Syslog TCP/UDP listener, including multiline settings.

Multiple cloud/Kafka inputs may be configured together. HTTP and Syslog are optional global listeners, serialized separately from ingestors. TLS Syslog is not offered by this form because it is not started by the current runtime server. The DCR form supports client-secret authentication, Managed Identity and AKS Workload Identity; in the identity modes, the identity must be attached to (or federated for) the Data Plane workload that emits events, not to the Manager. An optional managed identity client ID selects a user-assigned identity (Managed Identity mode) or a specific federated identity client ID (AKS Workload Identity mode, defaulting to the webhook-injected AZURE_CLIENT_ID); omit it for the workload's system-assigned identity.

Destination form coverage is distinct from Data Plane output capability. The current Manager exposes Azure Log Analytics (DCR), Azure Blob, Splunk HEC, AWS S3/SQS, GCP GCS/Pub/Sub, Kafka, RabbitMQ, SQL and JSONL file outputs. Other runtime emitter types (for example generic HTTP JSON or Elasticsearch, if enabled in the Data Plane build) must be authored through YAML. Manager validation and version persistence do not prove delivery to an external destination.

See Manager UI and connector configuration for field-level validation and the local smoke test.

Delivery and security contract

All emitters participate in the Data Plane batching, retry, metrics and dead-letter behavior. A destination outage can therefore produce retry counters and DLQ entries rather than silently dropping events. Validate replay and drain behavior with a reachable test destination before production rollout.

For Azure Event Hubs, configure the input with eventhub.auth_mode: workload_identity (AKS) or managed_identity (Azure-hosted compute). The azure_eventhub connector uses TLS 1.2+ with certificate verification and SASL/OAUTHBEARER; it does not accept an unauthenticated Kafka connection or SAS keys. Prefer a dedicated identity with only Azure Event Hubs Data Receiver on the source namespace/Event Hub. On AKS, federate that identity to the Data Plane ServiceAccount; managed_identity_client_id can select a receiver-only user-assigned identity separate from the Sentinel DCR identity.

With the opt-in durable outbox, Kafka/Event Hubs groups persist source receipts, transformed deliveries and capacity counters before broker offsets are committed; successful broker commits are followed by receipt compaction/checkpoint. HTTP ingestion can use the same store with atomic request acceptance before HTTP202, without fabricated offsets or retained broker receipts. These inputs can coexist. Synchronous destinations include Azure Blob, Sentinel DCR, Splunk HEC and whole-request-ACK HTTP JSON. Other source/destination types remain rejected in this mode. See durable connector support for the capability matrix, HTTP limits/retry semantics, transport limitations and Helm opt-in. Failed outputs retry independently; exhausted failures remain blocked for explicit replay. Full storage eventually backpressures all sources. Without the outbox, the legacy asynchronous emitter path does not provide crash-safe at-least-once delivery.

Failed batch outcomes are recorded in bounded atomic groups of at most 125 delivery rows. If that database write fails, the dispatcher retains the reserved IDs and retries the outcome write before releasing them; it does not resend the network batch solely because outcome persistence failed. Retrying an already-persisted outcome preserves its retry deadline and counters. A process crash before outcome persistence can still cause duplicate delivery.

The outbox is single-active-writer and at-least-once, not exactly-once. The source record is durably accepted before its offset is committed; destination delivery continues independently. A crash after remote acceptance but before recording success can duplicate delivery; stable event IDs and destination-side deduplication are recommended. A permanent destination failure becomes blocked and is not a successful delivery until an operator fixes the destination and replays it. Event Hubs retention must cover the maximum expected pause/outage. If outbox capacity is reached, partitions pause without committing new offsets. Do not claim protection against source-retention expiry, PVC loss, or multiple active Data Plane replicas.

Malformed/oversized JSON records and records whose planned destination fan-out exceeds the entire outbox cap are left uncommitted and retried with a delay. A valid record that matches no pipeline/route is an intentional filter: a payload-free source receipt is durably written before its offset is committed. The Event Hubs consumer client is recreated from the last committed offsets after an uncommitted record; this pauses that single Event Hubs consumer, not only one partition. There is no automatic poison-record skip or source quarantine in this pilot path; correct malformed/oversized input or carry out a separately reviewed Event Hubs offset remediation before continuing.

Blocked deliveries are retained without payload/error-body exposure in the Manager API. Viewer can inspect tenant outbox status; Operator/Admin can queue explicit blocked delivery IDs for replay. The Manager audits replay intent and outcome. A pending/blocked destination’s emitter configuration is fingerprint-pinned: reloads that change or remove that emitter are rejected until its deliveries are drained or replayed. To recover a changed/retired endpoint, restore the pinned configuration, drain/replay its deliveries, then perform the reviewed destination migration.

Splunk HEC indexer acknowledgement polling is optional and disabled by default. When enabled, configure ack.endpoint, a positive ack.timeout and ack.poll_interval; the HEC event POST response must contain a numeric ackId. The emitter creates a UUID X-Splunk-Request-Channel and reuses it for event and ACK requests, POSTs {"acks":[<ackId>]} to the ACK endpoint, and waits until the response maps that ID to true. Missing, malformed, rejected or timed-out acknowledgements follow the emitter's existing retry and DLQ path.

In durable outbox mode, due Splunk HEC deliveries for the same emitter configuration are sent together, split according to batch.max_events and batch.max_bytes. Each HTTP chunk receives one HEC ackId; its outbox rows are marked delivered only after a positive ACK. If a later chunk fails or its outcome is uncertain, already-acknowledged chunks in that outbox group can be resent, so consumers must tolerate duplicates. This optimization applies to the durable outbox path, not the legacy volatile emitter buffer.

Durable Azure Blob deliveries wait for the object upload call to succeed before the outbox row is marked delivered. The Blob object name is generated per upload attempt, so an upload that succeeds but has an uncertain client outcome can be repeated as another object. Preserve a stable event ID and deduplicate when reconciling archived records; the contract is at-least-once, not exactly-once.

Splunk HEC event and ACK endpoints must use HTTPS; configuration rejects plaintext HTTP and URLs with embedded credentials. The emitter uses the Go system trust store and does not expose an insecure certificate-verification bypass. Store auth.token as an environment reference backed by a Secret, never as a literal in a Manager version.

The generic type: kafka connector currently has no TLS/SASL configuration and must not be used across an untrusted network. The Azure Event Hubs Kafka-compatible connector is separately configured with TLS and Entra OAUTHBEARER; do not infer that generic Kafka inherits that authentication. The Manager's Event Hubs form does not yet expose eventhub.auth_mode or eventhub.resource_logs; set those in the Data Plane startup config through Helm and restart the Data Plane when changing them.

Keep credentials outside committed YAML and out of generated previews:

  • inject secrets through the deployment environment, Kubernetes Secrets or an external secret manager;
  • use managed identity where the cloud provider and deployment model support it;
  • do not place production tokens, DSNs or cloud credentials in values.yaml, screenshots or test fixtures;
  • restrict connector egress and listener ingress with the deployment network policy;
  • monitor slimstream_emitter_errors_total, slimstream_emitter_retries_total, DLQ metrics and destination-specific health signals.

Connector support is transport-specific. A successful Manager save confirms schema validation, serialization and persistence; it does not replace an end-to-end test against the actual destination and credentials.