Connectors¶
Logstrm separates ingestors (event inputs) from emitters (event outputs). A configuration can combine multiple inputs, pipelines and destinations; routes decide which processed events are sent to each emitter.
Event inputs¶
| Input | Configuration role | Notes |
|---|---|---|
| Azure Event Hubs via Kafka API | Individual cloud ingestor | Uses the Kafka-compatible consumer; configure brokers/topic and provider-supported authentication. |
| Kafka-compatible brokers | Individual ingestor | Configure brokers and topic for the source cluster. |
| Azure Blob / Event Grid | Individual cloud ingestor | BlobCreated notifications trigger blob download and record decoding. This is an input, not an output emitter. |
| AWS S3 | Individual cloud ingestor | The Manager UI requires a bucket; region and object prefix are optional. |
| AWS SQS | Individual cloud ingestor | The Manager UI requires a queue URL; region is optional. |
| GCP GCS | Individual cloud ingestor | The Manager UI requires a bucket; region is optional. |
| GCP Pub/Sub | Individual cloud ingestor | The Manager UI requires a project ID and subscription. |
| RabbitMQ | Individual message-queue ingestor | Requires an AMQP/AMQPS URL and queue; deliveries are acknowledged after successful event handling and requeued on handler failure. |
HTTP and Syslog are global Data Plane listeners, not individual entries in ingestors: HTTP accepts agent/application requests, while Syslog supports TCP and/or UDP. Their settings are written under global.http and global.syslog. These listeners are independent from cloud/Kafka ingestors, and multiple ingestors plus either or both listeners may be configured together. The separately named benchmark-only HTTP ingestor is disabled by default and is not the global HTTP listener.
Event outputs¶
| Output | Configuration role | Required configuration |
|---|---|---|
| HTTP JSON | Generic HTTP emitter | Endpoint; optional auth and request settings depend on the destination. |
| Splunk HEC | Security analytics output | HEC endpoint and token; optional ACK polling waits for Splunk acknowledgement before marking a batch delivered. |
| Elasticsearch | Search and analytics output | Endpoint and index, plus destination authentication as configured. |
| Microsoft Sentinel DCR | Azure security output | Logs ingestion endpoint, immutable rule ID, stream name, and either client-secret credentials or Managed Identity. |
| AWS S3 | Object-storage output | Bucket; region and object prefix are optional. |
| AWS SQS | Queue output | Queue URL; region is optional. |
| GCP GCS | Object-storage output | Bucket; region is optional. |
| GCP Pub/Sub | Messaging output | Project ID and topic configuration. |
| Kafka | Messaging output | Broker list and topic; JSON events are produced synchronously with record-level outcomes. |
| RabbitMQ | Messaging output | AMQP/AMQPS URL and either queue or exchange; optional routing key, publisher confirms determine acceptance. |
| SQL | Relational database output | Driver, DSN, table and parameterized insert query. |
| JSONL file | Local archive or verification output | File path. |
Connector type names in YAML are implementation identifiers such as http_json, splunk_hec, elasticsearch, sentinel_dcr, aws_s3, aws_sqs, gcp_gcs, gcp_pubsub, kafka, rabbitmq, sql and file. RabbitMQ uses the Go module rabbitmq/amqp091-go for AMQP 0-9-1 connections. Use the YAML specification as the source of truth for the complete configuration shape.
Manager UI coverage¶
The Manager Visual Pipeline Builder provides forms for these input/listener types:
- Azure Blob with Event Grid notifications;
- Azure Event Hubs through the Kafka-compatible input;
- Kafka-compatible brokers;
- RabbitMQ queues;
- AWS S3 and AWS SQS;
- GCP GCS and GCP Pub/Sub;
- global HTTP listener;
- global Syslog TCP/UDP listener, including multiline settings.
Multiple cloud/Kafka inputs may be configured together. HTTP and Syslog are optional global listeners, serialized separately from ingestors. TLS Syslog is not offered by this form because it is not started by the current runtime server. The DCR form supports client-secret authentication, Managed Identity and AKS Workload Identity; in the identity modes, the identity must be attached to (or federated for) the Data Plane workload that emits events, not to the Manager. An optional managed identity client ID selects a user-assigned identity (Managed Identity mode) or a specific federated identity client ID (AKS Workload Identity mode, defaulting to the webhook-injected AZURE_CLIENT_ID); omit it for the workload's system-assigned identity.
Destination form coverage is distinct from Data Plane output capability. The current Manager exposes Azure Log Analytics (DCR), Azure Blob, Splunk HEC, AWS S3/SQS, GCP GCS/Pub/Sub, Kafka, RabbitMQ, SQL and JSONL file outputs. Other runtime emitter types (for example generic HTTP JSON or Elasticsearch, if enabled in the Data Plane build) must be authored through YAML. Manager validation and version persistence do not prove delivery to an external destination.
See Manager UI and connector configuration for field-level validation and the local smoke test.
Delivery and security contract¶
All emitters participate in the Data Plane batching, retry, metrics and dead-letter behavior. A destination outage can therefore produce retry counters and DLQ entries rather than silently dropping events. Validate replay and drain behavior with a reachable test destination before production rollout.
Splunk HEC acknowledgement polling is optional and disabled by default. When enabled, configure ack.endpoint, a positive ack.timeout and ack.poll_interval; the HEC POST response must contain ackId, and the acknowledgement endpoint must report that ID as true. Missing, malformed or timed-out acknowledgements follow the emitter's existing retry and DLQ path.
Keep credentials outside committed YAML and out of generated previews:
- inject secrets through the deployment environment, Kubernetes Secrets or an external secret manager;
- use managed identity where the cloud provider and deployment model support it;
- do not place production tokens, DSNs or cloud credentials in
values.yaml, screenshots or test fixtures; - restrict connector egress and listener ingress with the deployment network policy;
- monitor
slimstream_emitter_errors_total,slimstream_emitter_retries_total, DLQ metrics and destination-specific health signals.
Connector support is transport-specific. A successful Manager save confirms schema validation, serialization and persistence; it does not replace an end-to-end test against the actual destination and credentials.