Architecture

Logstrm separates the fast path that handles telemetry from the management systems that change its behavior.

Data Plane

The Data Plane is a Go process responsible for the event lifecycle:

source -> ingestor -> router -> pipeline transform -> emitter -> destination
                                  |                     |
                                  +-> enrichment        +-> retry / DLQ

Ingestors and global listeners

Individual ingestors are configured as entries in ingestors and can be combined in one Data Plane configuration. Supported types include Azure Blob/Event Grid, Azure Event Hubs through the Kafka-compatible consumer, Kafka-compatible brokers, AWS S3/SQS and GCP GCS/Pub/Sub. Provider credentials and network access are supplied by the deployment/runtime environment as appropriate.

HTTP and Syslog are global listeners configured separately under global.http and global.syslog; they are not individual ingestors. HTTP accepts configured listen, timeout and request-size settings. Syslog supports TCP and/or UDP plus message-size and multiline settings. Do not infer TLS Syslog availability from configuration structs: the current runtime server does not start a TLS Syslog listener. A separate benchmark-only HTTP ingestor can be enabled explicitly for reproducible local tests; it is disabled by default and is distinct from the global HTTP listener.

Router and pipelines

The router evaluates compiled pipeline and route expressions, applies the configured transformations, and sends the resulting event to one or more emitters. Pipeline order and route conditions are configuration, not application code.

Emitters

Emitters batch records, compress where supported, retry transient errors and expose metrics. Sentinel DCR and Elasticsearch are HTTP emitters; JSONL is useful for archive and local verification. The DLQ persists exhausted delivery failures so downstream outages do not require retaining an unbounded in-memory backlog.

Control Plane

The optional Logstrm Manager stores configuration versions and rollout results in SQLite. It can push a configuration to registered Data Plane nodes, track status and support rollback. The Data Plane remains independently runnable: a Manager outage does not stop event processing on already-configured nodes.

operator -> Manager API/UI -> versioned config -> Data Plane reload endpoint
                                      ^                    |
                                      |                    v
                                  SQLite history       metrics / health

Operational boundaries

Concern	Data Plane	Control Plane
Event processing	Yes	No
Pipeline and emitter execution	Yes	No
Configuration authoring	Reloaded config	Yes
Version history and rollout state	No	Yes
Local health and metrics	Yes	Yes
Durable manager state	DLQ/archive as configured	SQLite PVC

Reliability principles

• Bound request bodies, queues, retries and DLQ retention.
• Prefer backpressure or explicit rejection to unbounded memory growth.
• Keep credentials outside committed YAML and inject them through deployment secrets.
• Treat configuration as production code: validate, review, roll out and retain the previous version for rollback.