Skip to content

YAML specification

Logstrm configuration is declarative and should be version controlled. The example below shows the current configuration shape. Adapt it to your enabled inputs, pipelines, routes and emitters before deployment. Replace placeholders and inject credentials through the environment or deployment secrets. The SLIMSTREAM_* environment variable prefix is retained by existing runtime interfaces; it does not indicate a different product name.

Secrets

Never commit tenant IDs, client secrets, tokens or private endpoints. The Data Plane expands environment variables before parsing, so ${AZURE_CLIENT_SECRET} or ${RABBITMQ_URL} can be supplied by the runtime. AMQP URLs may contain credentials; provide them through the deployment environment or secret manager.

Complete example

# Optional source ingestion. Disabled sources are ignored at startup.
ingestors:
  - name: azure-eventhub-premium
    type: azure_eventhub
    enabled: false
    brokers: ["myeventhub.servicebus.windows.net:9093"]
    topic: "frontdoor-logs"
    consumer_group: "logstrm-prod"
    eventhub:
      auth_mode: workload_identity
      resource_logs: true
      managed_identity_client_id: "${ENV:AZURE_EVENTHUB_CLIENT_ID}"

  # Buffered Blob/Event Grid input for pre-normalized JSON/JSONL. Raw Front Door resource logs use Event Hubs above.
  - name: azure-blob-budget
    type: azure_blob
    enabled: true
    storage:
      account_url: "https://mystorageaccount.blob.core.windows.net/"
      container: "frontdoor-diagnostics"
      credential_mode: "managed_identity"
      webhook_listen: "0.0.0.0:8090"
      webhook_path: "/webhooks/azure-event-grid"
      webhook_token: "${ENV:LOGSTRM_BLOB_WEBHOOK_TOKEN}"
      max_concurrent_jobs: 4
      job_queue_path: "/var/lib/logstrm/blob-jobs/azure-blob-budget.sqlite"
      job_queue_max_jobs: 1000

  # RabbitMQ input uses manual acknowledgements; failed handling requeues the delivery.
  - name: rabbitmq-source
    type: rabbitmq
    enabled: false
    rabbitmq:
      url: "${RABBITMQ_URL}"
      queue: "security-events"

# `global.performance` is retained for configuration compatibility but is not
# consumed by the current Data Plane runtime. Do not use it as a throughput
# control. Configure supported `emitters[].batch` and durable-outbox settings
# explicitly; see the delivery contract for their different code paths.

# Pipelines select and normalize source events.
pipelines:
  - name: azure_frontdoor
    match: 'category == "FrontDoorAccessLog" OR category == "FrontDoorWebApplicationFirewallLog"'
    transform:
      - action: KEEP
        fields:
          - time
          - resourceId
          - category
          - TrackingReference
          - ClientIp
          - RequestUri
          - HttpMethod
          - ResponseStatusCode
          - BackendStatusCode
          - UserAgent
          - Action
          - Policy
          - RuleName
      - action: DROP
        fields: [sourcePort, backendPort, rawMessage, debugInfo]
      - action: RENAME
        mapping:
          time: TimeGenerated
          ResponseStatusCode: response_status_code
          BackendStatusCode: backend_status_code
          ClientIp: client_ip
          RequestUri: request_uri
          TrackingReference: tracking_reference

# Routes fan out normalized events to named emitters.
routes:
  - name: frontdoor-security-events
    pipeline: azure_frontdoor
    condition: 'response_status_code < 200 OR response_status_code >= 400 OR category == "FrontDoorWebApplicationFirewallLog"'
    emitters: [sentinel-dcr]
  - name: frontdoor-access-archive
    pipeline: azure_frontdoor
    condition: 'response_status_code >= 200 AND response_status_code < 400'
    emitters: [archive-blob]

# Destinations. See the emitter-specific pages for production guidance.
emitters:
  - name: sentinel-dcr
    type: sentinel_dcr
    batch:
      max_bytes: 1048576
      flush_interval: "5s"
      compression: gzip
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"
    # Optional in-memory safeguards; rate_limit applies per downstream batch request.
    circuit_breaker:
      enabled: true
      failure_threshold: 5
      success_threshold: 2
      timeout: "30s"
      half_open_max_requests: 3
    rate_limit:
      enabled: true
      requests_per_second: 10
      burst: 20
    dcr:
      logs_ingestion_endpoint: "https://example.ingest.monitor.azure.com"
      rule_id: "00000000-0000-0000-0000-000000000000"
      stream_name: "Custom-Logstrm_CL"
      # auth_mode accepts managed_identity, workload_identity or client_secret.
      auth_mode: managed_identity
      # Optional. In managed_identity mode: omit for the system-assigned
      # identity, or set a user-assigned identity client ID. In
      # workload_identity mode: selects a specific federated identity client ID
      # (defaults to the webhook-injected AZURE_CLIENT_ID).
      managed_identity_client_id: "${AZURE_MANAGED_IDENTITY_CLIENT_ID}"
      # Optional (workload_identity only); federated token file path, defaults
      # to AZURE_FEDERATED_TOKEN_FILE from the AKS workload identity webhook.
      # federated_token_file: "/var/run/secrets/azure/tokens/azure-identity-token"
      scope: "https://monitor.azure.com/.default"

  - name: archive-blob
    type: azure_blob
    cloud:
      account_url: "https://<storage-account>.blob.core.windows.net/"
      container: "frontdoor-archive"
      object_prefix: "access-logs/"
    batch:
      max_events: 100
      max_bytes: 1048576
      flush_interval: "5s"
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"

  - name: security-kafka
    type: kafka
    brokers:
      - "kafka-1.example:9092"
      - "kafka-2.example:9092"
    topic: "security-logs"
    batch:
      max_events: 500
      flush_interval: "5s"
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"

  - name: rabbitmq-output
    type: rabbitmq
    rabbitmq:
      url: "${RABBITMQ_URL}"
      # Configure either queue or exchange; routing_key is optional.
      queue: "security-events"
      routing_key: "security.alert"
    batch:
      max_events: 500
      flush_interval: "5s"
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"

# Optional Splunk HEC indexer acknowledgement polling. Disabled by default.
# When enabled, POST responses must include a numeric ackId. The emitter adds
# X-Splunk-Request-Channel to event and ACK requests, then POSTs
# {"acks":[<ackId>]} to the ACK endpoint and waits for a true response value.
#   - name: splunk-hec-ack
#     type: splunk_hec
#     endpoint: "https://splunk.example/services/collector/event"
#     auth:
#       token: "${ENV:SPLUNK_HEC_TOKEN}"
#     ack:
#       enabled: true
#       endpoint: "https://splunk.example/services/collector/ack"
#       timeout: "30s"
#       poll_interval: "1s"

# Enable Prometheus metrics and optional local pprof.
observability:
  metrics_enabled: true
  metrics_path: "/metrics"
  enable_profiling: true

# Persist exhausted emitter failures in a bounded directory.
# Active writes use dlq_*.jsonl.active. Replay reads only dlq_*.jsonl.sealed.
# Malformed lines are copied to quarantine_*.jsonl (mode 0600) and are not replayed.
dead_letter:
  enabled: true
  directory: "./data/dlq"
  max_file_size: "64MiB"
  max_files: 16
  max_retries: 5

# API listener, health and authenticated reload endpoint.
# The validation endpoint is derived from reload_path by replacing /reload with /validate.
api:
  enabled: true
  listen: ":8080"
  auth_token: "${SLIMSTREAM_API_AUTH_TOKEN}"
  reload_path: "/api/v1/config/reload"
  health_path: "/api/v1/health"

Emitter circuit breaker and rate limiting

circuit_breaker and rate_limit are optional and disabled unless explicitly enabled. The circuit breaker opens after consecutive failed downstream batch attempts, waits for timeout, then admits a bounded number of half-open probes. A successful probe threshold closes it; a failed probe reopens it. The token bucket rate is measured in downstream batch requests per second, with burst controlling the initially available capacity. Waiting is cancellable during shutdown.

Circuit-open rejection stops retries for that batch immediately and sends its events through the existing DLQ path. These controls are process-local and recreated on configuration reload; they do not require a sidecar or shared service.

Configuration workflow

  1. Validate YAML and expression syntax before rollout. The read-only PUT /api/v1/config/validate endpoint performs this check without changing the active router, emitters, version or hash and returns the candidate SHA-256 on success.
  2. Test with representative events and inspect archive output.
  3. Confirm credentials are injected rather than committed.
  4. Deploy through the Manager, controlled reload endpoint, or SIGHUP.
  5. Monitor metrics, emitter errors and DLQ size after rollout.

Read-only validation endpoint

The Data Plane exposes PUT /api/v1/config/validate by default. It accepts the same YAML payload as the authenticated reload endpoint and uses the same bearer token. A successful response is shaped as:

{"valid":true,"sha256":"..."}

Invalid YAML, expressions or configuration sections return 400; authentication failures return 401. Validation does not create emitters, update the router or change the active configuration. The endpoint is intended for Manager rollout preflight and lightweight operational checks.

Production deployments must enable global.mtls and use HTTPS for Manager-to-Data-Plane validation and reload. The HTTP example below is for an isolated local development listener only; do not use it across an untrusted network.

curl -sS -X PUT http://127.0.0.1:8080/api/v1/config/validate \
  -H "Authorization: Bearer ${SLIMSTREAM_API_AUTH_TOKEN}" \
  -H 'Content-Type: application/yaml' \
  --data-binary @config.yaml

Reload with SIGHUP

When the Data Plane receives SIGHUP, it rereads the file supplied through the -config flag and applies it through the same controller as the authenticated HTTP reload endpoint. The new YAML and expressions are validated before the active configuration is replaced. Read, validation, or reload-boundary errors are logged and leave the previous configuration active; the active configuration version and SHA-256 remain unchanged after a rejected reload.

The controller allows these sections to be reloaded:

  • pipelines
  • routes
  • emitters
  • enrichment

The controller rejects any reload that changes these restart-only sections, rather than silently ignoring the change:

  • global (HTTP/syslog listeners, timeouts and transport settings)
  • ingestors
  • observability (metrics and profiling server settings)
  • dead_letter (writer/reader lifecycle and replay settings)
  • api (listener, paths and authentication)

On reload, omitted restart-only sections are carried forward from the active configuration. Within global, omitted subsections are also preserved. This lets Manager versions update reloadable routing and emitters without needing to serialize deployment-owned settings such as TLS certificate paths. Any explicitly supplied restart-only value that differs from the active value is rejected. A fresh process start still uses the defaults for fields omitted from its startup configuration.

Rejected changes require a process restart and do not create or close emitters, update the router, or alter the active version/hash. The reload lock also serializes DLQ replay with emitter replacement and shutdown, so replay always resolves the current emitter and cannot run against an emitter while it is being closed. File-watch reload is not enabled.

The top-level sections are intentionally small: sources (ingestors), processing (pipelines), fan-out (routes), destinations (emitters), enrichment (enrichment), safety (dead_letter) and operations (observability, api).

Configuration compatibility and evolution

The Data Plane YAML format currently has no top-level schema_version field. Do not add the Manager's schema_version metadata to Data Plane YAML: Manager schema version 1 describes its API/version-record contract and is excluded from the generated Data Plane configuration. See Control Plane and Manager.

Compatibility is determined by the fields understood by the particular Data Plane release and by the release notes for that version. YAML parsing currently ignores unknown mapping keys; this is not a compatibility guarantee and can hide misspelled keys. Validate candidate YAML with the same Data Plane release that will run it, and inspect the returned validation result before storing or rolling it out. Do not assume validation rejects every unknown key.

For a release upgrade, keep a known-good configuration artifact and its checksum, validate the candidate against the target binary, review release-specific configuration changes, and canary the rollout. Static listener, ingestor, API, observability and DLQ settings require process restart; only pipelines, routes, emitters and enrichment are reloadable at runtime. A rejected reload leaves the active configuration and its version/hash unchanged. Manager configuration versions and rollback are described in the Control Plane guide.

Administrative API and queue limits

An enabled api requires a non-empty auth_token; set it with an environment reference. Disabled APIs expose health/readiness/metrics and the UI shell, but no reload, validation, replay or Live Tail operations. The fallback API listen address is 127.0.0.1:8080 when no API/global HTTP listener address is set. Use private HTTPS listeners for production.

Live Tail requires api.live_tail_enabled: true; raw fields are omitted unless api.live_tail_include_raw: true. See Live Tail for authenticated connection steps.

Async emitters support batch.max_buffered_events and batch.max_buffered_bytes in addition to batch flush limits. Zero means the bounded default, not unlimited buffering. Queue admission errors propagate to retry-capable source handlers. See the delivery contract for defaults and partial-acceptance behavior. Invalid Grok patterns are rejected during configuration loading; compiled Grok processors and emitter fingerprints are reused until a configuration update.