YAML specification

Logstrm configuration is declarative and should be version controlled. The example below shows the current configuration shape. Adapt it to your enabled inputs, pipelines, routes and emitters before deployment. Replace placeholders and inject credentials through the environment or deployment secrets. The SLIMSTREAM_* environment variable prefix is retained by existing runtime interfaces; it does not indicate a different product name.

Secrets

Never commit tenant IDs, client secrets, tokens or private endpoints. The Data Plane expands environment variables before parsing, so ${AZURE_CLIENT_SECRET} or ${RABBITMQ_URL} can be supplied by the runtime. AMQP URLs may contain credentials; provide them through the deployment environment or secret manager.

Complete example

# Optional source ingestion. Disabled sources are ignored at startup.
ingestors:
  - name: azure-eventhub-premium
    type: azure_eventhub
    enabled: false
    brokers: ["myeventhub.servicebus.windows.net:9093"]
    topic: "frontdoor-logs"
    consumer_group: "logstrm-prod"

  # Buffered Blob/Event Grid ingestion.
  - name: azure-blob-budget
    type: azure_blob
    enabled: true
    storage:
      account_url: "https://mystorageaccount.blob.core.windows.net/"
      container: "frontdoor-diagnostics"
      credential_mode: "managed_identity"
      webhook_listen: "0.0.0.0:8090"
      webhook_path: "/webhooks/azure-event-grid"
      max_concurrent_jobs: 4

  # RabbitMQ input uses manual acknowledgements; failed handling requeues the delivery.
  - name: rabbitmq-source
    type: rabbitmq
    enabled: false
    rabbitmq:
      url: "${RABBITMQ_URL}"
      queue: "security-events"

# Global worker and batching defaults.
global:
  performance:
    worker_pool_size: 8
    batch_size: 500
    flush_interval: "5s"

# Pipelines select and normalize source events.
pipelines:
  - name: azure_frontdoor
    match: 'category == "FrontDoorAccessLog" OR category == "FrontDoorWebApplicationFirewallLog"'
    transform:
      - action: KEEP
        fields:
          - timestamp
          - trackingReference
          - clientIP
          - requestUri
          - httpMethod
          - httpStatusCode
          - userAgent
          - action
          - policy
          - ruleName
          - category
          - host
      - action: DROP
        fields: [sourcePort, backendPort, rawMessage, debugInfo]
      - action: RENAME
        mapping:
          clientIP: source_ip
          httpStatusCode: http_status_code
          requestUri: request_uri
          trackingReference: tracking_reference

# Routes fan out normalized events to named emitters.
routes:
  - name: frontdoor-http-errors
    pipeline: azure_frontdoor
    condition: 'http_status_code >= 400'
    emitters: [sentinel-dcr]
  - name: frontdoor-waf-blocks
    pipeline: azure_frontdoor
    condition: 'action == "Block"'
    emitters: [sentinel-dcr]
  - name: frontdoor-archive
    pipeline: azure_frontdoor
    condition: 'true'
    emitters: [archive-jsonl]

# Destinations. See the emitter-specific pages for production guidance.
emitters:
  - name: sentinel-dcr
    type: sentinel_dcr
    batch:
      max_bytes: 1048576
      flush_interval: "5s"
      compression: gzip
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"
    # Optional in-memory safeguards; rate_limit applies per downstream batch request.
    circuit_breaker:
      enabled: true
      failure_threshold: 5
      success_threshold: 2
      timeout: "30s"
      half_open_max_requests: 3
    rate_limit:
      enabled: true
      requests_per_second: 10
      burst: 20
    dcr:
      logs_ingestion_endpoint: "https://example.ingest.monitor.azure.com"
      rule_id: "00000000-0000-0000-0000-000000000000"
      stream_name: "Custom-Logstrm_CL"
      # auth_mode accepts managed_identity, workload_identity or client_secret.
      auth_mode: managed_identity
      # Optional. In managed_identity mode: omit for the system-assigned
      # identity, or set a user-assigned identity client ID. In
      # workload_identity mode: selects a specific federated identity client ID
      # (defaults to the webhook-injected AZURE_CLIENT_ID).
      managed_identity_client_id: "${AZURE_MANAGED_IDENTITY_CLIENT_ID}"
      # Optional (workload_identity only); federated token file path, defaults
      # to AZURE_FEDERATED_TOKEN_FILE from the AKS workload identity webhook.
      # federated_token_file: "/var/run/secrets/azure/tokens/azure-identity-token"
      scope: "https://monitor.azure.com/.default"

  - name: archive-jsonl
    type: file
    path: "./data/archive/frontdoor.jsonl"
    format: jsonl

  - name: security-kafka
    type: kafka
    brokers:
      - "kafka-1.example:9092"
      - "kafka-2.example:9092"
    topic: "security-logs"
    batch:
      max_events: 500
      flush_interval: "5s"
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"

  - name: rabbitmq-output
    type: rabbitmq
    rabbitmq:
      url: "${RABBITMQ_URL}"
      # Configure either queue or exchange; routing_key is optional.
      queue: "security-events"
      routing_key: "security.alert"
    batch:
      max_events: 500
      flush_interval: "5s"
    retry:
      max_attempts: 5
      initial_interval: "1s"
      max_interval: "30s"

# Optional Splunk HEC acknowledgement polling. Disabled by default.
# When enabled, POST responses must include ackId and the endpoint must expose
# Splunk's GET acknowledgement response: {"acks":{"<ackId>":true}}.
#   - name: splunk-hec-ack
#     type: splunk_hec
#     endpoint: "https://splunk.example/services/collector/event"
#     auth:
#       token: "${ENV:SPLUNK_HEC_TOKEN}"
#     ack:
#       enabled: true
#       endpoint: "https://splunk.example/services/collector/ack"
#       timeout: "30s"
#       poll_interval: "1s"

# Enable Prometheus metrics and optional local pprof.
observability:
  metrics_enabled: true
  metrics_path: "/metrics"
  enable_profiling: true

# Persist exhausted emitter failures in a bounded directory.
# Active writes use dlq_*.jsonl.active. Replay reads only dlq_*.jsonl.sealed.
# Malformed lines are copied to quarantine_*.jsonl (mode 0600) and are not replayed.
dead_letter:
  enabled: true
  directory: "./data/dlq"
  max_file_size: "64MiB"
  max_files: 16
  max_retries: 5

# API listener, health and authenticated reload endpoint.
# The validation endpoint is derived from reload_path by replacing /reload with /validate.
api:
  enabled: true
  listen: ":8080"
  auth_token: "${SLIMSTREAM_API_AUTH_TOKEN}"
  reload_path: "/api/v1/config/reload"
  health_path: "/api/v1/health"

Emitter circuit breaker and rate limiting

circuit_breaker and rate_limit are optional and disabled unless explicitly enabled. The circuit breaker opens after consecutive failed downstream batch attempts, waits for timeout, then admits a bounded number of half-open probes. A successful probe threshold closes it; a failed probe reopens it. The token bucket rate is measured in downstream batch requests per second, with burst controlling the initially available capacity. Waiting is cancellable during shutdown.

Circuit-open rejection stops retries for that batch immediately and sends its events through the existing DLQ path. These controls are process-local and recreated on configuration reload; they do not require a sidecar or shared service.

Configuration workflow

1. Validate YAML and expression syntax before rollout. The read-only PUT /api/v1/config/validate endpoint performs this check without changing the active router, emitters, version or hash and returns the candidate SHA-256 on success.
2. Test with representative events and inspect archive output.
3. Confirm credentials are injected rather than committed.
4. Deploy through the Manager, controlled reload endpoint, or SIGHUP.
5. Monitor metrics, emitter errors and DLQ size after rollout.

Read-only validation endpoint

The Data Plane exposes PUT /api/v1/config/validate by default. It accepts the same YAML payload as the authenticated reload endpoint and uses the same bearer token. A successful response is shaped as:

{"valid":true,"sha256":"..."}

Invalid YAML, expressions or configuration sections return 400; authentication failures return 401. Validation does not create emitters, update the router or change the active configuration. The endpoint is intended for Manager rollout preflight and lightweight operational checks.

curl -sS -X PUT http://127.0.0.1:8080/api/v1/config/validate \
  -H "Authorization: Bearer ${SLIMSTREAM_API_AUTH_TOKEN}" \
  -H 'Content-Type: application/yaml' \
  --data-binary @config.yaml

Reload with SIGHUP

When the Data Plane receives SIGHUP, it rereads the file supplied through the -config flag and applies it through the same controller as the authenticated HTTP reload endpoint. The new YAML and expressions are validated before the active configuration is replaced. Read, validation, or reload-boundary errors are logged and leave the previous configuration active; the active configuration version and SHA-256 remain unchanged after a rejected reload.

The controller allows these sections to be reloaded:

• pipelines
• routes
• emitters
• enrichment

The controller rejects any reload that changes these restart-only sections, rather than silently ignoring the change:

• global (HTTP/syslog listeners, timeouts and transport settings)
• ingestors
• observability (metrics and profiling server settings)
• dead_letter (writer/reader lifecycle and replay settings)
• api (listener, paths and authentication)

Rejected changes require a process restart and do not create or close emitters, update the router, or alter the active version/hash. The reload lock also serializes DLQ replay with emitter replacement and shutdown, so replay always resolves the current emitter and cannot run against an emitter while it is being closed. File-watch reload is not enabled.

The top-level sections are intentionally small: sources (ingestors), processing (pipelines), fan-out (routes), destinations (emitters), enrichment (enrichment), safety (dead_letter) and operations (observability, api).

Configuration compatibility and evolution

The Data Plane YAML format currently has no top-level schema_version field. Do not add the Manager's schema_version metadata to Data Plane YAML: Manager schema version 1 describes its API/version-record contract and is excluded from the generated Data Plane configuration. See Control Plane and Manager.

Compatibility is determined by the fields understood by the particular Data Plane release and by the release notes for that version. YAML parsing currently ignores unknown mapping keys; this is not a compatibility guarantee and can hide misspelled keys. Validate candidate YAML with the same Data Plane release that will run it, and inspect the returned validation result before storing or rolling it out. Do not assume validation rejects every unknown key.

For a release upgrade, keep a known-good configuration artifact and its checksum, validate the candidate against the target binary, review release-specific configuration changes, and canary the rollout. Static listener, ingestor, API, observability and DLQ settings require process restart; only pipelines, routes, emitters and enrichment are reloadable at runtime. A rejected reload leaves the active configuration and its version/hash unchanged. Manager configuration versions and rollback are described in the Control Plane guide.