Skip to content

Kubernetes with Helm

The Logstrm Helm chart deploys the Data Plane and optional Manager. It supports Data Plane Deployment or node-local DaemonSet mode, health probes, and persistent Manager state. Obtain the chart, image references, version compatibility information, and registry access instructions from the Logstrm distribution channel provided to your organization. This guide does not assume a public chart repository or anonymously accessible images.

Prerequisites

  • Kubernetes 1.25+ and Helm 3.12+;
  • the Logstrm chart and supported Data Plane/Manager images for the same release;
  • registry credentials if the supplied images are private;
  • a dynamic StorageClass for Manager persistence, unless using a pre-created PVC;
  • an approved secret-management method for the Data Plane hot-reload token and emitter credentials.

Install

Set LOGSTRM_CHART to the chart reference or local chart package supplied with your distribution, and LOGSTRM_VERSION to its matching release version. Configure image repositories and tags to the exact values supplied for that release. The example leaves these as explicit inputs rather than presenting placeholder coordinates as downloadable artifacts.

The chart requires TLS on the Data Plane API and global HTTP ingestion; rendering fails if dataplane.apiTLS.enabled=false. Create these Secrets in the release namespace before installing:

  • logstrm-dataplane-tls: server certificate tls.crt, tls.key, and ca.crt containing the client-certificate issuer CA.
  • logstrm-manager-dataplane-client: Manager client certificate tls.crt, tls.key, and ca.crt containing the CA that issued the Data Plane server certificate.

The server certificate needs serverAuth usage and DNS SANs for manager.nodes[0].endpoint (default logstrm), the ServiceMonitor TLS name (default <release>.<namespace>.svc, configurable with observability.serviceMonitor.dataplaneTLS.serverName), and any external HTTPS ingestion hostname. The Manager validates the Data Plane server certificate and hostname, and presents its client certificate on control requests. The default verify_if_given mode validates a client cert when supplied; use require_and_verify_client_cert only if every HTTPS client, including ingestion senders, can present one. The chart uses TCP probes in that mode so kubelet probes do not bypass or fail the client-certificate requirement.

Provision certificate files through your approved certificate/secret manager. Do not put private keys in Helm values, source control, or shell arguments.

For a namespace named logstrm, create the Secrets from protected files issued by your CA:

kubectl create secret generic logstrm-dataplane-tls -n logstrm \
  --from-file=tls.crt=/secure/certs/logstrm-server.crt \
  --from-file=tls.key=/secure/certs/logstrm-server.key \
  --from-file=ca.crt=/secure/certs/client-issuer-ca.crt

kubectl create secret generic logstrm-manager-dataplane-client -n logstrm \
  --from-file=tls.crt=/secure/certs/logstrm-manager-client.crt \
  --from-file=tls.key=/secure/certs/logstrm-manager-client.key \
  --from-file=ca.crt=/secure/certs/server-issuer-ca.crt

The chart does not generate certificates. To rotate the Manager client certificate or its trusted server CA, update the Secret and restart the Manager. Restart the Data Plane after certificate or client-issuer CA changes to ensure the mounted material is reloaded.

The Data Plane control API Service logstrm on port 8090 is fixed to ClusterIP. HTTP and optional Syslog ingestion use a separate logstrm-ingest Service, also ClusterIP by default. For direct external HTTP ingestion, expose only the ingestion Service:

--set dataplane.ingestService.type=LoadBalancer

The chart rejects LoadBalancer exposure for the API Service and Manager Service. HTTP ingestion on the ingestion Service uses the Data Plane TLS certificate. Syslog TCP/UDP does not support TLS in-process and its Service ports default to 0 (not published). To use Syslog, configure the listener and matching dataplane.ingestService.syslogTCPPort / syslogUDPPort; keep those ports on a trusted private network or use a TLS relay.

The chart also requires hot-reload authentication. Create the logstrm-hot-reload Secret before installation, as shown below; chart rendering fails if neither a Secret reference nor a demo token is configured. If the referenced Secret or key is absent, both workloads fail to start rather than running with unauthenticated reload.

export LOGSTRM_CHART='<chart reference supplied with your distribution>'
export LOGSTRM_VERSION='<supported release version>'
export LOGSTRM_IMAGE_REPOSITORY='<Data Plane image repository>'
export LOGSTRM_MANAGER_IMAGE_REPOSITORY='<Manager image repository>'
export LOGSTRM_OIDC_ISSUER='https://id.example.com/realms/logstrm'
export LOGSTRM_OIDC_CLIENT_ID='logstrm-manager'

helm upgrade --install logstrm "$LOGSTRM_CHART" \
  --namespace logstrm \
  --create-namespace \
  --set images.logstrm.repository="$LOGSTRM_IMAGE_REPOSITORY" \
  --set images.manager.repository="$LOGSTRM_MANAGER_IMAGE_REPOSITORY" \
  --set images.logstrm.tag="$LOGSTRM_VERSION" \
  --set images.manager.tag="$LOGSTRM_VERSION" \
  --set manager.oidc.issuer="$LOGSTRM_OIDC_ISSUER" \
  --set manager.oidc.clientID="$LOGSTRM_OIDC_CLIENT_ID" \
  --set dataplane.hotReload.existingSecret=logstrm-hot-reload

The Manager OIDC issuer and client ID are mandatory and must match the provider and browser client you configured. The chart sets SLIMSTREAM_OIDC_REQUIRED=true and fails rendering without these values. TLS Secret names are configurable with dataplane.apiTLS.existingSecret and manager.dataplaneTLS.existingSecret; both TLS settings are enabled by default. Create logstrm-hot-reload through your secret manager or approved Kubernetes secret workflow before installation, using the key expected by the supplied chart version (hot-reload-token by default). Use a high-entropy token of at least 32 random bytes and ensure the secret value has no trailing newline. Do not put token values in shell history or commit them to a values file. Configure registry pull credentials as documented by your registry and chart distribution.

AKS Workload Identity

For Sentinel DCR emitters with auth_mode: workload_identity, enable the chart's AKS Workload Identity wiring so the webhook projects a federated token into the Data Plane pods:

helm upgrade --install logstrm "$LOGSTRM_CHART" \
  --namespace logstrm \
  --create-namespace \
  --set dataplane.hotReload.existingSecret=logstrm-hot-reload \
  --set manager.oidc.issuer="$LOGSTRM_OIDC_ISSUER" \
  --set manager.oidc.clientID="$LOGSTRM_OIDC_CLIENT_ID" \
  --set serviceAccount.workloadIdentity.enabled=true \
  --set serviceAccount.workloadIdentity.clientID='<federated-identity-client-id>'

With serviceAccount.workloadIdentity.enabled=true the chart adds the azure.workload.identity/use: "true" pod label only to the Data Plane and forces automountServiceAccountToken: true on its dedicated ServiceAccount, which the AKS webhook requires to project the service account token. The Manager uses a separate ServiceAccount with token automount disabled and does not receive this workload identity. Setting clientID adds the azure.workload.identity/client-id annotation to the Data Plane ServiceAccount; omit it when the webhook should use its default client ID mapping.

Cluster prerequisites: an AKS cluster with the OIDC issuer and the Workload Identity add-on enabled, an Entra managed identity with a federated identity credential bound to the Data Plane ServiceAccount (namespace logstrm, service account logstrm by default), and the Monitoring Metrics Publisher role for that identity scoped to the Data Collection Rule. The Data Plane resolves AZURE_CLIENT_ID, AZURE_TENANT_ID and AZURE_FEDERATED_TOKEN_FILE from the webhook and fails at startup when any of them cannot be resolved.

Durable Event Hubs delivery outbox (pilot)

The opt-in outbox supports Azure Event Hubs, ordinary Kafka and optional HTTP ingestion, with synchronous azure_blob, sentinel_dcr, splunk_hec and http_json destinations. It persists transformed per-destination deliveries in SQLite/WAL before broker commits or HTTP202. Inputs can coexist; destination dispatch remains independent within finite shared capacity. Use exactly one Data Plane Deployment replica, a dedicated persistent encrypted StorageClass, API bearer authentication and required verified client mTLS. Event Hubs still requires Workload Identity/Managed Identity. Other source/output types remain rejected. Existing releases keep the HTTP listener/Service disabled; dataplane.outbox.httpIngest.enabled=true, dataplane.config.httpListen and explicit networkPolicy.ingress rules opt in. See durable connector support for acknowledgment boundaries, limits, ordinary Kafka transport limitations and rollback requirements.

dataplane:
  mode: Deployment
  replicaCount: 1
  outbox:
    enabled: true
    path: /var/lib/logstrm/outbox/outbox.sqlite
    maxBytes: 1073741824
    maxAttempts: 5
    retryInterval: 5s
    maxRetryInterval: 5m
    persistence:
      enabled: true
      size: 20Gi
      storageClass: "<encrypted-storage-class>"
  config:
    httpListen: ""
    ingestors:
      - name: eventhub
        type: azure_eventhub
        enabled: true
        brokers: ["<namespace>.servicebus.windows.net:9093"]
        topic: "<event-hub-name>"
        consumer_group: "logstrm-<tenant>"
        eventhub:
          auth_mode: workload_identity
          managed_identity_client_id: "<eventhub-receiver-identity-client-id>"

Grant a dedicated receiver identity Azure Event Hubs Data Receiver on the narrowest source scope. The Data Plane Workload Identity federated credential must trust the Data Plane ServiceAccount subject; the Event Hubs client ID can select a receiver identity separate from the Sentinel DCR writer identity. For Azure Blob output, grant the identity selected by DefaultAzureCredential Storage Blob Data Contributor scoped to the archive container. Splunk HEC tokens must arrive from a namespace Secret through dataplane.extraEnvFrom; HEC event and ACK endpoints require HTTPS and a certificate trusted by the runtime system roots.

The outbox uses at-least-once delivery, not exactly-once. A crash after remote acceptance and before local success-state persistence may duplicate delivery. Azure Blob retries can create another object when an upload's outcome is uncertain; keep a stable event ID and deduplicate during archive reconciliation. Splunk HEC ACK mode can wait for a positive indexer ACK; Sentinel 2xx is only Logs Ingestion API acceptance, not query visibility. Capped exponential retries that exhaust are retained as blocked, with no automatic deletion. If the payload-byte cap is reached, the single Event Hubs consumer closes and rejoins from its last committed offsets, stopping consumption until capacity returns; Event Hubs retention must cover this outage window. A malformed/oversized record or no-match route likewise remains uncommitted and needs producer/config repair, not automatic skipping. maxBytes counts serialized payloads per destination delivery, not SQLite/WAL/page overhead, so keep it well below PVC capacity. Keep the Event Hubs consumer group stable with this outbox: committed-offset checkpoints suppress replay of already-committed offsets when the group is rewound. An intentional source replay requires a new consumer group and a separately planned outbox/retention lifecycle.

The outbox status/replay API is available only when the feature is enabled:

Manager endpoint Role Behavior
GET /api/v1/nodes/{id}/outbox Viewer or higher Aggregate counts and up to 100 blocked delivery descriptors; no payloads or raw downstream errors
POST /api/v1/nodes/{id}/outbox/replays Operator or Admin Queue up to 50 explicit blocked integer IDs; Manager records audit intent/completion

Restore the original fingerprinted emitter configuration before replay. Reloads that change or remove a destination with pending/blocked rows are rejected to prevent silent retargeting. Changing a destination requires draining the old deliveries first.

This design has one active writer; it is not an HA/shared SQLite outbox and must not be horizontally scaled. Back up and encrypt the PVC and its backups. Alert on slimstream_outbox_pending_deliveries, slimstream_outbox_blocked_deliveries, slimstream_outbox_payload_bytes, slimstream_outbox_backpressure_total and PVC free space. The pilot runbook also requires a recovery test from PVC restart and an explicit full-capacity test in a disposable tenant.

Separate Manager per tenant

The current Manager is not a multi-tenant control plane: its SQLite database holds its configuration versions, registered nodes and node credentials, and each rollout sends the selected version to every enabled node. Until tenant-scoped authorization and rollout groups exist, deploy a dedicated Helm release and namespace per tenant/trust domain. This gives each tenant a separate Manager database/PVC, Data Plane, ServiceAccounts, Secrets and OIDC client configuration. Do not register different tenants' nodes in the same Manager.

For tenants hosted in one AKS cluster, use one namespace per tenant and a unique release name. Set fullnameOverride=logstrm to keep the Manager's default same-namespace Data Plane endpoint https://logstrm:8090; Kubernetes service DNS is namespace-scoped, so the same service name remains isolated. Supply a distinct OIDC client ID/allowed users or groups, reload token, server/client TLS material and Data Plane Workload Identity for each tenant. Create GHCR pull credentials in each tenant namespace. Scope each workload identity to only that tenant's DCR/destinations. Apply namespace RBAC and restrictive NetworkPolicies; restrict ingress to that tenant's sources and Manager UI ingress.

Example namespace/release pattern (repeat with unique tenant identifiers and reviewed per-tenant values):

TENANT="acme"
NAMESPACE="logstrm-${TENANT}"
RELEASE="logstrm-${TENANT}"

kubectl create namespace "$NAMESPACE"
# Provision this namespace's GHCR, hot-reload and TLS Secrets using the approved secret manager.
helm upgrade --install "$RELEASE" "$LOGSTRM_CHART" \
  --namespace "$NAMESPACE" \
  --set fullnameOverride=logstrm \
  --set manager.oidc.issuer="$TENANT_OIDC_ISSUER" \
  --set manager.oidc.clientID="$TENANT_OIDC_CLIENT_ID" \
  --set manager.nodes[0].endpoint=https://logstrm:8090 \
  --set serviceAccount.workloadIdentity.enabled=true \
  --set serviceAccount.workloadIdentity.clientID="$TENANT_DP_IDENTITY_CLIENT_ID" \
  --set dataplane.hotReload.existingSecret=logstrm-hot-reload \
  -f "$TENANT_VALUES_FILE"

Do not place secrets or tenant-specific credentials in shared values files, command-line arguments or source control. Create separate protected values/configuration and namespace Secrets per tenant. The example intentionally does not expose the Manager Service; provision a tenant-specific TLS ingress or private access path separately. manager.replicaCount remains 1 because the Manager currently uses a local SQLite database/PVC rather than a shared HA database.

Separate namespaces on a shared cluster provide Kubernetes resource and network-policy boundaries, not a hard isolation guarantee against cluster administrators, compromised nodes or cluster-level components. Use a separate AKS cluster/subscription/network boundary when tenant threat models, regulatory requirements or operators demand stronger isolation. Region alone is not a tenant boundary: for one tenant spanning regions, a shared Manager is appropriate only if the same configuration may safely be rolled out to every registered node and the private control-plane paths are available.

The AKS Workload Identity path has been smoke-tested end-to-end with one schema-matched event confirmed in a custom Log Analytics table. This validates the basic federated-authentication and delivery path only; it is not a capacity, resilience or production-readiness result. See the Sentinel DCR validation notes.

Operational checks

kubectl rollout status deployment/logstrm-dataplane -n logstrm
kubectl get pods,svc,pvc -n logstrm

The Data Plane exposes /api/v1/ready for readiness and /api/v1/health for liveness. Readiness returns 503 during graceful drain while liveness remains healthy. The Manager uses /api/v1/health. Keep the Manager PVC backed up; it contains configuration history and rollout state.

Security baseline

Use the release's supported non-root images, read-only filesystems, dropped capabilities, image pull secrets for private registries, and network policies restricting access and egress. Do not expose profiling or benchmark-only listeners through a public Service. Inject emitter credentials using an approved secret-management mechanism, not chart values or ConfigMaps. serviceAccount.automountServiceAccountToken stays false unless explicitly enabled; the AKS Workload Identity wiring is the documented exception and forces it to true together with the azure.workload.identity/use pod label.

Upgrade note: chart 0.1.3 enables Data Plane TLS and separates the former combined Service into the private API Service (logstrm, ClusterIP) and ingestion Service (logstrm-ingest, ClusterIP). Before upgrading, create the Data Plane server and Manager client certificate Secrets, keep the hot-reload Secret and OIDC settings, update HTTP senders to use HTTPS and the ingestion Service, and move any external address to dataplane.ingestService.type=LoadBalancer. If using --reuse-values, explicitly set dataplane.service.type=ClusterIP; old LoadBalancer API Service values are rejected. Move old dataplane.service.syslogTCPPort / syslogUDPPort settings to dataplane.ingestService.* and enable only ports matching configured Syslog listeners. Existing Manager node records must use HTTPS URLs with tls_verify=true; the Manager now rejects insecure endpoints. The API Service cannot be exposed as a LoadBalancer. Syslog remains plaintext and must stay on trusted networks or use a TLS relay.

Chart 0.1.4 gives the Manager a dedicated ServiceAccount with Kubernetes token automount disabled. AKS Workload Identity is now attached only to the Data Plane ServiceAccount/workload; the Manager does not need the DCR ingestion identity. Existing Data Plane federated credentials remain bound to the logstrm ServiceAccount. Review this behavior when upgrading, and see Separate Manager per tenant before deploying multiple tenants.

Chart 0.1.5 adds the opt-in, SQLite-backed Event Hubs outbox. It requires explicit single-replica Deployment mode, an encrypted persistent PVC, an Event Hubs-only input configuration and NetworkPolicies with tenant-specific ingress/egress rules. It disables the HTTP ingestion Service in this mode. Do not enable it with --reuse-values until these requirements and the source/destination routes are reviewed.

Durable Blob jobs

Enabled azure_blob sources require persistent job storage. Configure one Data Plane Deployment replica and enable the Blob jobs PVC:

dataplane:
  mode: Deployment
  replicaCount: 1
  blobJobs:
    persistence:
      enabled: true
      size: 1Gi
      storageClass: "<approved persistent block storage class>"
      # existingClaim: "<dedicated pre-created claim>"
  config:
    ingestors:
      - name: azure-blob-budget
        type: azure_blob
        enabled: true
        storage:
          account_url: "https://<account>.blob.core.windows.net/"
          container: diagnostics
          webhook_listen: ":8092"
          webhook_path: /webhooks/event-grid
          webhook_token: "${ENV:LOGSTRM_BLOB_WEBHOOK_TOKEN}"
          job_queue_path: /var/lib/logstrm/blob-jobs/azure-blob-budget.sqlite
          job_queue_max_jobs: 1000

Inject the webhook secret through your approved Secret/environment mechanism (dataplane.extraEnvFrom where appropriate). Use a separate private Service/HTTPS ingress for the configured webhook listener; the chart's HTTP ingest Service does not automatically expose Blob webhook ports. Avoid sharing the API/HTTP listen port. Restrict webhook ingress and storage-account egress with the release's NetworkPolicy.

The chart mounts the PVC at /var/lib/logstrm/blob-jobs/, requires distinct database files underneath it, rejects DaemonSet/multiple-replica/ephemeral configurations and uses Recreate updates. A pre-created existingClaim suppresses creation of another Blob PVC. Storage must support SQLite locking and persistence; monitor capacity, back up and encrypt it according to your requirements. This topology provides one active owner, not HA or active/active coordination.

Webhook acknowledgement follows job commit. Interrupted files replay from the beginning, and successful file routing does not make asynchronous destinations durable. See Azure Blob recovery and migration.

Bounded durable dispatcher (restart required)

dataplane.outbox.dispatcher is passed through to Data Plane. Existing values remain valid. For example, allow four batches per emitter while preserving capacity for other destinations:

dataplane:
  outbox:
    dispatcher:
      max_active_batches: 8
      max_active_bytes: 16777216
      defaults:
        max_in_flight_batches: 4
        max_events_per_batch: 125
        max_serialized_bytes_per_batch: 1048576
        max_batch_wait: 1s
        delivery_timeout: 1m
      emitters:
        slow_hec:
          max_in_flight_batches: 1

Override keys must match configured emitter names. Omitted numeric settings inherit defaults; defaults are 2 global tasks, 16 MiB active serialized bytes, 1 batch per emitter, 125 events, 1 MiB per batch, 1 s wait, 1 m delivery timeout. Per-emitter tasks and active bytes cannot exceed half the respective global budget. A partial batch waits up to max_batch_wait from its earliest due time, subject to capacity, scheduler polling and SQLite availability. A full batch can run immediately. Poll interval remains an upper bound on discovering new work; this is not an end-to-end latency guarantee.

The byte budget covers the canonical JSON array, not RSS, gzip or transport buffers. Oversized events remain durably blocked with error class dispatcher_payload_too_large; retune, restart and explicitly replay them. Concurrency can reorder deliveries. At-least-once can produce duplicates after ambiguous delivery/commit; downstream deduplication may be needed.

Use one process per outbox on a local filesystem with working advisory locks. The new process rejects a second cooperating owner. Never run old/new binaries concurrently on the same database. On shutdown cancel and join the dispatcher before closing the database; tune termination grace for downstream cancellation. Shared disk exhaustion still backpressures all sources despite receiver isolation. Monitor slimstream_outbox_active_batches, slimstream_outbox_active_serialized_bytes, slimstream_outbox_reserved_deliveries and existing pending age/blocked metrics.

Durable source group commit

dataplane.outbox.groupCommit bounds source records admitted to each durable write group and is rendered into the Data Plane ConfigMap. The defaults are 25 source records, 4 MiB of Kafka record value bytes, a 4 MiB canonical serialized-delivery byte target (capped by dataplane.outbox.maxBytes) and a 5 ms maximum wait:

dataplane:
  outbox:
    groupCommit:
      max_records: 25
      max_source_bytes: 4194304
      max_serialized_bytes: 4194304
      max_wait: 5ms

Records are grouped within a Kafka/Event Hubs partition. Their source receipts, planned deliveries and outbox capacity counters are committed atomically using SQLite WAL with synchronous=FULL; the Kafka offset is committed only after that durable write succeeds. The outbox checkpoint/receipt cleanup happens after the broker commit. If a bounded write is split into more than one SQLite transaction and a later transaction fails, already committed receipts make broker redelivery idempotent. A single source record may exceed the group byte target by itself; it still must fit the durable outbox capacity. This does not make SQLite and Kafka a distributed transaction and does not provide exactly-once delivery.

After new delivery rows commit, the dispatcher is notified immediately rather than waiting for its normal poll interval; polling remains a recovery fallback. Group size and wait change transaction rate, source linger and the shape of pending-work peaks. Tune them with the local durable benchmark before any separately authorized live test.

Multi-source durable candidate migration

New multi-source candidates retain schema v2 and existing single-owner/PVC/mTLS requirements. Existing outbox values keep HTTP disabled. To use Kafka or durable HTTP/HTTP JSON, select a matching candidate image and follow the capability/migration guide. Historical Chart 0.1.5 notes above describe its original Event Hubs-only scope, not current candidate support.