Azure Blob integration¶
Logstrm supports two separate Azure Blob roles: an Event Grid-triggered ingestor that downloads diagnostic files, and an emitter that archives routed events. They have different configuration and delivery guarantees.
ingestors:
- name: azure-blob-budget
type: azure_blob
enabled: true
storage:
account_url: "https://<storage-account>.blob.core.windows.net/"
container: "frontdoor-diagnostics"
credential_mode: "managed_identity"
webhook_listen: "0.0.0.0:8090"
webhook_path: "/webhooks/azure-event-grid"
webhook_token: "${ENV:LOGSTRM_BLOB_WEBHOOK_TOKEN}"
max_concurrent_jobs: 4
job_queue_path: "/var/lib/logstrm/blob-jobs/azure-blob-budget.sqlite"
job_queue_max_jobs: 1000
The webhook receives Event Grid metadata, not arbitrary log batches. The event must identify a newly created Blob, and the runtime identity must be able to read that blob.
Azure Blob emitter (output)¶
The azure_blob emitter writes each JSON event batch as one Blob object. It uses the Azure SDK default credential chain; in AKS, configure the Data Plane Workload Identity and grant it the Storage Blob Data Contributor role on the target container.
emitters:
- name: frontdoor-archive
type: azure_blob
cloud:
account_url: "https://<storage-account>.blob.core.windows.net/"
container: "frontdoor-archive"
object_prefix: "access-logs/"
batch:
max_events: 100
max_bytes: 1048576
flush_interval: "5s"
retry:
max_attempts: 5
initial_interval: "1s"
max_interval: "30s"
With the durable Event Hubs outbox enabled, Blob delivery is synchronous: the outbox row is marked delivered only after the Azure upload call succeeds. An upload error is returned to the outbox for retry. Delivery is still at least once, not exactly once. If Azure completes an upload but the client loses the response, a retry can create another object. Preserve a stable event identifier such as EventID_s or tracking_reference and deduplicate downstream when exact counts matter.
Without the durable outbox, the emitter uses its bounded in-memory batch buffer; a process crash can lose events that have not yet reached Blob Storage. Do not treat that mode as durable archival.
Webhook authentication and limits¶
Set LOGSTRM_BLOB_WEBHOOK_TOKEN to a random secret in the Data Plane environment. Configure the Event Grid subscription with a static Authorization delivery property containing Bearer <secret> and mark it as secret. See Microsoft's delivery properties reference. The Manager stores only an environment reference such as ${ENV:LOGSTRM_BLOB_WEBHOOK_TOKEN}.
Terminate HTTPS at a private ingress/reverse proxy in front of the webhook HTTP listener. Restrict its network access to the intended sender; the native webhook listener does not terminate TLS. Authentication is required for both notifications and subscription validation. Validation must be a single event containing a non-empty validation code; a mixed validation/notification batch is rejected. Keep the subscription and runtime token consistent when rotating it.
The runtime accepts only the configured storage account and container, rejecting foreign hosts/containers and URLs containing credentials, query strings or fragments. Its Azure identity should have read access only to that container. It bounds concurrent webhook requests to 16, each request to 10 MiB / 100 notifications, and download workers to at most 8. Blob names are limited to 4 KiB. A full queue or storage failure returns 503; the entire notification batch is committed or rolled back together.
Downloads decode object, array-of-objects or JSONL records incrementally, with a 256 MiB total blob limit and a 1 MiB per-record limit. When an emitter queue is temporarily full, the worker retains the current record and waits for capacity before reading the next record. It retries admission only to that destination; already accepted fan-out branches are not repeated. A full destination can therefore pause its Blob worker, and sustained backpressure can fill the job queue and cause new webhooks to return 503.
Durable jobs and restart recovery¶
An HTTP 200 notification response includes acceptance: "durable_job" and delivered: false. The referenced jobs have committed to a local SQLite database using WAL and synchronous=FULL before that acknowledgement. The database stores Blob references and processing state, not downloaded records or credentials. Workers resume pending jobs and reset interrupted processing claims on startup. A failed download, parse or routing admission keeps its job for retry with exponential backoff from one second to five minutes; there is no automatic expiry or deletion after a fixed number of attempts.
storage.job_queue_path is required for every enabled Blob input. Use a dedicated absolute file path per source, on a persistent disk/volume writable by the Data Plane. Omitted or zero job_queue_max_jobs means 1,000 active jobs; the configurable range is 1–100,000. Pending and currently processing jobs both consume capacity. The job count bounds metadata, not filesystem bytes; monitor disk space. Only one process may own a database: startup retries transient SQLite BUSY/LOCKED errors for up to five seconds, then fails if another owner still holds it. A queue is bound to the input name, account and container; do not rename or repurpose it while work remains.
Repeated notifications with the same Event Grid ID/source/version share a pending job. Completed notification receipts are retained for up to 24 hours and at most the queue's configured capacity. Receipts can be evicted earlier at high volume. Notifications without an ID or version are treated independently. This bounds notification deduplication; it does not deduplicate individual telemetry records or provide exactly-once processing.
Interrupted and failed files replay from the beginning. Late parse/size failures can occur after earlier records were routed; retries can repeat those records and previously accepted fan-out branches. Fix malformed/oversized blobs or routing failures promptly; they otherwise remain queued and continue retrying. Keep source blobs available and immutable for the recovery period: Logstrm downloads the current object at that path, and job storage is not a snapshot of an earlier Blob version.
A job completes when the whole file has successfully passed to the routing handler, including configured policy drops. This is not confirmed destination delivery. Asynchronous emitter queues remain volatile; a crash after file completion can lose records that were admitted to those queues but not delivered. This job queue is separate from the Event Hubs durable outbox and does not extend that outbox contract to Blob sources.
Storage migration and operations¶
- Before upgrading an existing in-memory Blob installation, pause new Event Grid deliveries and drain its existing jobs. In-memory work from the old process cannot be recovered by the new database.
- Add a unique
job_queue_pathto each enabled Blob input and provision persistent storage. Existing configurations without that path now fail validation. In Kubernetes, enabledataplane.blobJobs.persistence.enabled, useDeploymentwith one replica, and put database files under/var/lib/logstrm/blob-jobs/. The chart usesRecreateupdates to avoid overlapping owners; see Helm deployment. - In a VM/container, give the runtime user a dedicated writable directory on a persistent disk. Never rely on a container's disposable filesystem or
/tmpfor production recovery. SQLite WAL requires storage with reliable filesystem locking; avoid shared network filesystems. - Monitor
slimstream_blob_jobs{source,state}(pendingandprocessing),slimstream_blob_job_capacity{source}, andslimstream_blob_job_failures_total{source,reason}(processing,storage,capacity). Alert on queue saturation, sustained processing failures, and low disk space. Job completion metrics describe routing handoff, not SIEM delivery. - Stop the Data Plane before taking a filesystem backup. After clean shutdown the database is checkpointed; after an unclean shutdown retain its WAL and copy the complete queue directory, not just the main database. Protect source references and backups with your storage encryption/access controls. Do not delete the database to clear errors: that abandons accepted jobs. There is no job purge/replay UI in this implementation.
- Validate actual persistent-volume retention, storage failure, backup/restore and Blob availability in your environment. Local process-kill tests and Helm rendering do not prove node-failure or Azure delivery guarantees.
For low-latency streams, use the Event Hubs/Kafka-compatible path. For deterministic local load testing of routing and transformations, use the explicitly disabled-by-default benchmark HTTP ingestor instead.