Skip to content

Pipelines

A pipeline defines which events are relevant and how they are normalized before routing. Logstrm compiles pipeline matches during configuration load and applies transformations in a deterministic order.

pipelines:
  - name: azure_frontdoor
    match: 'category == "FrontDoorAccessLog" OR category == "FrontDoorWebApplicationFirewallLog"'
    transform:
      - action: KEEP
        fields: [time, resourceId, category, ResponseStatusCode, BackendStatusCode, RequestUri, TrackingReference, Action, RuleName]
      - action: RENAME
        mapping:
          time: TimeGenerated
          ResponseStatusCode: response_status_code
          BackendStatusCode: backend_status_code
          RequestUri: request_uri
          TrackingReference: tracking_reference

For Azure Monitor resource logs arriving through Event Hubs, set ingestors[].eventhub.resource_logs: true. Logstrm expands an optional records envelope and lifts fields from properties while retaining the original map. This example routes by the client-facing ResponseStatusCode; retain BackendStatusCode separately because an origin response and the status AFD returns to a client can differ.

Transformation actions

Action Purpose Guidance
KEEP Allowlist fields Prefer it for controlled SIEM schemas.
DROP Remove fields Use it for raw payloads, debug data and internal metadata.
RENAME Normalize names Keep destination schemas stable across sources.
MASK Redact values Use RE2-safe patterns and scope rules to fields.
ENRICH Add local context Keep lookups bounded and measure cache behavior.

Keep and drop operations should be designed together: allowlisting is usually safer for external destinations, while dropping is useful for preserving a broad internal schema.

Design checklist

  • Give each pipeline a descriptive stable name.
  • Match on source-specific fields before generic fallbacks.
  • Remove raw messages after extracting required fields.
  • Mask before an event leaves the Data Plane.
  • Test representative events, including malformed and unusually large input.