Pipelines

A pipeline defines which events are relevant and how they are normalized before routing. Logstrm compiles pipeline matches during configuration load and applies transformations in a deterministic order.

pipelines:
  - name: azure_frontdoor
    match: 'category == "FrontDoorAccessLog" OR category == "FrontDoorWebApplicationFirewallLog"'
    transform:
      - action: KEEP
        fields: [timestamp, clientIP, requestUri, httpStatusCode, action, category]
      - action: RENAME
        mapping:
          clientIP: source_ip
          httpStatusCode: http_status_code
          requestUri: request_uri

Transformation actions

Action	Purpose	Guidance
KEEP	Allowlist fields	Prefer it for controlled SIEM schemas.
DROP	Remove fields	Use it for raw payloads, debug data and internal metadata.
RENAME	Normalize names	Keep destination schemas stable across sources.
MASK	Redact values	Use RE2-safe patterns and scope rules to fields.
ENRICH	Add local context	Keep lookups bounded and measure cache behavior.

Keep and drop operations should be designed together: allowlisting is usually safer for external destinations, while dropping is useful for preserving a broad internal schema.

Design checklist

• Give each pipeline a descriptive stable name.
• Match on source-specific fields before generic fallbacks.
• Remove raw messages after extracting required fields.
• Mask before an event leaves the Data Plane.
• Test representative events, including malformed and unusually large input.