Pipelines¶
A pipeline defines which events are relevant and how they are normalized before routing. Logstrm compiles pipeline matches during configuration load and applies transformations in a deterministic order.
pipelines:
- name: azure_frontdoor
match: 'category == "FrontDoorAccessLog" OR category == "FrontDoorWebApplicationFirewallLog"'
transform:
- action: KEEP
fields: [time, resourceId, category, ResponseStatusCode, BackendStatusCode, RequestUri, TrackingReference, Action, RuleName]
- action: RENAME
mapping:
time: TimeGenerated
ResponseStatusCode: response_status_code
BackendStatusCode: backend_status_code
RequestUri: request_uri
TrackingReference: tracking_reference
For Azure Monitor resource logs arriving through Event Hubs, set ingestors[].eventhub.resource_logs: true. Logstrm expands an optional records envelope and lifts fields from properties while retaining the original map. This example routes by the client-facing ResponseStatusCode; retain BackendStatusCode separately because an origin response and the status AFD returns to a client can differ.
Transformation actions¶
| Action | Purpose | Guidance |
|---|---|---|
KEEP |
Allowlist fields | Prefer it for controlled SIEM schemas. |
DROP |
Remove fields | Use it for raw payloads, debug data and internal metadata. |
RENAME |
Normalize names | Keep destination schemas stable across sources. |
MASK |
Redact values | Use RE2-safe patterns and scope rules to fields. |
ENRICH |
Add local context | Keep lookups bounded and measure cache behavior. |
Keep and drop operations should be designed together: allowlisting is usually safer for external destinations, while dropping is useful for preserving a broad internal schema.
Design checklist¶
- Give each pipeline a descriptive stable name.
- Match on source-specific fields before generic fallbacks.
- Remove raw messages after extracting required fields.
- Mask before an event leaves the Data Plane.
- Test representative events, including malformed and unusually large input.