Control Plane and Manager¶
Logstrm Manager is the optional Control Plane for authoring, versioning and rolling out Data Plane configuration. The Data Plane can run independently; a Manager outage does not stop an already configured Data Plane node.
Responsibilities¶
The Manager provides:
- a browser-based Visual Pipeline Builder;
- configuration version history stored in SQLite;
- YAML generation compatible with the Data Plane
config_v4.yamlartifact shape; - Config v1 metadata validation and bounded version diffs;
- Data Plane node registration;
- authenticated rollouts with bounded timeouts and response sizes;
- rollback by creating a new version linked to an earlier version;
- audit logging for mutating requests.
The Manager does not process telemetry itself. It sends a versioned configuration to enabled Data Plane nodes through the rollout API. Connector coverage and the distinction between UI-configurable and YAML-configured integrations are documented in the Connector reference.
Local start¶
From the repository root, start the Manager:
cd slimstream-manager
SLIMSTREAM_ALLOW_UNAUTHENTICATED_LOOPBACK=true \
go run ./cmd/manager -db /tmp/logstrm-manager-test.db -listen 127.0.0.1:8091
Open the UI at http://127.0.0.1:8091/. The default port is 8090; use another port when the Data Plane or another local service already occupies it.
The command above explicitly enables development access without OIDC. This mode requires an IP loopback listener and loopback clients; forwarded headers do not grant access. Never expose it through a reverse proxy or tunnel. Production startup requires a valid SLIMSTREAM_OIDC_ISSUER and SLIMSTREAM_OIDC_CLIENT_ID. The default listener is 127.0.0.1:8090.
The database path controls the local history. Use a persistent path for a real environment and a temporary path for disposable tests.
API endpoints¶
| Endpoint | Method | Purpose |
|---|---|---|
/api/v1/health |
GET |
Unauthenticated health check |
/api/v1/versions |
GET |
List configuration versions, including schema_version metadata |
/api/v1/versions |
POST |
Validate, serialize and store a configuration version |
/api/v1/versions/{id}/diff |
GET |
Return a bounded diff against the parent version, or an explicit ?against={id} version |
Data Plane /api/v1/config/validate |
PUT |
Validate candidate YAML without changing active Data Plane state |
/api/v1/versions/{id}/rollback |
POST |
Create a new version from an earlier version |
/api/v1/nodes |
GET |
List registered Data Plane nodes |
/api/v1/nodes |
POST |
Register or update a node |
/api/v1/nodes/{id} |
DELETE |
Remove a node |
/api/v1/rollouts |
POST |
Roll out a version to enabled nodes |
/api/v1/nodes/{id}/outbox |
GET |
Read bounded durable-outbox counts and blocked delivery IDs, without event payloads |
/api/v1/nodes/{id}/outbox/replays |
POST |
Queue explicit blocked durable-outbox delivery IDs for retry |
Manager requires OIDC by default. All protected API endpoints require Authorization: Bearer <token>. Setting SLIMSTREAM_OIDC_ISSUER automatically enables required authentication; invalid or unavailable OIDC configuration fails closed. Admin may mutate state; Viewer may read state only.
The outbox endpoints exist only for Data Plane nodes with durable_outbox.enabled=true. GET /api/v1/nodes/{id}/outbox is Viewer/Operator/Admin-readable and returns aggregate counts plus up to 100 blocked delivery descriptors, never event payloads or raw downstream errors. POST /api/v1/nodes/{id}/outbox/replays is Operator/Admin-only, accepts at most 50 explicit blocked delivery IDs, and records audit intent/completion. Manager forwards both operations to the node using its verified HTTPS/mTLS client. The Data Plane rejects configuration changes to an emitter while that emitter has pending or blocked deliveries.
Config v1 and version diffs¶
Manager configuration requests use schema version 1. Omitting schema_version selects the current version. Unsupported values are rejected with 400 Bad Request before persistence. The response and SQLite history include schema_version.
schema_version is Manager metadata only. It is deliberately excluded from the generated Data Plane config_v4.yaml artifact so existing Data Plane parsers and rollout contracts remain compatible.
A version diff response has this shape:
{
"version_id": 2,
"against": 1,
"diff": "--- version-1\\n+++ version-2\\n ..."
}
Without against, the endpoint compares the requested version with its parent_id. A root version returns an empty diff. The formatter is line-oriented, deterministic and bounded; oversized input or output is rejected rather than allowing unbounded Manager CPU or memory use.
curl -s http://127.0.0.1:8091/api/v1/versions/2/diff
curl -s 'http://127.0.0.1:8091/api/v1/versions/2/diff?against=1'
Node registration and rollout¶
Before enabling a production rollout, use the Data Plane validation endpoint against the exact generated YAML. It parses and compiles the candidate using the same path as reload, but does not create emitters, update routing, or change the active version/hash. The endpoint uses the node's configured bearer token and returns {"valid":true,"sha256":"..."} on success.
Manager rollout performs this preflight automatically for every enabled node at /api/v1/config/validate. It verifies that the returned SHA-256 matches the stored version hash before sending the mutating /api/v1/config/reload request. A validation failure is recorded for that node and no reload is sent; other nodes continue independently. This makes rollout validation a per-node safety gate rather than an operator-only check.
Manager versions may omit Data Plane restart-only sections that the Manager does not own. During reload, the Data Plane preserves those omitted active settings while applying the version's pipelines, routes, emitters and enrichment. Explicit changes to restart-only settings are still rejected. In particular, Data Plane TLS and listener settings remain deployment-owned and must be changed through the deployment configuration and a process restart, not a Manager rollout.
Register a node with its Data Plane base URL and bearer token. Keep credentials out of shell history in production; the example uses test values only:
curl -s -X POST http://127.0.0.1:8091/api/v1/nodes \
-H 'content-type: application/json' \
-d '{"name":"dataplane-test","url":"https://logstrm:8090","auth_token":"test-token","tls_verify":true,"enabled":false}'
Node registration accepts only HTTPS endpoints with certificate verification enabled; HTTP, disabled verification, and credentials embedded in the URL are rejected. The Manager rollout client uses the system trust store plus any CA bundle configured through SLIMSTREAM_DATAPLANE_CA_FILE. SLIMSTREAM_DATAPLANE_CLIENT_CERT_FILE and SLIMSTREAM_DATAPLANE_CLIENT_KEY_FILE configure an optional Manager client certificate for mTLS. Ensure the server certificate SAN matches the registered hostname.
When the Data Plane has the Event Hubs durable outbox enabled, GET /api/v1/nodes/{id}/outbox is available to Viewer/Operator/Admin roles; it returns aggregate counts and at most 100 blocked delivery descriptors, never event payloads or raw downstream errors. POST /api/v1/nodes/{id}/outbox/replays is Operator/Admin-only, requires explicit blocked delivery IDs (maximum 50), and records audit intent/completion. The Manager forwards these requests to the Data Plane over its configured verified HTTPS/mTLS client. Replay only after resolving the destination fault; an emitter with pending/blocked deliveries cannot be silently retargeted through config rollout.
Set enabled to true only when the node is ready to receive configuration. Start a rollout with a saved version ID:
curl -s -X POST http://127.0.0.1:8091/api/v1/rollouts \
-H 'content-type: application/json' \
-d '{"version_id":1}'
A rollout targets enabled nodes only. Each node must pass read-only validation before it receives the mutating reload request. If the request is cancelled, nodes that have not been started are returned as cancelled and are not pushed. Results are persisted with the version and node status. A response header X-Logstrm-Persistence: failed means that persistence failed after the push; inspect the Manager log before treating the stored rollout history as complete.
OIDC and roles¶
Configure the verifier with:
SLIMSTREAM_OIDC_ISSUER;SLIMSTREAM_OIDC_CLIENT_ID;- optional
SLIMSTREAM_OIDC_ROLES_CLAIM(defaultroles); - optional
SLIMSTREAM_OIDC_GROUPS_CLAIM(defaultgroups).
The exact claim values Admin and Viewer are accepted. A user with both roles retains Admin permissions. Do not paste production tokens into screenshots, tickets or documentation.
Empty collections¶
A new Manager database has no versions and no nodes. Depending on the current JSON serialization path, an empty collection may be displayed as null rather than []; this is not a health failure. The recommended future API refinement is to normalize empty collections to [] for a more predictable client contract.
Operational guidance¶
- Back up the Manager SQLite database; it contains configuration history and rollout state.
- Treat generated YAML as production configuration: review it before rollout.
- Use environment variables or deployment secrets for credentials.
- Test with disabled nodes before enabling a rollout target.
- Keep the previous working version available for rollback.