Observability

Sukko ships with an optional bundled monitoring stack (Prometheus, Grafana, Loki), but every observability signal is designed to integrate with your existing infrastructure.

What Sukko Exposes

Signal	Format	Endpoint / Output
Metrics	Prometheus	/metrics on every service
Logs	Structured JSON	stdout (standard K8s log collection)
Traces	OpenTelemetry (OTLP)	Configurable exporter endpoint
Dashboards	Grafana JSON	Bundled in Helm chart, exportable

Using the Bundled Stack

Sukko's Helm chart includes a monitoring profile with Prometheus, Grafana, and Loki pre-configured. Enabled by default:

helm install sukko sukko-dev/sukko --set monitoring.enabled=true

For local development with Docker Compose, observability is opt-in via profile:

sukko init  # Select "yes" for observability when prompted
sukko up

Using Your Own Stack

If you already run Prometheus, Grafana, Datadog, or another monitoring platform, disable the bundled stack and point Sukko at your infrastructure.

Step 1: Disable Bundled Monitoring

helm install sukko sukko-dev/sukko --set monitoring.enabled=false

Step 2: Scrape Metrics

Every Sukko service exposes Prometheus metrics at /metrics. Add them as scrape targets in your existing Prometheus:

# prometheus.yml
scrape_configs:
  - job_name: sukko-gateway
    static_configs:
      - targets: ['sukko-gateway:3000']
  - job_name: sukko-core
    static_configs:
      - targets: ['sukko-server:3005']
  - job_name: sukko-provisioning
    static_configs:
      - targets: ['sukko-provisioning:8080']

Step 3: Collect Logs

All services log structured JSON to stdout. Your existing log collector (Fluentd, Fluent Bit, Datadog Agent, Vector, etc.) picks them up automatically through standard Kubernetes log collection — no Loki dependency required.

Step 4: Import Dashboards

Sukko's Grafana dashboards are JSON files bundled in the Helm chart. To use them in your own Grafana instance, export them from:

deployments/helm/sukko/charts/monitoring/dashboards/

Import the JSON files into your Grafana via the UI or API.

Step 5: Enable Tracing (Optional)

Sukko supports OpenTelemetry tracing. Point it at any OTLP-compatible collector (Jaeger, Tempo, Datadog, Honeycomb, etc.):

helm install sukko sukko-dev/sukko \
  --set monitoring.enabled=false \
  --set env.OTEL_TRACING_ENABLED=true \
  --set env.OTEL_EXPORTER_ENDPOINT=your-collector:4317

Step 5: Enable Profiling (Optional)

Sukko supports continuous profiling via Pyroscope:

helm install sukko sukko-dev/sukko \
  --set env.PYROSCOPE_ENABLED=true \
  --set env.PYROSCOPE_ADDR=http://your-pyroscope:4040

Broadcast Delivery Signals

The core server fans every ingested message out through the Valkey broadcast bus. If the bus is unreachable, the Kafka consumer deliberately stops advancing rather than committing past messages it could not deliver — delivery is at-least-once, so an outage costs latency and consumer lag, never silent message loss.

That trade-off is worth understanding before it happens: while the bus is down, the consumer blocks on the undelivered message and all tenants sharing that consumer wait behind it. Messages are not dropped; they are delayed until the bus returns, then delivered (possibly more than once — each carries a stable message id for de-duplication).

Three counters make the condition visible:

Metric	Meaning
ws_broadcast_publish_failures_total	Broadcast publishes that failed — non-zero means the bus is unreachable or rejecting
ws_consumer_broadcast_retries_total	Retry attempts on held messages; climbing means the consumer is stalled on an undelivered message
ws_consumer_broadcast_blocked_seconds_total	Cumulative time the consumer spent blocked — the direct measure of ingestion backpressure

Recommended alerts for your own stack:

# Bus unreachable — delivery is stalled, not losing data
- alert: SukkoBroadcastPublishFailing
  expr: rate(ws_broadcast_publish_failures_total[5m]) > 0
  for: 2m
  labels: { severity: critical }

# Sustained ingestion backpressure
- alert: SukkoConsumerBlocked
  expr: rate(ws_consumer_broadcast_blocked_seconds_total[5m]) > 0.1
  for: 5m
  labels: { severity: warning }

Pair these with Kafka consumer lag: rising lag with blocked-seconds climbing is a bus outage; rising lag without it is a throughput problem.

Subscription health

The core server holds one Valkey subscription per tenant, and it reconciles what it should be subscribed to against what it has confirmed — so a lost subscription repairs itself rather than leaving a tenant silently undelivered.

Metric	Meaning
ws_broadcast_subscriptions_desired	Subscriptions this pod should hold
ws_broadcast_subscriptions_established	Subscriptions confirmed on its current connection

established below desired while publishes succeed is the signal to watch. It means messages are being published that this pod's subscribers cannot receive. It is normal for a fraction of a second after a tenant's first subscribe or last unsubscribe, and during bus reconnection; sustained divergence is not.

- alert: SukkoSubscriptionsNotConverged
  expr: ws_broadcast_subscriptions_established < ws_broadcast_subscriptions_desired
  for: 2m
  labels: { severity: warning }

GET /health on the core server reports the same condition under checks.broadcast, with publish_healthy, subscriptions_converged, and the desired/established pair. Non-convergence is reported as degraded with HTTP 200 — deliberately, so it never fails the liveness probe. A pod whose subscription is still converging continues serving its existing connections, and restarting pods during a bus outage would turn a recoverable blip into a fleet-wide one.

Summary

Scenario	Helm Config
Use bundled stack	monitoring.enabled=true (default)
Use your own stack	monitoring.enabled=false + scrape /metrics + collect stdout logs
Add tracing	OTEL_TRACING_ENABLED=true + OTEL_EXPORTER_ENDPOINT=...
Add profiling	PYROSCOPE_ENABLED=true + PYROSCOPE_ADDR=...

Tracing Details PRO

Tracing is instrumented on cold paths only — operations that happen infrequently (connection setup, auth validation, Kafka rebalance). Hot paths (per-message delivery, broadcast fan-out) are not traced to protect latency; they're covered by Prometheus histograms instead.

Instrumented paths:

• Gateway: WebSocket upgrade, auth validation, auth refresh, publish authorization
• Core: Kafka consumer setup/rebalance, shard creation
• Provisioning: API request handling, database queries, gRPC stream events

Exporters: OTLP/gRPC (default), OTLP/HTTP, stdout (development)

If the tracing backend is unreachable, spans are dropped silently — tracing never blocks the message pipeline.

Alerting PRO

The bundled Helm chart includes PrometheusRule definitions with default alerts:

Alert	Condition	Severity
Connection capacity > 80%	5 min sustained	Warning
Connection capacity > 95%	2 min sustained	Critical
Error rate > 1%	5 min sustained	Warning
Pod restarts > 2	10 min window	Warning
Kafka consumer lag > 1000	5 min sustained	Warning
Memory > 80% of limit	10 min sustained	Warning
License expiry within 7 days	—	Warning
Service down	2 min	Critical

AlertManager supports Slack, PagerDuty, OpsGenie, email, and generic webhooks. Configure via Helm values — no chart modification needed.

Operators can add custom alert rules via Helm values alongside the defaults.

Next Steps

• Architecture — How services connect and expose metrics
• Configuration Reference — All observability-related env vars
• Testing — Validate your deployment with built-in test suites