Private Preview

This feature is in private preview and is not publicly available.

Monitoring SPIRL Reflector

Overview

Monitoring SPIRL Reflector provides valuable insights into the performance and health of your Defakto deployment. By enabling the Prometheus metrics endpoint, you can track key metrics such as API response times, request volumes, and system resource utilization. This visibility helps you identify potential bottlenecks, monitor system health, and make data-driven decisions about scaling and optimization. The metrics are particularly useful for understanding the behavior of gRPC APIs and ensuring your deployment is operating efficiently.

Prerequisites

1. SPIRL Reflector enabled in SPIRL System

Enabling SPIRL Reflector Prometheus Endpoint

Add the following to your SPIRL System values.yaml file to enable the metrics endpoint in the SPIRL Reflector.

telemetry:
  enabled: true

If you want to include latency metrics for gRPC APIs then include this.

telemetry:
  enabled: true
  collectors:
    grpc:
      emitLatencyMetrics: true

By default this enables the endpoint at port :9090. If you would like it to be enabled at a different location then you can specify it by using the below configuration.

Important: Note that the telemetry configuration in the Helm chart applies to both the Defakto Agent and the SPIRL Reflector. In the future we may separate these configurations but at present they apply to both components.

telemetry:
  enabled: true
  metricsAPI:
    port: 9090

KEK Rotation Metrics

A Reflector that stores its Key Encrypting Key (KEK) in a Kubernetes Secret exports a metric for each step of the rotation lifecycle, under the spirl_reflector_ prefix:

• spirl_reflector_kek_rotation_total - Attempts to add a new KEK generation, by source and outcome
• spirl_reflector_kek_last_rotation_timestamp_seconds - Unix time of the last KEK rotation that succeeded
• spirl_reflector_kek_rotation_failure_streak_seconds - Length of the current run of KEK rotation failures. Reads 0 when healthy
• spirl_reflector_kek_reload_total - Reloads of the KEK into memory, by source and outcome
• spirl_reflector_kek_watch_errors_total - Kubernetes watch failures on the KEK Secret, by reason
• spirl_reflector_kek_rewrap_total - DEK rewrap passes, by outcome
• spirl_reflector_kek_rewrap_objects_total - Stored objects rewrapped under the current KEK
• spirl_reflector_kek_rewrap_duration_seconds - Duration of DEK rewrap passes, by outcome
• spirl_reflector_kek_verify_total - Scans checking no stored data uses a superseded KEK generation, by outcome
• spirl_reflector_kek_retire_total - Attempts to delete unreferenced KEK generations, by outcome
• spirl_reflector_kek_generations - KEK generations currently held
• spirl_reflector_kek_last_reconcile_timestamp_seconds - Unix time of the last KEK reconciliation that completed

The outcome label identifies what happened at each step. Key values:

outcome	Meaning
blocked	The step could not proceed but did not fail. Verification reports this while stored data still references a superseded generation
conflict	Kubernetes optimistic concurrency let another pod create the generation first. Benign when several pods rotate at once
error	The step failed
ok	Successful

The source label appears on two metrics and carries different values on each. On spirl_reflector_kek_rotation_total it is schedule for a rotation the interval drove, manual for one someone triggered by annotating the KEK Secret, or initialization for the first key written into an empty Secret. On spirl_reflector_kek_reload_total it is watch or self_heal. A self_heal reload happens when a pod encounters a generation it does not hold and re-reads the Secret directly instead of waiting for the Kubernetes watch. A rising self_heal rate means the watch is missing updates.

The reason label on spirl_reflector_kek_watch_errors_total is either connection (the Kubernetes watch on the Secret dropped and is reconnecting) or secret_deleted (the KEK Secret was deleted).

The Reflector rotates its own KEK, independently of any Trust Domain Server. A blocked outcome on spirl_reflector_kek_verify_total means the verification scan found a record still wrapped under a superseded KEK generation, which happens when the pod that wrote it had not yet picked up the new one. The Reflector's SVID cache writes continuously, so this is routine rather than alarming. Blocking that persists across passes prevents retirement.

For the lifecycle these metrics track, and how to configure a schedule, see Reflector KEK Rotation.

Dashboard

A Grafana dashboard template for the Reflector is available in the Defakto dashboards GitHub repository:

• SPIRL Reflector Dashboard: spirl-reflector.json

The Reflector dashboard requires Agent Helm chart version v0.22.2 or later. To import it, download the JSON payload and, in Grafana, go to Dashboards > New > Import, then paste the JSON content.

Resource Sizing

Use the following formula as a starting point for Reflector memory sizing:

Memory (MiB) = 50 + (0.5 × Agents) + (0.004 × DistinctSVIDs)

• 50 MiB — base memory plus offline-events buffer overhead.
• 0.5 MiB per agent — gRPC connection overhead.
• 0.004 MiB per SVID — ~4 KiB per distinct SVID.

Example: 50 agents + 100 SVIDs = 75.4 MiB.

The formula was calibrated against small-to-medium deployments. For clusters with several hundred agents or more, treat it as a lower bound and validate against observed memory usage; if you see numbers significantly higher than the formula predicts, contact Defakto support.

During a node scale-out or autoscaling event, capture peak memory for the Reflector and compare against steady state. Size against the peak, not the average, if the burst-to-steady ratio is large.