Skip to main content

Kubernetes Key Encrypting Key (KEK) Rotation

Rotate the Key Encrypting Key (KEK) held in a Kubernetes Secret, on a schedule or on demand, without restarting workloads and without re-encrypting stored payloads.

Rotation covers only the default Kubernetes Secret backend. AWS KMS, Azure Key Vault, and Google Cloud KMS own the lifecycle of the keys they hold, so a KEK stored in one of those services rotates through the provider instead. See AWS KMS, Azure Key Vault, or GCP Cloud KMS.

What rotation applies to​

The Trust Domain Server holds its KEK in a Secret named <release>-td-server-enc-keys, in the deployment namespace. Rotation is configured in the spirl-server chart.

Separate Trust Domain Server deployments rotate independently, because each deployment has its own namespace, Secret, and encrypted store.

What a rotation does​

Storage uses envelope encryption. Every signing key write generates a fresh AES-256-GCM data-encryption key (DEK), encrypts the signing key with that DEK, wraps the DEK with the KEK, and stores the encrypted payload and the wrapped DEK together.

A rotation adds a new KEK generation and re-encrypts each stored DEK with it. The signing keys themselves are never decrypted or rewritten, nothing restarts, and SVID issuance continues throughout. Existing data converges onto the new generation on its own, and how long that takes scales with the number of stored signing keys.

The activation delay​

A new KEK generation can decrypt immediately, but it does not start encrypting new writes for 10 minutes. Writes during that window use the previous generation.

The delay gives every running pod time to pick up the new KEK generation before anything is written under it. Pods pick it up without a restart.

KEK generations​

A rotation always keeps two KEK generations: The current generation and the one before it. Keeping the previous generation protects writes that were already in flight when the new one took over.

Fully purging a specific KEK generation, therefore, takes two completed rotations.

To purge a KEK generation you believe is exposed, request a second rotation once the first has completed.

A KEK generation is never deleted while stored data still references it. When a rotation cannot finish, extra generations are kept rather than risk losing a KEK the data still needs. To tell whether a rotation is stuck and find the failing step, see KEK rotation not progressing in the Trust Domain Server Runbook.

Configuring a schedule​

Set an interval in your spirl-server values:

trustDomainDeployment:
  kekRotation:
    interval: "720h"

The interval is a whole number of hours from 24h (one day) through 8760h (one year), written as a string. Whole hours are the only accepted spelling, so 1440m and 30d are rejected even where the equivalent hour value would be accepted. The chart fails to render on an interval it cannot accept:

trustDomainDeployment.kekRotation.interval must be a whole number of hours, e.g. "720h", got "1h30m"
trustDomainDeployment.kekRotation.interval must be between 24h (1 day) and 8760h (1 year), got "12h"

The schedule counts from the last rotation, not from pod start, so restarts and rescheduling do not reset it.

Omitting the kekRotation block disables the schedule. Manual rotation still works, and a rotation already under way still finishes.

Unsupported combinations​

The chart rejects a kekRotation block on a deployment configured with a cloud KMS:

trustDomainDeployment.kekRotation requires the Kubernetes-Secret KEK backend; it cannot be combined with aws.kmsKeyARN, azure.keyVault, or gcp

Requesting rotation on demand​

Trigger a rotation by overwriting the defakto.security/kek-rotation-requested-at annotation on the KEK Secret with an RFC 3339 timestamp. The server already holds a Kubernetes watch on that Secret, so the change creates no Kubernetes object and needs no change to the server's own permissions. You need permission to patch the Secret.

Removing a specific KEK that you believe is exposed takes two rotations, because rotation always keeps the generation before the current one.

kubectl annotate secret -n <namespace> <release>-td-server-enc-keys \
  'defakto.security/kek-rotation-requested-at='"$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --overwrite

To rotate more than one Trust Domain Server deployment, annotate each deployment's Secret.

A request needs no configured interval. It does need the Kubernetes Secret KEK backend.

How a request is interpreted​

  • The value must be an RFC 3339 timestamp, with or without fractional seconds.
  • Each later rotation needs a strictly newer timestamp. Leave the annotation in place and overwrite it.
  • An absent, repeated, equal, older, or invalid value does not rotate. An invalid value is logged.
  • Each timestamp produces at most one new KEK generation, however many pods are running and whatever restarts happen in between.
  • A timestamp set during the activation delay, or while a previous rotation is still converging, stays pending and takes effect once rotation is safe.

A new KEK generation appears shortly after you set the annotation. It starts encrypting new writes once the activation delay passes, and stored data converges onto it afterwards. Nothing is written back to the annotation to report completion.

warning

Do not edit the data fields of a KEK Secret. The KEK material and the rotation state live there, and hand-editing them can make stored data impossible to decrypt. The annotation is the only supported way to drive a rotation from kubectl.

Restoring a KEK Secret from backup is not automatic. If the Secret is lost, restore the Secret and the encrypted resources from the same point in time.

Monitoring​

The server exports metrics covering rotation. See Defakto Server Metrics for the list and KEK Rotation Metrics for what to alert on, and the Trust Domain Server Runbook for what to check when rotation stops progressing.