Kubernetes
This guide walks you through installing the Defakto Agent on a Kubernetes cluster using the official Helm chart. The Defakto Agent runs as a DaemonSet and provides SPIFFE identity services to workloads running in your cluster.
Prerequisites
Before installing the Defakto Agent, ensure you have:
- A running Kubernetes cluster
kubectlconfigured to access your cluster- Helm 3.x installed
- A deployed Trust Domain Server
- Cluster registration details from your Trust Domain Server
Step 1 — Choose an agent attestation method
The recommended attestation method for Kubernetes clusters is the Kubernetes Service Account Token. No secret material is required; the agent presents its pod's service account token to authenticate.
See Agent Attestation Methods for the full list of available methods and server-side configuration.
Step 2 — Create the Helm values
Create a values.yaml file:
agent:
endpoint:
endpoint: "td-server.defakto.example.com"
auth:
clusterId: c-xxxxxx
attestors:
- type: k8s_token
To reference the cluster by name instead of ID, use clusterName in place of clusterId. See Identifying the cluster.
Step 3 — Download and install the Helm chart
helm pull oci://ghcr.io/spirl/charts/defakto-system -d .
helm upgrade --install --namespace defakto-system \
--create-namespace \
--values ./values.yaml \
defakto-system ./defakto-system-0.48.2.tgz
You should see:
Release "defakto-system" does not exist. Installing it now.
NAME: defakto-system
LAST DEPLOYED: Tue Dec 19 14:02:09 2023
NAMESPACE: defakto-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
Step 4 — Verify
Check your target namespace for the Defakto Agent pods and verify that they are running.
When the agent is up and running, you'll see a health check confirmation log:
{
"logger": "agent.readyChecker",
"msg": "Health check passed - all systems operational"
}
Check the logs of the Defakto Agent pods to verify that they are connected to the Trust Domain Server:
kubectl -n defakto-system logs $(kubectl -n defakto-system get pods -o name | grep "defakto-agent" | head -n 1)
When a deployment is receiving and processing bundle updates successfully, you should see the following log message:
{
"logger": "agent.bundleRefresher",
"msg": "Bundle added or updated",
"trustDomain": "defakto.example.com"
}
The trustDomain key will contain the name of your trust domain.
TLS configuration
How the agent trusts the Trust Domain Server, including trusting a private CA and disabling TLS, is the same on every platform. See Agent TLS configuration for the Helm values to use on Kubernetes.
If you need to dial a different address than the one contained in the Subject Alternative Name (SAN) on the Trust Domain Server certificate, map each dial endpoint to the certificate's SAN. This is useful if you are using a gateway like AWS PrivateLink to reach the servers.
agent:
endpoint:
endpoint: "primary.internal:443"
endpoints:
- "backup.internal:443"
serverNameOverrides:
"primary.internal:443": "tdd.prod.defakto.com"
"backup.internal:443": "tdd.prod.defakto.com"
The agent will dial the endpoint and validate the server TLS certificate as if it had dialed to the serverName directly.
Working with Node Taints and Tolerations
If your Kubernetes cluster has tainted nodes that require tolerations for pod scheduling, you can configure tolerations for the Defakto Agent components. This is particularly useful when deploying on:
- Nodes with dedicated workloads
- Master/control plane nodes
- Nodes with special hardware requirements
- Nodes with specific scheduling constraints
Different Defakto components have different scheduling requirements:
- Agent & CSI Driver: Must run on every node that hosts workloads requiring SPIFFE identities. These components need the same tolerations to ensure they can be scheduled on all workload nodes.
- Controller: Only needs to run somewhere in the cluster and can be scheduled independently. You may want different (or no) tolerations for the controller, allowing it to run on dedicated management nodes.
Example: Agent and CSI Driver Tolerations
For workloads running on tainted nodes, configure the agent and CSI driver with matching tolerations:
agent:
endpoint:
endpoint: "td-server.defakto.example.com"
auth:
clusterId: c-xxxxxx
attestors:
- type: k8s_token
tolerations:
- key: "node.kubernetes.io/unschedulable"
operator: "Exists"
effect: "NoSchedule"
- key: "dedicated"
operator: "Equal"
value: "workload"
effect: "NoSchedule"
csiDriver:
tolerations:
- key: "node.kubernetes.io/unschedulable"
operator: "Exists"
effect: "NoSchedule"
- key: "dedicated"
operator: "Equal"
value: "workload"
effect: "NoSchedule"
Example: Controller with Different Tolerations
The controller can be configured with different tolerations, or run on untainted nodes:
# Option 1: Controller on management nodes
controller:
tolerations:
- key: "node-role.kubernetes.io/master"
operator: "Exists"
effect: "NoSchedule"
# Option 2: Controller with no specific tolerations (runs on any untainted node)
# controller: {}
# Option 3: Controller tolerating all taints
controller:
tolerations:
- operator: "Exists"
Common Toleration Patterns
Tolerate all taints:
agent:
tolerations:
- operator: "Exists"
csiDriver:
tolerations:
- operator: "Exists"
Tolerate specific key with any value:
agent:
tolerations:
- key: "dedicated"
operator: "Exists"
effect: "NoSchedule"
csiDriver:
tolerations:
- key: "dedicated"
operator: "Exists"
effect: "NoSchedule"
Tolerate specific key-value pair:
agent:
tolerations:
- key: "node-type"
operator: "Equal"
value: "compute"
effect: "NoSchedule"
csiDriver:
tolerations:
- key: "node-type"
operator: "Equal"
value: "compute"
effect: "NoSchedule"
If you use AWS EKS with some workloads scheduled on Fargate, you may see some agent or CSI driver pods stuck in
Pending if you use a broad toleration that matches the taint used by Fargate nodes. You can
use node affinity in addition to your toleration to ensure that Fargate nodes are excluded. For example:
agent:
tolerations:
- operator: "Exists"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: eks.amazonaws.com/compute-type
operator: NotIn
values:
- fargate
csiDriver:
tolerations:
- operator: "Exists"
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: eks.amazonaws.com/compute-type
operator: NotIn
values:
- fargate
Working with Custom Docker Registries
Many enterprise environments require pulling images from private registries or through registry proxies. The Defakto Agent Helm chart supports custom registries and image pull secrets for secure image access.
Defakto Agent uses multiple container images from different sources:
- Defakto Images: Agent and Controller (default:
ghcr.io/spirl) - SPIFFE CSI Driver: From the SPIFFE project (default:
ghcr.io/spiffe) - Kubernetes CSI Registrar: From Kubernetes SIG Storage (default:
registry.k8s.io/sig-storage)
You can configure each component to use your custom registry independently.
Creating Image Pull Secrets
Before deploying, create the required image pull secrets:
kubectl create secret docker-registry my-registry-secret \
--namespace defakto-system \
--docker-server=my-registry.company.com \
--docker-username=myuser \
--docker-password=mypassword
Configuration Examples
Private Registry with Authentication:
images:
repository: "my-registry.company.com/defakto"
pullSecrets:
- name: "my-registry-secret"
csiDriver:
repository: "my-registry.company.com/spiffe"
csiNodeDriverRegistrar:
repository: "my-registry.company.com/sig-storage"
Registry Proxy/Mirror:
images:
repository: "proxy.company.com/ghcr.io/defakto"
csiDriver:
repository: "proxy.company.com/ghcr.io/spiffe"
csiNodeDriverRegistrar:
repository: "proxy.company.com/registry.k8s.io/sig-storage"
Air-Gapped Environment (Pin Versions):
images:
repository: "internal-registry.local/defakto"
pullSecrets:
- name: "internal-registry-secret"
agent:
tag: "v1.2.3"
controller:
tag: "v1.2.3"
csiDriver:
tag: "0.2.7"
repository: "internal-registry.local/spiffe"
csiNodeDriverRegistrar:
tag: "v2.14.0"
repository: "internal-registry.local/sig-storage"
Helm Values Reference
This section provides a comprehensive reference for all configuration options available in the Defakto Agent Helm chart. These values can be used to customize your Defakto Agent deployment to meet specific requirements.
Your values file is merged over the defaults the chart ships, key by key. You only need to set what you want to change, and any key you leave out keeps the chart's default. The values.yaml you create in Step 2 is your own file. The chart carries a separate file of defaults inside the downloaded .tgz.
When a key is set in more than one place, later sources win, in this order: the chart's defaults, then each --values file in the order given, then --set.
Agent Settings
These settings configure the agent on any Kubernetes cluster. Set them at the top level of your values file.
socketLayout: Socket layout preset. The only value isISTIO, which places the workload socket where Istio expects to find it. Cannot be set together withsocket. Configures the following presets:spiffeSocketPath:"/run/spirl/sockets/socket",volumeName:"workload-socket", andvolumeMountPath:"/run/secrets/workload-spiffe-uds"(default: unset, available indefakto-system>= 0.47.0)socket: Explicit workload socket settings, as an alternative tosocketLayout. Cannot be set together withsocketLayout. Each key is optional, and a key you leave out falls back to the blockplatformselects, then to the defaults below (available indefakto-system>= 0.47.0)socket.spiffeSocketPath: Path where the agent creates the Workload API socket. Must name a file not a directory (default:"/run/spirl/sockets/agent.sock", available indefakto-system>= 0.47.0)socket.volumeName: Name the controller gives the CSI volume carrying the socket (default:"defakto-agent-socket", or"spirl-agent-socket"beforedefakto-system0.48.2; available indefakto-system>= 0.47.0)socket.volumeMountPath: Path where the controller mounts that volume in workload pods (default:"/defakto-agent-socket", or"/spirl-agent-socket"beforedefakto-system0.48.2; available indefakto-system>= 0.47.0)
skipKubeletVerification: Whether the agent skips verification of the kubelet's TLS certificate (default:true, so the certificate is not verified). Set it tofalseto verifykubeletHost: Kubelet host override. Unset by default, in which case the agent uses the node's IP address. An empty string also means the node's IP address, even whenk8s.kubeletHostis setincludeAnnotations: Array of pod annotation prefixes to include in attestation (default:[]). Each prefix must start with a domain name and end in a slash, such asapp.internal.dev/includeContainerAttributes: Whether container attributes are included in workload attestation (default:false). Leave thisfalsewhen Istio is configured withholdApplicationUntilProxyStarts=true, because the workload container may not be ready at the time of attestationupdateStrategy: DaemonSet update strategy (default: RollingUpdate with maxUnavailable: 0, maxSurge: 10%). Keys you set are merged with the default, so setting onlyrollingUpdate.maxSurgekeepsmaxUnavailable: 0
Settings for your cluster
Each row produces the same deployment as the matching platform value.
| Cluster | Settings |
|---|---|
| Kubernetes | None. The defaults apply. |
| Kubernetes with Istio | socketLayout: ISTIO |
| Amazon EKS | skipKubeletVerification: false |
| Amazon EKS with Istio | socketLayout: ISTIO and skipKubeletVerification: false |
Platform Configuration (Deprecated)
New values files should use Agent Settings rather than platform and the per-platform blocks. platform is still supported so that existing values files keep working, but we do not recommend it for new configuration. To see which Agent Settings reproduce a platform, see Settings for your cluster.
A setting you set in Agent Settings takes precedence over the same setting in the block platform selects. A setting you leave out uses the block's value, not the Agent Settings default, so you can move one setting at a time. For example, with platform: EKS and no top-level skipKubeletVerification, the agent verifies the kubelet, because the eks block's false applies.
platform: Selects one of the platform-specific blocks. Optional. When omitted, the chart behaves asplatform: K8S. Thek8sblock applies, and Agent Settings take precedence over it. Values are uppercase. Supported values:K8S- KubernetesISTIO- Kubernetes with an Istio service mesh. Places the workload socket where Istio expects to find itEKS- Amazon EKS. The agent verifies the kubelet's TLS certificate by defaultEKS_ISTIO- Amazon EKS with an Istio service mesh. Both of the above
Helm validates your values file against the chart schema before rendering anything, and the schema accepts only the uppercase spellings above. Setting platform: k8s fails with:
Error: values don't meet the specifications of the schema(s) in the following chart(s):
defakto-system:
- at '/platform': value must be one of 'K8S', 'ISTIO', 'EKS', 'EKS_ISTIO', 'LINUX'
The schema also accepts LINUX, but Linux hosts do not use this chart. See Linux.
The per-platform setting blocks below use lowercase names, so platform: ISTIO selects the istio block. Only the platform value itself is uppercase.
The casing rule covers only the Helm platform value. The defakto --platform flag is separate, and takes lowercase values such as istio and eks-istio.
ArgoCD Configuration
argocd.enabled: Enable ArgoCD annotations for Kubernetes resources (default:false). When enabled, all Kubernetes resources will be annotated with ArgoCD-specific annotations that govern resource lifecycle management.
Common Labels
commonLabels: Map of labels added to every resource and pod the chart renders, such as labels that admission policies require for ownership or cost allocation (default:{}). A component'sadditionalLabelstake precedence on a key conflict. Setting a key the chart owns or selects on (app,app.kubernetes.io/name,app.kubernetes.io/instance,app.kubernetes.io/version,app.kubernetes.io/managed-by,helm.sh/chart) fails the render. CRDs don't receive these labels (available indefakto-system>= 0.48.0)
Images Configuration
images.repository: Base repository for all images (default:"ghcr.io/spirl")images.pullPolicy: Image pull policy (default:"IfNotPresent")images.pullSecrets: Array of image pull secrets (default:[])images.pullCredentials: Not recommended, useimages.pullSecretsinstead. Array of registries and corresponding secrets in the formbase64("username:password"). The array will be used to generate a dockerconfigjson secret that will be used as an imagePullSecret (default:[])images.controller.name: Controller image name (default:"defakto-controller")images.controller.tag: Controller image tag (default: Chart appVersion)images.controller.repository: Controller-specific repository (optional)images.agent.name: Agent image name (default:"defakto-agent")images.agent.tag: Agent image tag (default: Chart appVersion)images.agent.repository: Agent-specific repository (optional)images.csiDriver.name: CSI driver image name (default:"spiffe-csi-driver")images.csiDriver.tag: CSI driver image tag (default:"0.2.7")images.csiDriver.repository: CSI driver repository (default:"ghcr.io/spiffe")images.csiNodeDriverRegistrar.name: CSI node driver registrar image name (default:"csi-node-driver-registrar")images.csiNodeDriverRegistrar.tag: CSI node driver registrar image tag (default:"v2.14.0")images.csiNodeDriverRegistrar.repository: CSI node driver registrar repository (default:"registry.k8s.io/sig-storage")
Agent Configuration
Endpoint Configuration
agent.endpoint.endpoint: Primary trust domain server endpoint (default:"")agent.endpoint.endpoints: Array of additional fallback endpoints, in priority order (default:[])agent.endpoint.supplementalRootsPEM: Additional root CAs for endpoint trust (PEM format)agent.endpoint.serverNameOverrides: Map of dial endpoint addresses to the TLS server name used for certificate validation (default:{}). See TLS configuration.agent.endpoint.connectionMaxAge: Maximum age for endpoint connections (default:"30m")agent.endpoint.useGRPCFastRedial: Enable fast redialing for gRPC connections (default:true)
For a Trust Domain Server deployed across multiple regions or replicas for HA, list all endpoints under endpoints in priority order:
Authentication Configuration
Kubernetes Service Account Token
agent.auth.attestors: List of attestation methods. Add an entry withtype: k8s_tokento enable Kubernetes Service Account Token authentication.
Cluster Version Key (CVK)
agent.auth.key.id: Cluster version IDagent.auth.key.pem: Private key for cluster authentication (PEM format)agent.auth.suppliedSecretName: Name of pre-existing secret for authentication
Security and Resources
agent.runWithoutPrivileged: Run the agent withoutprivileged: trueagent.namespaceRBAC: Grant the agentget/list/watchon namespaces (default:true). The agent needs this to attest the workload pod's Namespace object askubernetes.namespace.*attributes, which it reads from the API server rather than from the kubelet. Namespace attestation requires thedefakto-systemchart at version 0.46.0 or later. See Rollout order on existing deployments. Set it tofalseto withhold cluster-wide namespace read. Attestation then emits nokubernetes.namespace.*attributes, and any path template, certificate or JWT template, or issuance policy referencing one stops matching.agent.supplementalRootsPEM: Additional trust anchors for trust bundleagent.podSecurityContext: Pod security context configurationagent.containerSecurityContext: Container security context configurationagent.resources: Resource requests and limitsagent.priorityClassName: Priority class name for pod scheduling
Scheduling and Placement
agent.affinity: Node affinity rulesagent.tolerations: Array of node tolerations for tainted nodes (default:[])agent.nodeSelector: Map of node selector labels (default:{})agent.annotations: Map of pod annotations (default:{})agent.additionalLabels: Map of additional labels for the agent DaemonSet and its pods (default:{}). Beforedefakto-system0.48.0, these labeled only the pods
Advanced Configuration
agent.env: Array of environment variables (default:[])agent.additionalVolumes: Array of additional volumes (default:[])agent.additionalVolumeMounts: Array of additional volume mounts (default:[])agent.allowedAttributes: Array of allowed workload attributes (default:[])
Controller Configuration
controller.env: Array of environment variables (default:[])controller.podSecurityContext: Pod security context configurationcontroller.containerSecurityContext: Container security context configurationcontroller.resources: Resource requests and limitscontroller.affinity: Node affinity rulescontroller.tolerations: Array of node tolerations (default:[])controller.nodeSelector: Map of node selector labels (default:{})controller.annotations: Map of pod annotations (default:{})controller.additionalLabels: Map of additional labels for the controller Deployment and its pods (default:{}). Beforedefakto-system0.48.0, these labeled only the podscontroller.priorityClassName: Priority class name
CSI Driver Configuration
csiDriver.env: Array of environment variables (default:[])csiDriver.podSecurityContext: Pod security context configurationcsiDriver.containerSecurityContext: Container security context configurationcsiDriver.resources: Resource requests and limitscsiDriver.affinity: Node affinity rulescsiDriver.tolerations: Array of node tolerations (default:[])csiDriver.nodeSelector: Map of node selector labels (default:{})csiDriver.annotations: Map of pod annotations (default:{})csiDriver.additionalLabels: Map of additional labels for the CSI driver DaemonSet and its pods (default:{}). Beforedefakto-system0.48.0, these labeled only the podscsiDriver.pluginName: Name the CSI driver registers with kubelet and the controller's webhook injects into workload pods (default:"csi.spiffe.io"). Set a unique name for each agent when running more than one agent in a clustercsiDriver.priorityClassName: Priority class name
CSI Node Driver Registrar Configuration
csiNodeDriverRegistrar.env: Array of environment variables (default:[])csiNodeDriverRegistrar.containerSecurityContext: Container security context configurationcsiNodeDriverRegistrar.resources: Resource requests and limits
Node Configuration
node.kubeletRootDirectory: Kubelet root directory path (default:/var/lib/kubelet)node.spirlAgentSocketDirectory: Defakto agent socket directory path
Platform-Specific Configuration
These blocks exist for values files that set platform. New values files should use Agent Settings instead.
Each block below applies when platform is set to the matching value. The k8s block also applies when platform is omitted. Block names are lowercase, so platform: EKS_ISTIO selects the eksIstio block. Only a top-level setting you set yourself takes precedence over the same key in a block.
The platforms differ in socket layout, which the key names below make plain, and in kubelet verification, which they do not. The agent reaches the kubelet over TLS to gather pod attributes during workload attestation. By default the k8s and istio blocks skip verification of the kubelet's certificate, and the eks and eksIstio blocks verify it, using the service account CA bundle that Kubernetes mounts into every pod. skipKubeletVerification in each block changes the default.
Kubernetes (k8s)
k8s.spiffeSocketPath: SPIFFE socket path (default:"/run/spirl/sockets/agent.sock")k8s.skipKubeletVerification: Whether the agent skips verification of the kubelet's TLS certificate (default:true, so the certificate is not verified). Set it tofalseto verifyk8s.kubeletHost: Kubelet host override. Unset by default, in which case the agent uses the node's IP addressk8s.includeAnnotations: Array of pod annotation prefixes to include in attestation (default:[])k8s.includeContainerAttributes: Whether container attributes are included in workload attestation (default:false)k8s.updateStrategy: DaemonSet update strategy configuration (default: RollingUpdate with maxUnavailable: 0, maxSurge: 10%)
Istio (istio)
istio.spiffeSocketPath: SPIFFE socket path (default:"/run/spirl/sockets/socket")istio.skipKubeletVerification: Whether the agent skips verification of the kubelet's TLS certificate (default:true, so the certificate is not verified). Set it tofalseto verifyistio.socketVolumeName: Socket volume name (default:"workload-socket")istio.socketVolumeMountPath: Socket volume mount path (default:"/run/secrets/workload-spiffe-uds")istio.socketFileName: Socket file name (default:"socket"). Must be the file name at the end ofistio.spiffeSocketPathistio.includeAnnotations: Array of pod annotation prefixes to include in attestation (default:[])istio.includeContainerAttributes: Whether container attributes are included in workload attestation (default:false). Leave thisfalsewhen Istio is configured withholdApplicationUntilProxyStarts=true, because the workload container may not be ready at the time of attestationistio.updateStrategy: DaemonSet update strategy configuration (default: RollingUpdate with maxUnavailable: 0, maxSurge: 10%)
Amazon EKS (eks)
eks.spiffeSocketPath: SPIFFE socket path (default:"/run/spirl/sockets/agent.sock")eks.skipKubeletVerification: Whether the agent skips verification of the kubelet's TLS certificate (default:false, so the certificate is verified). Set it totrueto skipeks.includeAnnotations: Array of pod annotation prefixes to include in attestation (default:[])eks.includeContainerAttributes: Whether container attributes are included in workload attestation (default:false)eks.updateStrategy: DaemonSet update strategy configuration (default: RollingUpdate with maxUnavailable: 0, maxSurge: 10%)
Amazon EKS with Istio (eksIstio)
eksIstio.spiffeSocketPath: SPIFFE socket path (default:"/run/spirl/sockets/socket")eksIstio.skipKubeletVerification: Whether the agent skips verification of the kubelet's TLS certificate (default:false, so the certificate is verified). Set it totrueto skipeksIstio.socketVolumeName: Socket volume name (default:"workload-socket")eksIstio.socketVolumeMountPath: Socket volume mount path (default:"/run/secrets/workload-spiffe-uds")eksIstio.socketFileName: Socket file name (default:"socket"). Must be the file name at the end ofeksIstio.spiffeSocketPatheksIstio.includeAnnotations: Array of pod annotation prefixes to include in attestation (default:[])eksIstio.includeContainerAttributes: Whether container attributes are included in workload attestation (default:false). Leave thisfalsewhen Istio is configured withholdApplicationUntilProxyStarts=true, because the workload container may not be ready at the time of attestationeksIstio.updateStrategy: DaemonSet update strategy configuration (default: RollingUpdate with maxUnavailable: 0, maxSurge: 10%)
Telemetry Configuration
telemetry.enabled: Enable telemetry collection (default:false)telemetry.metricsAPI.port: Port for metrics API exposure (default:9090)telemetry.collectors.grpc.emitLatencyMetrics: Enable gRPC latency metrics (default:false)telemetry.collectors.workload.emitSVIDExpiry: Enable workload SVID expiry timestamp metrics (default:false)
Webhook Configuration
webhook.namespaceSelector: Namespace selector configuration for admission webhook (default:{})webhook.objectSelector: Object selector configuration for admission webhook (default: matches labels withk8s.spirl.com/spiffe-csi: enabled)
Experimental Features
experimental.supplementalAttestation.jwt.issuer: JWT attestation issuerexperimental.supplementalAttestation.jwt.jwksUrl: JWKS URL for JWT validationexperimental.supplementalAttestation.jwt.supplementalRootsFile: Supplemental roots file pathexperimental.useLegacyEnvoyValidation: Use legacy Envoy validation (default:false)