Troubleshooting
This page covers the issues an operator runs into standing up or maintaining the Broker Endpoint: The socket, permissions, and the endpoint failing to come up at all. A broker rejected at the policy or reference-resolution layer returns a gRPC status carrying its own message, and denials are counted in spirl_agent_broker_policy_denials_total broken out by reason. See Denial reasons for what each reason value means, and Broker Policy for configuring the allowlist.
Connection refused, or permission denied on the socket
A broker that fails before it gets a gRPC status is failing on the socket itself. The operating system error tells you which of three problems it is:
| Error the broker reports | Meaning |
|---|---|
no such file or directory | The path the broker dialed does not exist in the broker's own mount namespace. |
connection refused | The path exists, and nothing is listening on it. |
permission denied | The socket exists and is listening, and the broker's user and groups do not have access to it. |
Diagnose in this order.
1. Confirm the endpoint is enabled. spirl_agent_broker_endpoint_enabled reads 1 when the Agent is serving, and the Agent logs Starting Broker Endpoint with the socket path. See Verifying the endpoint is up. The metric only exists once agent.brokerEndpoint.socketPath is set, so an absent series means the endpoint is off. A series present but reading 0 means the socket path is set but the Agent has not yet started listening, or has since shut down. See The endpoint never comes up.
2. Confirm the socket exists on the node, at the path the broker dials. The Agent does not create the socket until it holds its own serving credential, so a missing socket on an Agent that is otherwise running usually means the Agent has not finished starting. Check the socket field on the Starting Broker Endpoint log line against the path the broker is configured with, since the two are configured independently and a mismatch produces no such file or directory.
3. Confirm the broker's pod mounts the socket's directory. The chart mounts the socket's parent directory into the Agent only. The broker's pod needs its own hostPath volume for the same directory. A CSI-injected Workload API mount does not reach that directory, because the socket directory is required to be outside the Workload API socket directory. See Socket path constraints.
4. Confirm group ownership. This is the most common first-attempt failure and the cause of permission denied. The socket is mode 0770, owned by the Agent process's uid and gid, and a broker running with capabilities dropped can neither traverse a directory nor connect to a socket that it does not own and shares no group with:
kubectl exec -n spirl-system <agent-pod-name> -- ls -ld /run/spirl/broker /run/spirl/broker/broker.sock
Compare the group shown against the groups the broker's pod runs with, including securityContext.supplementalGroups. See Filesystem permissions for how to coordinate the two.
The endpoint never comes up
Three causes account for most of these, and each presents differently.
The Agent is running, and the endpoint is off. spirl_agent_broker_endpoint_enabled is absent from the scrape entirely, and there is no Starting Broker Endpoint log line. The Agent has no socket path. Confirm that the Helm values actually reached the DaemonSet:
kubectl get daemonset -n spirl-system spirl-agent \
-o jsonpath='{.spec.template.spec.containers[*].args}' | tr ' ' '\n' | grep broker
An empty result means agent.brokerEndpoint.socketPath is unset in the values that were applied. See Enabling via Helm.
The Agent will not start at all. A socket path that collides with the Workload API socket directory is a fatal error rather than a warning. The Agent exits with:
broker socket "/run/spirl/sockets/broker.sock" must not share a directory with
the Workload API socket "/run/spirl/sockets/agent.sock", nor sit inside or
around its directory
The chart catches the same collision first, so a helm upgrade fails before anything is applied:
agent.brokerEndpoint.socketPath must not live in or around the world-writable
Workload API socket directory (/run/spirl/sockets); the agent refuses to start
if either directory contains the other
Move the broker socket to a directory that neither contains nor sits inside the Workload API socket directory. /run/spirl/broker/broker.sock alongside the default /run/spirl/sockets works. Symlinks are not resolved during that comparison, so a symlinked path that overlaps the Workload API directory passes both checks and should be avoided. See Socket path constraints.
The Agent pod stays NotReady. When a socket path is configured, the Agent's /ready health check includes a check that only passes once the broker socket is actually listening, alongside its existing checks like the bundle cache being populated. This is an AND across all checks: A Broker Endpoint that never starts listening holds the whole pod NotReady, not just broker traffic. Since the socket isn't opened until the Agent has obtained its own serving SVID, an Agent stuck NotReady with the Broker Endpoint enabled is often stuck earlier, on attestation or SVID issuance, rather than on the endpoint itself. Check for Starting Broker Endpoint in the Agent's logs. A missing Starting Broker Endpoint line, combined with NotReady, points at the Agent's own identity bootstrap rather than the Broker Endpoint. See Understanding Agent Health Checks for how the Agent's readiness checks work.