Operations

This page covers how the Agent caches credentials for brokered subscriptions, and the metrics available for observing broker activity.

Caching

The Agent keeps one cached credential per resolved workload, node-wide and shared across subscriptions, up to a soft bound of 1,000 entries. A broker that subscribes again for a workload that already has a valid cached credential adopts that credential instead of issuing a new one, so broker reconnects and repeated subscriptions to the same pod do not multiply issuance requests.

What shares an entry

An entry is keyed on the workload the reference resolved to:

• Kubernetes object references key on the pinned pod UID alone. Namespace and name are deliberately excluded, so a name-only reference and a UID-only reference to the same pod hit the same entry.
• Process-ID references key on the PID together with the pinned process's start time, so two PIDs in the same pod do not share an entry.

The broker's identity is not part of the key either. Entitlement is granted per reference type rather than per workload, so the credential for a resolved workload does not vary by which broker asked for it, and two entitled brokers subscribing to the same pod share one entry.

A reference that cannot be keyed has its credential issued uncached rather than failing.

A cached credential is not a stale credential

The cache stores credentials and makes no freshness judgment of its own. Every check that applies to a freshly issued credential applies identically to one the subscription adopted:

• Adoption is declined outright when the cached credential's attested attributes differ from what the subscription just attested, so attribute drift produces a fresh issuance.
• A credential past its half-life is re-issued.
• A credential signed by a tainted keyset is re-issued, subject to the pacing described below.
• A credential that fails validation against the Agent's current trust bundles is re-issued.

A stale, drifted, or tainted credential is therefore re-issued, never served.

A keyset rotation does not become a thundering herd. Brokered subscriptions follow the same tainting and pacing behavior as the Workload API, described in Signing Key Rotation. The pacing applies only to taint-triggered re-issuances. Rotations driven by expiry or by a change in the workload's attributes are not paced.

Eviction

The bound is soft. No request fails because the cache is over its 1,000-entry bound. An entry a live subscription is using is never evicted to satisfy it, because evicting it would force a working stream to obtain a fresh credential over unrelated cache pressure.

Eviction follows three rules:

• An entry a live subscription holds is never evicted, however far the cache is over its bound.
• An unclaimed entry past its expiry is evicted unconditionally, whether or not the cache is over its bound.
• Unclaimed entries are evicted in least-recently-used order while the cache is over its bound.

Eviction is opportunistic rather than scheduled. Eviction runs when an entry is stored, claimed, or released, and there is no background reaper. A node with no broker traffic at all can hold an expired entry until its next issuance or release. That is deliberate, and it is not a leak: An expired entry occupies space until the next piece of work sweeps it, and it is never served to a broker, because a subscription obtains a fresh credential rather than adopting one that fails its own checks.

Observing the cache

Four metrics describe the cache: spirl_agent_broker_ref_cache_entries, spirl_agent_broker_ref_cache_claims, spirl_agent_broker_ref_cache_lookups_total (labeled result, either hit or miss), and spirl_agent_broker_ref_cache_evictions_total (labeled reason, either expired or bound). A hit means an entry was found and offered to a subscription, which may still decline it and request a fresh credential.

The over-bound state is reported as a pair of edge-triggered log lines, one when the cache goes over its bound with nothing idle to evict and one when it comes back within it:

WARN  Broker per-reference SVID cache is over its bound with no idle entries to evict
        entries=1043  maxEntries=1000
INFO  Broker per-reference SVID cache is back within its bound
        entries=1000  maxEntries=1000

The warning is not an error condition on its own. The warning only reports that the node is holding more live brokered workloads than the bound anticipated and therefore may consume more memory than expected.

Observability

Broker metrics are collected whenever Agent metrics are enabled. The full typed and labeled list is in the metrics catalog. The series worth watching are:

Metric	Key Labels	What it measures
spirl_agent_broker_endpoint_enabled	—	Whether the endpoint is listening. Only exists when the broker socket path is set, then reads 0 at startup and 1 once the socket is serving. An absent series means either the socket path isn't set or Agent metrics are off
spirl_agent_broker_subscriptions	method	Live subscription streams the endpoint is holding, per subscription method. Pre-materialized at 0, so an idle endpoint reports 0 rather than nothing
spirl_agent_broker_policy_denials_total	reason	Denied requests, broken out by why they were denied. See Denial reasons
spirl_agent_broker_reference_resolution_total	svid_type status_code reference_type	Reference resolutions and their outcome, by reference type. reference_type is unknown when the request failed before a resolver was chosen
spirl_agent_broker_reference_resolution_duration_seconds	svid_type status_code reference_type	How long resolution took. Resolution attests the workload, so this is the latency a broker waits through before its first credential
spirl_agent_broker_ref_cache_lookups_total	result	Credential cache hits and misses. See Observing the cache
spirl_agent_broker_ref_cache_entries	—	Current cache size, against the 1,000-entry soft bound

Brokered requests also appear in the Agent's existing series. spirl_agent_broker_requests_total, labeled method and status_code, counts every request the endpoint handled, including the unary FetchJWTSVID. Credentials issued for brokered workloads are counted in the Agent's ordinary spirl_agent_mint_svid_total, and a brokered resolution is counted both in spirl_agent_broker_reference_resolution_total and in the shared spirl_agent_workload_attestation_total. The same resolution appearing in two attestation counters is expected.

Denial reasons

spirl_agent_broker_policy_denials_total is labeled reason, which takes one of:

Value	Meaning
no_caller_id	The request reached the policy check with no authenticated caller. Callers are identified at the connection's front door, so this counter should stay at zero
no_policy_source	The Agent holds no BrokerPolicy at all, so it denies every broker rather than serve without a check
not_allowlisted	The broker's SPIFFE ID has no entry in the cluster's BrokerPolicy
not_entitled	The broker is allowlisted, but not for the reference type its request used
local_only	A reference type was used over a transport that is not the local socket. Both reference types are local-only, and the endpoint listens only on a Unix domain socket, so this counter should stay at zero

This breakdown is the only signal that separates no_policy_source from not_allowlisted, because spirl_agent_broker_requests_total collapses both into one PermissionDenied bucket. The difference matters: The first means no policy reached the Agent, and the second means the policy reached it and does not list this broker.