Vault Configuration¶

The vault plugin integrates DRP with HashiCorp Vault as a secure parameter store. DRP parameters can reference secrets stored in Vault KV engines using a URI-based lookup scheme instead of holding plaintext values. The plugin supports KV v1 and KV v2 engines, multiple authentication methods (token, AppRole, JWT), and both read (decrypt) and write (encrypt) operations.

Installation¶

drpcli catalog item install vault

The plugin requires drp-community-content version 4.8.0 or later and the more-external-secrets license feature. Installations on a license without more-external-secrets will fail at catalog install time with an error referencing the missing feature.

Running Vault in a container¶

Production deployments typically run Vault in a container alongside DRP. The minimum viable container deployment:

• Pin a specific image tag. hashicorp/vault:latest is not stable. Pick a release version (e.g., hashicorp/vault:1.21.4) and pin to it.
• Mount a persistent volume for the file storage backend so unseal state and sealed-storage data survive container restarts.
• Grant SETFCAP capability for Vault 2.x images so the entrypoint can set file capabilities. If your runtime forbids SETFCAP (rare), set the SKIP_SETCAP=true environment variable inside the container instead.

A concrete podman run (or docker run) example with a host-side data directory:

podman run -d \
  --name vault \
  --cap-add IPC_LOCK \
  --cap-add SETFCAP \
  -p 8200:8200 \
  -v vault-data:/vault/file \
  -e VAULT_LOCAL_CONFIG='{"storage":{"file":{"path":"/vault/file"}},"listener":{"tcp":{"address":"0.0.0.0:8200","tls_disable":true}},"default_lease_ttl":"168h","max_lease_ttl":"720h","ui":true}' \
  hashicorp/vault:1.21.4 server

After every container restart Vault comes up sealed; an operator must unseal it manually with vault operator unseal <key> before the DRP plugin can read or write secrets. For production deployments where this is operationally impractical, see HashiCorp's auto-unseal documentation for the available auto-unseal backends (transit, AWS KMS, GCP KMS, Azure Key Vault).

Plugin Parameters¶

Configure these parameters on the vault plugin object:

Plugin Creation¶

drpcli plugins create '{
  "Name": "my-vault",
  "Provider": "vault",
  "Params": {
    "vault/address": "https://vault.example.com:8200",
    "vault/token": "s.your-vault-token",
    "vault/kv-version": "v2",
    "vault/cache-timeout": 300
  }
}'

The Name field matters: DRP routes <scheme>:// URIs to the plugin instance whose Name matches the URI scheme. See Dispatch rule below. Examples on this page use Name: my-vault, so their URIs use the my-vault:// scheme.

For Vault instances using a self-signed certificate, provide the CA certificate:

drpcli plugins update Name:my-vault '{"Params": {"vault/ca-cert": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----"}}'

Dispatch rule: URI scheme matches plugin Name¶

DRP's auto-decode dispatcher routes <scheme>://... URIs to the plugin instance whose Name matches the URI scheme. The provider name (vault) is not what the dispatcher matches against.

A decrypt/lookup-uri value of my-vault://foo?path=hello looks for a plugin instance named my-vault and asks it to decrypt. If no plugin instance is named my-vault, the DRP server log shows:

Action decrypt on machines for plugin my-vault Not Found

and drpcli ... --decode returns null silently. The fix is to name your plugin instance to match the URI scheme you want to use, or to use the URI scheme that matches the plugin instance's Name.

Examples on this page use Name: my-vault paired with my-vault:// URIs so the correlation is explicit in every example.

Multiple Vault instances¶

The Name-matches-scheme rule means you can run multiple vault plugin instances side by side, each routed by its own URI scheme:

drpcli plugins create '{"Name":"vault-prod","Provider":"vault","Params":{"vault/address":"https://vault-prod.example.com:8200","vault/token":"s.prod-token"}}'
drpcli plugins create '{"Name":"vault-dr","Provider":"vault","Params":{"vault/address":"https://vault-dr.example.com:8200","vault/token":"s.dr-token"}}'

drpcli machines set Name:my-prod-server param ipmi/password to '{"LookupUri":"vault-prod://ipmi-password?path=servers/prod"}'
drpcli machines set Name:my-dr-server   param ipmi/password to '{"LookupUri":"vault-dr://ipmi-password?path=servers/dr"}'

Each instance uses its own vault/token. Because Vault's token policies control which paths a token can read, you can use the same pattern for per-consumer access isolation: run separate plugin instances pointing at the same Vault, each authenticating with a token whose Vault policy restricts read access to that consumer's path prefix.

For example, to isolate two OpenShift clusters' kubeconfigs:

# vault-ocp-a uses a token whose policy permits only secret/data/ocp-a/*
drpcli plugins create '{"Name":"vault-ocp-a","Provider":"vault","Params":{"vault/address":"https://vault.example.com:8200","vault/token":"s.ocp-a-scoped-token"}}'

# vault-ocp-b uses a token whose policy permits only secret/data/ocp-b/*
drpcli plugins create '{"Name":"vault-ocp-b","Provider":"vault","Params":{"vault/address":"https://vault.example.com:8200","vault/token":"s.ocp-b-scoped-token"}}'

The DRP-side seam is the plugin instance + its vault/token. The Vault-side seam is the token's policy. See HashiCorp's Vault policy documentation for the policy syntax itself.

Authentication¶

The plugin supports three authentication methods, dispatched by the vault/auth-method parameter (defaults to token).

Token authentication¶

Set vault/token to a Vault token. This is the default and matches the simple example in Plugin Creation. Suitable for development and for deployments where a long-lived token (root or otherwise) is acceptable.

AppRole authentication¶

Set vault/auth-method: approle and provide a role-id + secret-id pair. The plugin exchanges them for a Vault token at Config() time and manages the token's lifecycle itself. Suitable for production deployments where per-application credentials and least-privilege policies are required.

vault/token is not required when vault/auth-method is approle; the plugin manages the token lifecycle itself.

Example:

drpcli plugins create '{
  "Name": "my-vault",
  "Provider": "vault",
  "Params": {
    "vault/address": "https://vault.example.com:8200",
    "vault/auth-method": "approle",
    "vault/approle-role-id": "abc-role",
    "vault/approle-secret-id": "xyz-secret"
  }
}'

A least-privilege Vault policy example for an AppRole reading secrets under a drp/ prefix:

path "secret/data/drp/*"     { capabilities = ["read"] }
path "secret/metadata/drp/*" { capabilities = ["read"] }

The plugin does NOT require read access to the kv-mount root. Config()-time validation uses auth/token/lookup-self, which Vault's built-in default policy grants to every non-root token.

JWT authentication¶

Set vault/auth-method: jwt and provide a Vault role + a JWT. The plugin presents the JWT to Vault's JWT auth backend to obtain a Vault token. Suitable for deployments where an identity provider Vault already trusts issues short-lived JWTs that DRP can present (OIDC IDPs like Auth0/Keycloak, Kubernetes API servers federated with Vault, etc.).

The plugin is a JWT consumer, not a JWT minter. Whatever identity source the operator uses is responsible for producing the JWT and updating vault/jwt when rotation is needed; the plugin presents whatever is currently in vault/jwt at each login.

vault/token is not required when vault/auth-method is jwt; the plugin manages the token lifecycle itself.

Example:

drpcli plugins create '{
  "Name": "my-vault",
  "Provider": "vault",
  "Params": {
    "vault/address": "https://vault.example.com:8200",
    "vault/auth-method": "jwt",
    "vault/jwt-role": "drp-vault",
    "vault/jwt": "eyJhbGciOiJSUzI1NiIs..."
  }
}'

A least-privilege role example for a Kubernetes service account named drp-vault in the drp namespace, presenting tokens with audience vault:

vault write auth/jwt/role/drp-vault \
    role_type=jwt \
    bound_audiences=vault \
    bound_subject=system:serviceaccount:drp:drp-vault \
    user_claim=sub \
    token_policies=drp-vault \
    token_ttl=1h \
    token_max_ttl=4h

Vault's "Kubernetes" auth method shares the role + jwt login payload with the generic JWT method; set vault/jwt-mount to the Kubernetes auth mount (typically kubernetes) to use it.

Token Auto-Renewal¶

For renewable Vault tokens, the plugin keeps the token alive in the background so the plugin instance stays usable across the token's max_ttl. The renewal machinery is shared by all three auth methods.

At Config() time, the plugin calls Vault's LookupSelf on the configured token to discover whether the token is renewable and what its current TTL is.

• Renewable tokens with a positive TTL are picked up by the Vault SDK's LifetimeWatcher, which renews at roughly half the TTL with jitter, up to the token's max_ttl.
• Non-renewable tokens (e.g., the root token) skip auto-renewal with an info-level log entry.

When the LifetimeWatcher terminates (token cannot be kept alive any longer), the plugin classifies the cause and writes a descriptive entry to its PluginErrors field so the failure is visible to operators via drpcli plugins show <name>. The classification falls into one of:

• renewal call failed: ... if the renewal call returned an error directly.
• token unreadable post-renewal: ... if the token is gone or rejected (typical aftermath of revocation).
• max_ttl reached or token expired if Vault still knows the token but its TTL is now zero or negative.
• grace period exhausted with <ttl> TTL remaining if the watcher gave up while the token was still notionally valid (almost always because transient renewal errors ate the grace window).

Operator-visible signals. drpcli plugins show <name> exposes two distinct fields:

• Errors: per-action errors from the most-recent failed Config or action attempt. Cleared on a successful reconfig.
• PluginErrors: terminal background-renewal failures appended by the plugin. The DRP server does NOT auto-clear these. Operators clear them explicitly when the underlying issue is resolved.

Re-login on watcher termination¶

For AppRole authentication, when the LifetimeWatcher terminates (token revoked, max_ttl reached, grace period exhausted), the plugin transparently re-logs in using the cached AppRole credentials to mint a fresh token. Re-login is bounded to 6 attempts: one immediate, then 5 retries spaced 5s, 30s, 2m, 5m, 10m apart. Total budget is about 17 minutes. If all attempts fail, the final error is appended to PluginErrors with a message like vault: AppRole re-login exhausted (6 attempts): ....

For JWT authentication, the same bounded-retry machinery applies, with one extra detail: each login attempt reads vault/jwt fresh from the DRP API rather than reusing a value captured at Config() time. That means a rotator that updates vault/jwt between login attempts (or between the prior successful login and the current re-login) is observed at the next natural login moment without requiring a plugin reconfig. The exhaustion message reads vault: JWT re-login exhausted (6 attempts): ....

The vault/token-short-lived incompatibility documented below also applies to JWT auth.

Short-lived tokens with non-token auth¶

vault/token-short-lived: true is incompatible with AppRole and JWT. Short-lived asks the plugin to re-read vault/token from DRP on every fetch (the operator's external rotator owns the token's lifecycle), while AppRole and JWT ask the plugin to manage the token's lifecycle itself via login + renewal + re-login. If both are configured, the plugin logs a warning, appends a deduplicated PluginErrors entry describing the conflict, then runs as the configured non-token method only (short-lived is silently disabled in memory). Short-lived is only meaningful for token auth.

Reading Secrets¶

Set the decrypt/lookup-uri on a machine parameter to reference a secret in Vault:

# Store an IPMI password reference (secret stays in Vault)
drpcli machines set Name:my-server param ipmi/password to \
  '{ "LookupUri": "my-vault://foo?path=hello" }'

# Retrieve and decrypt the secret
drpcli machines get Name:my-server param ipmi/password --decode

The URI format is: <plugin-name>://<key-to-lookup>?path=<path-to-secret>

• <plugin-name> is the plugin instance's Name (NOT the provider name, which is always vault). With Name: my-vault, use my-vault://.
• <key-to-lookup> is the key name within the Vault secret (e.g., foo for a secret with key foo).
• path=<path> is the path to the secret in the KV store.

Object-shaped params and format=json¶

When the Secure param you are filling has Schema.type: object (for example openshift/kubeconfig), append &format=json to the LookupUri so the plugin parses the stored value as a structured object before returning it. Without format=json, the plugin returns the value as a string, and template expansions that expect a structure (e.g., {{.ParamExpand "openshift/kubeconfig" | toJson}}) will double-encode and break downstream consumers.

drpcli profiles set ocp1 param openshift/kubeconfig to \
  '{ "LookupUri": "my-vault://kubeconfig?path=drp/ocp1-kubeconfig&format=json" }'

The same format=json query parameter is honored on the encrypt path: a value written with format=json round-trips through encrypt + decrypt byte-identically.

Writing Secrets¶

The plugin exposes an encrypt action symmetric to decrypt: it writes a value back to Vault through the same URI shape used for reads.

Action surface:

The URI honors the same fields as the read path: scheme matches plugin Name (the dispatch rule applies), host+path is the key, path= is the Vault secret path, and format=json triggers JSON marshaling. The optional prefix= query parameter overrides the configured KV mount for a single write.

# Plain string write
drpcli machines runaction <uuid> encrypt \
  encrypt/lookup-uri "my-vault://ipmi-password?path=servers/prod" \
  encrypt/value "s3cret"

# JSON-shaped write (symmetric with the read path's format=json).
# encrypt/value must be valid JSON; the structure below is abbreviated
# but well-formed -- substitute your real kubeconfig contents.
drpcli machines runaction <uuid> encrypt \
  encrypt/lookup-uri "my-vault://kubeconfig?path=drp/ocp1-kubeconfig&format=json" \
  encrypt/value '{"clusters":[{"name":"ocp1"}],"contexts":[{"name":"ocp1"}]}'

KV v1 uses a flat write: the target path is <mount>/<vault-path> and the body is {<key>: <value>}.

KV v2 uses a read-merge-write so existing keys at the same Vault path are preserved. If the pre-write read fails, the write is aborted (returns HTTP 500 to the caller) rather than overwriting the path with an empty data map and destroying sibling keys.

Value handling: with format=json, the value is JSON-marshaled before storage, symmetric with the read path's unmarshal. Without format=json, strings pass through unchanged; non-string types fall back to JSON-marshaling so structured values still survive the round-trip with the read path's string assertion.

Cache invalidation: a successful write invalidates the writing machine's cached entries (ClearMachineCache(machine.Uuid)), so the next decrypt from that machine reads the freshly-written value from Vault rather than returning a stale cache hit.

vault/token-short-lived is honored on the write path the same way as on read: when enabled, the token is re-read from DRP before every write.

Error codes: HTTP 400 on invalid URI or value-marshaling failure; HTTP 500 on Vault-side write failure or (for KV v2) a failed pre-write read.

KV Version Path Formatting¶

Path formatting differs between KV versions:

• KV v2: Omit the mount prefix from the path. For a secret at secret/data/servers/prod, use path=servers/prod.
• KV v1: Use the full path including the mount. For a secret at kv/servers/prod, use path=kv/servers/prod.

To determine which version your Vault KV engine uses:

vault secrets list -detailed
# Look for "version:2" in the Options column

Short-Lived Tokens¶

When vault/token-short-lived is true, the plugin re-reads the vault/token parameter from DRP on every secret lookup instead of caching it. This is useful when the token itself is managed by another DRP secret store plugin (e.g., the token is stored in cmdvault and rotated frequently). See Token Auto-Renewal for the interaction with non-token auth methods.

Template Expansion¶

The URI supports DRP parameter template expansion for machine-specific lookups:

my-vault://password?path=servers/{{.Machine.Name}}

Plugin Actions¶

The vault plugin exposes four operator-callable actions:

Invocation:

# Empty the whole cache after rotating a Vault token externally
drpcli system runaction clearCache --plugin my-vault

# Clear one machine's cache
drpcli machines runaction <uuid> clearMachineCache

clearCache and clearMachineCache do not contact Vault; they only invalidate the in-memory plugin cache so the next read pulls fresh from Vault.

Caching¶

To reduce Vault API calls, the plugin caches secret reads in memory. Cache behavior:

• Key: (machineId, lookupUri). Two different machines with the same LookupUri have separate cache entries; the same machine with two different LookupUris has two entries.
• Expiry: lazy. An entry is kept until its vault/cache-timeout interval has elapsed since the cached read; the next read after expiry triggers a fresh Vault fetch. There is no proactive background refresh.
• Single-flight deduplication: concurrent reads for the same key collapse to a single Vault call. The non-leader callers wait on the leader's result.
• Encrypt invalidation: a successful encrypt call from a given machine clears that machine's entire cache so the next read returns the freshly-written value rather than a stale hit. It does NOT clear other machines' caches; other readers may still see the pre-write value until their cached entries expire.
• Manual cache invalidation: use the clearCache and clearMachineCache actions when an out-of-band change (e.g., a Vault token rotation via the cmdvault plugin) must be reflected immediately.

Failure Modes¶

The plugin surfaces failures at three distinct layers. Operators monitoring for problems should look at all three.

What the user sees from --decode¶

When a secret is resolved through the auto-decode path -- drpcli ... --decode, {{.ParamExpand}} in a template, or any other place DRP dispatches a LookupUri automatically -- a read that yields no value comes back as null with no error and a zero (success) exit code. This covers:

• Path not found in Vault.
• Sealed Vault.
• Unreachable Vault (network failure).
• The dispatch-rule miss case (URI scheme does not match any plugin instance's Name).

This silent fall-through is specific to the auto-decode path. Calling the action directly with drpcli machines runaction <uuid> decrypt ... does NOT swallow the failure: a missing path or Vault-side error returns a non-zero exit code and a models.Error (see Server log and action errors). Use runaction decrypt when you want a failure to be loud.

Pre-flight checks (vault status, vault kv get <path>) are recommended for production workflows whose machine readiness depends on Vault-backed values.

Server log and action errors¶

Action invocations that fail at the plugin layer return a models.Error with a Code (typically HTTP 400 for input problems, HTTP 500 for Vault-side problems) and a structured Messages list. These appear in the DRP server log. The encrypt action returns 500 on Vault write failure or KV v2 read-before-write failure. The decrypt action's silent-null behavior is intentional for template-friendly fall-through; the action's own machinery DOES log misses at info level in the DRP server log.

Plugin-health signals on the plugin object¶

drpcli plugins show <name> exposes two distinct fields:

• Errors: per-action errors from the most-recent failed Config or action attempt. Cleared on a successful reconfig.
• PluginErrors: terminal background-renewal or re-login failures appended by the plugin (e.g., vault: JWT re-login exhausted (6 attempts): ...). The DRP server does NOT auto-clear these; operators clear them explicitly when resolving the underlying issue. This is the canonical channel for "the plugin is unhealthy in a way that wasn't tied to a specific drpcli call."

When to check which¶

• --decode returns null: check the dispatch rule (Name matches scheme?), then vault status + vault kv get against the expected path, then drpcli plugins show <name> for Errors and PluginErrors.
• --decode or runaction encrypt returns an error: read the message. It will be Vault-side, URI-shape (400), or transport (500).
• Periodic background flake (decrypts work, but the plugin reports unhealthy after some time): check PluginErrors for renewal or re-login exhaustion. Check the token's policy in Vault. For JWT, check that whatever rotates vault/jwt is keeping the param fresh.

Upgrading from prior versions¶

Prior versions of the plugin set AutoStart: true on the plugin provider, which caused DRP to auto-create a Name: vault plugin instance at install time with no parameters. That instance was immediately broken (vault/address is required) and silently captured vault:// URI dispatch.

After upgrading to the current version:

• If you have a broken vault instance with empty parameters, destroy it: drpcli plugins destroy vault.
• If you have a configured vault instance you want to keep, no action needed.
• For new installs, create plugin instances explicitly: drpcli plugins create '{"Name":"my-vault","Provider":"vault","Params":{...}}'. See Plugin Creation for the canonical example.

The auto-create-then-fail behavior was the source of rackn/product-backlog#1418.