Secret Management

8.1 Secret Management Architecture Principles

Kubernetes native Secrets are insufficient as a standalone enterprise secret management solution for four interconnected reasons.

First, storage : Kubernetes Secrets are stored in etcd as base64-encoded values — base64 is a transport encoding, not encryption. Without explicit at-rest encryption configuration, any entity with direct etcd access or the Kubernetes API get secret permission can read plaintext secret values.

Second, rotation : Kubernetes has no native secret rotation mechanism — credentials must be manually updated and consuming pods must be restarted to pick up new values.

Third, access control : the only access control primitive is Kubernetes RBAC, which is coarse-grained — a service account with get secret rights in a namespace can read all secrets in that namespace, with no support for time-bounded access, conditional grants, or per-secret scoping below the namespace level.

Fourth, auditability : Kubernetes API audit logging records API calls at the request level but does not provide per-access audit trails at the secret value level — who read which secret, when, and why.

Enterprise-grade solutions address all four. Portainer's boundary: Portainer holds platform integration credentials for its own operations — registry credentials, Git repository credentials, and IdP client secrets. These are protected by Portainer's own credential model (AES-256-GCM database encryption, write-only access, bcrypt for internal passwords) as described in Chapters 2, 4, 5, and 6.

Application secrets — database passwords, API keys, TLS certificates consumed by workloads — are outside Portainer's direct scope. Portainer surfaces Kubernetes Secrets through the management UI (with visibility controlled by RestrictSecrets , Chapter 4) but does not generate, rotate, or govern them. The patterns in this chapter describe how application secrets should be sourced, protected, and delivered to workloads in the environments Portainer manages.

Secrets never touch Git: A governing constraint for all secret management patterns in GitOps-integrated environments: no secret value should appear in a Git repository, whether as plaintext, base64-encoded content, or any reversible encoding. GitOps pipelines (Chapter 6) manage application manifests, Helm values, and platform configuration as versioned Git content; secret values must not be part of that content. The correct model is that Git holds only references — ExternalSecret CRDs pointing to paths in the secrets store, or SealedSecret resources containing asymmetric ciphertext that requires the cluster's private key to decrypt. The secrets store is the only system holding actual secret material. This constraint applies to Portainer's GitOps stacks as well as application manifests — stack definitions committed to Git must not contain inline credential values. A committed credential, even if rotated immediately after discovery, creates a permanent record in Git history that represents an ongoing disclosure risk for the lifetime of the repository.

8.2 Secret Management Patterns

Five patterns address these four dimensions with different trade-offs. The table below maps coverage before the detailed sections that follow — use it to identify which pattern matches your environment tier and compliance requirement, then read the corresponding section for integration and configuration guidance.

💡

† ESO is a synchronisation layer — storage, rotation, and auditability coverage depend entirely on the backend chosen. ESO itself provides only Kubernetes API-level audit for the sync operation.

Native Kubernetes Secrets

Kubernetes Secrets are API objects stored in etcd. Values are base64-encoded in transit and at rest by default — but base64 is not encryption. Any principal with direct etcd access or with Kubernetes RBAC permission to get secrets in a namespace can decode the value in one step. In the default cluster configuration, Kubernetes Secrets represent a significant risk if etcd is compromised or if RBAC assignments are overly permissive.

Encryption at rest via EncryptionConfiguration: Kubernetes supports provider-based encryption at rest applied at the API server. Providers include AESGCM (AES-GCM using a key stored in the API server configuration file) and external KMS plugins (AWS KMS, Azure Key Vault, GCP Cloud KMS via the KMS v2 plugin). KMS-based encryption is strongly preferred — the encryption key lives in the external KMS, not in the cluster configuration file, preventing a compromised etcd backup from yielding plaintext secrets. Without at-rest encryption, etcd backups must be treated as sensitive data and protected equivalently to the secrets they contain.

Limitations: No rotation mechanism — secrets must be manually updated and pods must be restarted to pick up changes (unless the secret is mounted as a volume without subPath , in which case Kubernetes updates the mounted file within the kubelet sync period). No dynamic generation — every secret is manually created and persisted indefinitely. No per-access audit trail beyond Kubernetes API audit logging. RBAC is the only access control mechanism — no time-bounded grants, no identity-conditional access, no per-secret scoping below the namespace.

When acceptable: Native Kubernetes Secrets are appropriate in development environments and for non-sensitive configuration (public endpoints, feature flags). In production environments handling credentials, tokens, certificates, or PII, native Kubernetes Secrets should be augmented with at-rest encryption and accompanied by one of the external secret management patterns below.

HashiCorp Vault

Licence note — OpenBao as the OSS alternative: HashiCorp moved Vault to the Business Source Licence (BSL) in 2023. BSL is not OSI-approved and may conflict with open-source-only procurement policies or legal frameworks that prohibit non-OSI-approved software. For these environments, OpenBao (Linux Foundation, MPL 2.0) is the drop-in replacement. OpenBao is a fork of Vault at its last MPL 2.0 release, maintaining full API and tooling compatibility — all patterns described in this section (Vault Secrets Operator, ESO with Vault backend, Kubernetes auth method, dynamic secrets, PKI engine) apply to OpenBao without modification. OpenBao includes features previously gated behind Vault Enterprise: namespaces for multi-tenancy, performance standbys for horizontal read scalability, and integrated Raft storage for HA without an external Consul dependency. Where Vault is referenced in this section, substitute OpenBao for environments with OSS-only constraints.

HashiCorp Vault is a purpose-built secret management service providing centralised storage, dynamic generation, access control, rotation, and per-access audit logging. It is the reference pattern for regulated environments with complex credential requirements.

Architecture: Vault organises functionality around auth methods (how clients prove their identity to Vault), secret engines (how secrets are stored or generated), and leases (every secret has a TTL; dynamic secrets are automatically revoked when the lease expires and not renewed). The KV v2 engine stores static secrets with versioning; the database engine generates dynamic credentials on demand; the PKI engine issues and renews TLS certificates; the transit engine provides encryption-as-a-service without exposing the key.

Dynamic secrets: Vault's most significant capability over static secret stores. Rather than storing a long-lived database password, Vault generates a unique credential on demand with a configurable TTL — the application receives a credential that is automatically revoked after the lease period expires. If a credential is compromised, its exposure window is bounded by the TTL. Long-lived credentials shared across services are eliminated from the runtime environment.

Kubernetes integration patterns — three options:

Vault Agent Sidecar (Vault Injector): A mutating webhook injects a Vault Agent container into application pods at admission time. The Agent authenticates to Vault using the pod's Kubernetes service account token, fetches the required secrets, writes them to a shared in-memory volume, and renews leases before expiry. The application reads secrets from the filesystem. No application code changes are required. The cost is an additional container per pod and dependency on the mutating webhook.

Vault Secrets Operator (VSO): A Kubernetes operator that manages VaultStaticSecret and VaultDynamicSecret CRDs. VSO authenticates to Vault, fetches secrets, and synchronises them into Kubernetes Secret objects that applications consume natively. VSO is operationally simpler than the sidecar pattern — no per-pod sidecar, standard Kubernetes Secret consumption. This is the recommended pattern for new Vault deployments.

Secrets Store CSI Driver with Vault Provider: Mounts secrets directly into pod filesystems as ephemeral CSI volumes without creating Kubernetes Secret objects. Secrets exist only in pod memory as mounted files — they are never written to etcd. The strongest isolation pattern, but operationally more complex to manage at fleet scale.

Authentication to Vault: The Kubernetes auth method is the recommended pattern for in-cluster workloads — Vault validates the pod's projected service account token against the Kubernetes API server, confirming pod identity without a static credential. AppRole (role_id + secret_id) is appropriate for external systems that cannot use the Kubernetes auth method. OIDC integration delegates human operator authentication to the enterprise IdP.

When to use Vault: Regulated environments (PCI-DSS, HIPAA, FedRAMP) with explicit requirements for dynamic secrets, per-access audit logging at the value level, and automated credential rotation. High-complexity environments with diverse secret types (database credentials, PKI, cloud IAM credentials, encryption services). Vault requires dedicated operational capacity to install, configure, harden, and operate — it is appropriate where that investment is justified by the security and compliance requirements.

External Secrets Operator (ESO)

External Secrets Operator is a Kubernetes operator that acts as a synchronisation bridge between external secret management backends and Kubernetes native secrets. ESO is widely adopted across cloud-native environments: teams running EKS with AWS Secrets Manager, AKS with Azure Key Vault, GKE with GCP Secret Manager, and Vault deployments of all sizes use ESO to decouple secret storage and access control from workload consumption. GitOps platforms such as Argo CD and Flux pair naturally with ESO — application manifests in Git reference ExternalSecret resources rather than raw Secret values, keeping the repository free of credential material while the operator handles synchronisation at runtime. ESO allows applications to consume secrets as standard Kubernetes Secret objects while the authoritative storage and access control live in an external backend.

Architecture: ESO defines three CRDs. A SecretStore (namespace-scoped) defines connection parameters and authentication credentials for a specific external backend — for example, a specific Azure Key Vault or Vault path accessible to that namespace. A ClusterSecretStore (cluster-scoped) provides the same backend connection available to all namespaces. An ExternalSecret defines which keys to fetch from the backend, how to map them into a Kubernetes Secret object, and the refresh interval on which ESO re-fetches the current value. When the backend secret is rotated, ESO propagates the updated value to the Kubernetes Secret on the next refresh cycle, providing a rotation propagation mechanism without manual Secret updates.

Backend integrations: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, HashiCorp Vault, IBM Secrets Manager, 1Password, and others. ESO provides a consistent ExternalSecret / SecretStore interface regardless of backend, allowing organisations to change or add backends without modifying application configurations or consuming Deployment manifests.

SecretStore vs ClusterSecretStore : Prefer namespace-scoped SecretStore over ClusterSecretStore wherever possible. A ClusterSecretStore grants all namespaces access to the configured backend path, which violates least-privilege scoping. Use ClusterSecretStore only when a genuinely shared credential (e.g., a cluster-wide TLS certificate) must be available across all namespaces.

When to use ESO: Cloud-hosted environments where secrets already live in a cloud-native backend. Organisations that want native Kubernetes Secret consumption for application compatibility while using an enterprise backend for storage, rotation, and audit. Multi-cloud or multi-backend environments where a single CRD interface reduces operational complexity. ESO can also sit in front of Vault, providing the CRD-based interface while Vault provides the backend governance — this combination is common in hybrid environments.

Sealed Secrets

Bitnami Sealed Secrets provides asymmetric encryption of Kubernetes Secret manifests, making encrypted secrets safe to store in Git repositories — the foundational requirement for GitOps-first secret management.

Architecture: A controller running in the cluster generates an RSA key pair on installation and exposes the public key. The kubeseal CLI encrypts a Kubernetes Secret manifest using the cluster's public key, producing a SealedSecret resource containing the ciphertext. Only the in-cluster controller — which holds the private key — can decrypt a SealedSecret back into a Kubernetes Secret. Ciphertext is cluster-bound and namespace-bound by default: a SealedSecret sealed for one cluster cannot be decrypted by another, and cannot be re-used in a different namespace without re-sealing.

GitOps compatibility: SealedSecret manifests contain ciphertext that is safe to commit to a Git repository alongside other Kubernetes manifests. Unlike raw Kubernetes Secret manifests (which contain base64 plaintext), a SealedSecret in Git cannot be decoded without the cluster's private key. This makes Sealed Secrets the natural fit for GitOps pipelines (Chapter 6) where all cluster state — including secrets — must live in the repository. The GitOps pipeline applies the SealedSecret manifest to the cluster, the controller decrypts it into a native Kubernetes Secret, and the workload consumes it normally.

Limitations: No rotation mechanism — when the underlying credential changes, it must be manually re-sealed with kubeseal and the updated SealedSecret committed to Git. No centralised access control — once decrypted into a Kubernetes Secret, access is governed only by Kubernetes RBAC. No per-access audit trail. Controller private key rotation requires re-sealing all existing SealedSecret resources in the cluster — a significant operational event that must be planned and scripted.

When to use: GitOps-first teams that need all cluster state — including secrets — stored in Git. Simpler environments where secret rotation frequency is low. Development, staging, or edge environments where the operational overhead of Vault or ESO is not warranted. As a stepping stone toward a more complete secret management posture.

Cloud-Provider Secret Stores (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager)

AWS Secrets Manager, Azure Key Vault, and GCP Secret Manager are managed secret storage services that provide at-rest encryption, versioning, access auditing, and automated rotation as platform capabilities. This section covers what each service offers natively — authentication, rotation, and audit. The Kubernetes consumption layer (how secrets stored in these backends reach workloads) is handled by ESO as described in the section above; the patterns here describe the backend side of that integration.

Identity-based access: The preferred authentication pattern for cloud-provider secret stores in Kubernetes environments eliminates static credentials entirely. AWS uses IRSA (IAM Roles for Service Accounts) or EKS Pod Identity — the pod's Kubernetes service account is bound to an IAM role; AWS STS issues temporary credentials for that role. Azure uses Workload Identity — a Kubernetes service account is federated to a Managed Identity via a federated credential, and the pod receives an Azure AD token without a stored secret. GCP uses Workload Identity — a Kubernetes service account is mapped to a GCP service account, and the pod authenticates to GCP APIs using that identity. In all three cases, the ESO operator (or the CSI driver) authenticates to the cloud secret store using the pod or operator's workload identity — no static API key or service account JSON file is needed in the cluster.

Rotation: AWS Secrets Manager natively rotates supported secret types (RDS database credentials, Redshift, DocumentDB) using Lambda-based rotation functions. Azure Key Vault provides rotation policies for supported secret and certificate types. GCP Secret Manager does not rotate secrets natively but integrates with Secret Manager Notifications to trigger rotation workflows. When consumed via ESO, backend rotation is propagated to the Kubernetes Secret on the configured ExternalSecret refresh interval — see the ESO section above for rotation propagation behaviour.

8.3 Secrets in Portainer's Platform Integrations

Portainer holds three categories of platform credentials for its own operations. Registry credentials (Chapter 5) are stored in Portainer's AES-256-GCM encrypted database under the write-only model and injected just-in-time at deployment time. Git repository credentials (Chapter 6) are stored under the same model and used by Portainer's GitOps engine at reconciliation time. IdP client secrets (Chapter 4) are stored encrypted in Portainer's database and used for OIDC/SAML authentication flows. All three are protected by the write-only model — they cannot be read back through the Portainer UI or API — and are subject to Portainer's AES-256-GCM database encryption when enabled.

The primary integration point between Portainer and an enterprise secret store is Portainer's database encryption key . Portainer's BoltDB database can be encrypted at rest using a key provided via a file mounted into the container. On Kubernetes, this is a Kubernetes Secret mounted as a volume — VSO or ESO can manage this Secret directly, ensuring the encryption key is governed by enterprise secret store policies and audit logging. This is the documented path for integrating Portainer's credential store with an enterprise secrets platform.

The initial admin password can also be sourced externally at setup time via --admin-password-file , which works with Docker Secrets on Swarm and Kubernetes Secret volume mounts on Kubernetes — both of which can be populated from an enterprise secret store via VSO or ESO.

Portainer's application-level credentials — registry credentials, Git repository credentials, and IdP client secrets — do not have an equivalent external-reference mechanism. These are configured through the Portainer UI and stored in the encrypted database; there is no supported path for Portainer to pull them from Vault, ESO, or any other external secret store at runtime. The correct posture is to protect these credentials by ensuring database encryption is enabled with a key managed through the enterprise secret store, and to rotate them through the Portainer UI or API on the organisation's standard credential rotation schedule. Portainer's write-only model means rotation requires overwriting the credential through the UI or API — the previous value is not retrievable and does not need to be explicitly revoked within Portainer.

8.4 Non-Functional Considerations

Secret rotation and workload impact: The rotation strategy must account for how the application consumes the secret. For Vault Agent Sidecar, the Agent renews leases automatically before expiry — the application sees a continuously valid credential with no restart required. For VSO and ESO, the Kubernetes Secret object is updated when the backend value changes; if the pod mounts the secret as an environment variable , it must be restarted to pick up the new value (environment variables are set at container start and do not update dynamically); if the secret is mounted as a volume file , Kubernetes updates the file within the kubelet sync period (typically 1–2 minutes) without a pod restart. Design the rotation strategy around the consumption pattern — volume mounting is required for zero-downtime rotation. For Sealed Secrets, rotation requires re-sealing, committing to Git, and reconciliation. For native Kubernetes Secrets with manual rotation, a rolling restart strategy must be defined and automated.

Audit logging: Secret access audit coverage varies significantly by pattern and must be matched to the compliance requirement. HashiCorp Vault provides per-access audit logging at the secret value level — every read of a Vault path is recorded with the entity, timestamp, method, and path. Cloud-provider secret stores (AWS Secrets Manager, Azure Key Vault, GCP Secret Manager) provide per-access audit events in CloudTrail, Azure Monitor / Activity Log, and Cloud Audit Logs respectively. ESO and native Kubernetes Secrets provide API-level audit logging only — the get secret API call is logged if Kubernetes audit logging is enabled, but individual pod reads of mounted secrets are not. For PCI-DSS and HIPAA environments, per-access audit at the secret value level is typically required — Vault or a cloud-provider backend is the minimum compliant pattern.

Least-privilege access: Apply minimum-privilege scoping at every level. For Vault: policies must be scoped to the minimum required secret paths; Kubernetes auth roles must bind to specific service accounts in specific namespaces, not to all service accounts cluster-wide. For ESO: prefer namespace-scoped SecretStore over ClusterSecretStore ; the ESO operator's own service account must be scoped to minimum required backend paths. For native Kubernetes Secrets: RBAC must grant get on specific named Secrets rather than list / get on all secrets in a namespace — list on Secrets exposes all secret names and enables bulk enumeration; get on a specific named Secret limits exposure to that value only.

Compliance alignment: PCI-DSS requires encryption of cardholder data at rest and in transit, strict access controls, and audit logging of access to sensitive data — Vault or a cloud-provider backend with audit logging is the minimum compliant pattern; native Kubernetes Secrets with EncryptionConfiguration is marginal and requires compensating controls. HIPAA requires equivalent protections for PHI, including access audit trails and minimum necessary access enforcement. SOC 2 Type II requires evidence of access controls and audit logging presentable during audit. Match the secret management pattern to the compliance requirements of the workloads in each environment tier — a development cluster running no regulated data may use native secrets; a production cluster processing payment data must use Vault or a cloud-provider backend with full audit logging.

Scenarios

Supplemental — ESO + AWS Secrets Manager (AWS variant of Level 4)

AWS enterprises running EKS follow the same ESO pattern using AWS Secrets Manager as the backend. IRSA (IAM Roles for Service Accounts) provides workload-identity-based authentication: the ESO operator's Kubernetes service account is annotated with an IAM role ARN; AWS STS issues temporary credentials for that role, removing the need for any stored IAM access keys in the cluster. SecretStore resources reference the AWS Secrets Manager endpoint in the appropriate region. ExternalSecret resources map Secrets Manager secret paths to Kubernetes Secret keys. AWS Secrets Manager's native rotation support (Lambda-based rotation functions for RDS and other managed service credentials) rotates credentials on schedule; ESO's refresh interval propagates the updated value to the Kubernetes Secret within the configured window. CloudTrail records per-access events for all Secrets Manager API calls and forwards them to the SIEM via the pipeline configured in Chapter 10.