Secret Management (Quản lý Secret)Secret Management

Part of the DevOps Roadmap knowledge base.

Overview

Secret management is the discipline of storing, distributing, rotating, and auditing sensitive credentials — API keys, database passwords, TLS certificates, SSH keys, encryption keys, OAuth client secrets, and access tokens. Every non-trivial system depends on dozens of these, and each one is a potential breach vector: a leaked AWS key can cost thousands of dollars in cryptomining within hours, and a leaked database password can expose your entire customer dataset. Secrets are the keys to the kingdom, and unlike a stolen server or a defaced website, a stolen credential often grants silent, authenticated access that looks exactly like legitimate traffic.

The core problem is that secrets naturally want to spread. Developers paste them into .env files, commit them to Git “just for now”, bake them into Docker images, print them in CI logs, hardcode them in Terraform, and share them over Slack. Studies of public GitHub repositories consistently find millions of live credentials committed by accident every year. Once a secret touches Git history, it must be considered compromised — rewriting history does not un-leak it from forks, clones, CI caches, and the automated scrapers that watch every public push within seconds.

For DevOps engineers, secret management sits at the intersection of security and automation: pipelines, containers, and infrastructure-as-code all need credentials at runtime, but none of them should ever contain credentials. Mature setups centralize secrets in a dedicated store (Vault, a cloud secret manager), inject them at runtime with short lifetimes, rotate them automatically, enforce least privilege per workload, and continuously scan for leaks. The mental model to internalize: treat every secret as a live, dangerous thing with a definite owner, a short lifetime, and a full audit trail — not as a static string you configure once and forget.

Why this matters more than it used to

• Attack surface exploded. Microservices, ephemeral containers, and multi-cloud mean hundreds of workloads each needing credentials, versus a handful of long-lived servers a decade ago.
• Automation reads secrets constantly. CI/CD, GitOps controllers, and autoscalers fetch credentials thousands of times a day. Any one of those paths is an exfiltration route.
• Supply-chain attacks target secrets. Compromised npm/PyPI packages and malicious GitHub Actions routinely exfiltrate environment variables and ~/.aws/credentials.
• Compliance demands it. SOC 2, ISO 27001, PCI-DSS, and HIPAA all require access control, rotation, and audit logging for credentials — exactly what a secret manager provides.

Fundamentals

Why secrets don’t belong in code or env files

• Git history is forever. A secret committed once survives in history, forks, mirrors, and CI caches even after deletion. Attackers run automated scanners against every public push within seconds; even private repos leak through clones, backups, and misconfigured access.
• Env files spread uncontrolled. .env files get copied between machines, attached to tickets, emailed, included in backups, and mounted into containers. There is no audit trail of who read them and no way to revoke a single reader.
• No rotation. Hardcoded secrets are effectively permanent, because rotating them requires a code change and redeploy — so in practice they never rotate, and a credential leaked in 2019 is often still valid today.
• No least privilege. A config file is all-or-nothing: anyone who can read the repo or the host reads every secret in it. There is no way to grant “read only the database password to this one service”.
• Blast radius. One compromised laptop, CI runner, or container image exposes every secret it ever held, across every environment those secrets touch.
• Secrets end up in logs and telemetry. Env vars leak through crash dumps, /proc/<pid>/environ, error-tracking tools (Sentry), and printenv in a debug endpoint.

The goal: secrets should be centralized, encrypted at rest and in transit, access-controlled, audited, short-lived, and injected at runtime — never stored alongside the code that uses them, and never persisted anywhere a human can casually read them.

The secret zero problem

Every distribution scheme faces a chicken-and-egg question: to fetch a secret, a workload must first authenticate — but with what? If the answer is “another secret”, you have merely moved the problem. This is the secret zero (or bootstrap) problem. The modern answer is workload identity: the platform vouches for the workload’s identity (a Kubernetes service account token, an AWS instance role, a GCP metadata-server identity, an OIDC token from your CI), and the secret store trusts that platform-issued identity instead of a pre-shared password. Secret zero becomes something the platform manages and rotates for you, not something you store.

The secret lifecycle

1. Creation — generated with sufficient entropy, ideally by the secret store itself so the plaintext never exists on a developer’s machine.
2. Storage — encrypted at rest in a dedicated system, not in Git/wikis/chat/ticketing systems.
3. Distribution — delivered to workloads at runtime via authenticated identity (not another shared secret — the secret zero problem).
4. Usage — read into memory at the last possible moment, never written to disk in plaintext, never logged.
5. Rotation — replaced on a schedule or on demand, ideally automatically and without downtime.
6. Revocation — invalidated immediately on compromise or when a lease expires.
7. Audit — every read and write logged with who/what/when/where, and alertable on anomalies.

Static vs dynamic secrets

Dynamic secrets are the single biggest lever in secret management: a database credential with a 1-hour lease turns a breach from a catastrophe into a nuisance, and gives you per-consumer attribution for free.

HashiCorp Vault

Vault is the de facto standard self-hosted / cloud-agnostic secret manager. Key architectural pieces:

• Storage backend — encrypted storage (integrated Raft storage is the common modern choice; Consul was traditional) holding all data. Vault encrypts everything with its own barrier key before it touches the backend, so the backend operator never sees plaintext.
• Seal/unseal — Vault starts sealed and can decrypt nothing. The master (root) key that decrypts the barrier is itself split via Shamir’s Secret Sharing into key shares given to different trusted operators (e.g., 3-of-5 needed to unseal), or Vault is auto-unsealed via a cloud KMS/HSM (recommended for production, since it removes the manual quorum ceremony on every restart).
• Auth methods — how clients prove identity: Kubernetes service accounts, AWS/Azure/GCP IAM, OIDC/JWT, AppRole (for machines/CI), TLS certs, LDAP, GitHub, userpass. Authentication yields a token with attached policies and a TTL.
• Secrets engines — pluggable backends mounted at paths:
  • kv — static key-value; v2 adds versioning, soft-delete, and check-and-set.
  • database — dynamic DB credentials (Postgres, MySQL, MongoDB, MSSQL, and more).
  • aws / azure / gcp — dynamic cloud credentials (temporary IAM users/roles/STS).
  • pki — an internal certificate authority issuing short-lived TLS certs.
  • transit — encryption-as-a-service: apps send plaintext, Vault returns ciphertext; the key never leaves Vault (“encryption without exposing keys”). Also does signing, HMAC, and rewrapping.
  • ssh — signed SSH certificates / one-time passwords for host access.
  • totp — generates and validates TOTP codes.
• Policies — path-based ACLs (HCL) granting create/read/update/delete/list on secret paths, with support for templating ({{identity.entity.name}}) so one policy scopes each caller to its own namespace.
• Dynamic secrets and leases — Vault generates credentials on demand, each attached to a lease with a TTL and max_ttl. Clients renew leases while in use; when the lease expires or is revoked, Vault runs the engine’s revoke logic (e.g., DROP USER) and the credential is gone. Revoking a lease (or all leases under a prefix) is a first-class operation for incident response.
• Namespaces (Enterprise) — multi-tenant isolation; Performance/DR replication for HA across regions.

# Static KV (v2) — versioned key-value
vault kv put secret/myapp/db password='s3cr3t' username='app'
vault kv get -field=password secret/myapp/db
vault kv get -version=3 secret/myapp/db      # recover a prior version

# Dynamic database credentials — new user per request, auto-expiring
vault read database/creds/readonly-role
# Key                Value
# lease_id           database/creds/readonly-role/abc123
# lease_duration     1h
# username           v-token-readonly-x7Kq...   (created just now)
# password           A1a-9nB2...                (dies when the lease ends)

# Encryption-as-a-service: the app never sees the key
vault write transit/encrypt/orders plaintext=$(base64 <<< "4111-1111-1111-1111")

# Incident response: revoke everything a compromised role ever issued
vault lease revoke -prefix database/creds/readonly-role

A HashiCorp Vault mental model: Vault is a broker, not a filing cabinet. Its highest value is not storing static strings (cloud managers do that too) but minting short-lived, per-consumer credentials and cryptographic operations on demand, so long-lived secrets stop existing in your fleet at all.

Cloud secret managers

Cloud managers win on operational simplicity (no cluster to run, no seal/unseal ceremony), native IAM integration (workload identity is built in), and pay-as-you-go. Vault wins on dynamic secrets across many systems, multi-cloud portability (one API regardless of provider), and advanced engines (transit encryption, internal PKI, SSH CA). Many organizations run both: a cloud manager for cloud-native static secrets, Vault for dynamic credentials and cross-cutting crypto.

KMS vs secret managers — a common confusion

A Key Management Service (AWS KMS, Cloud KMS, Azure Key Vault keys) manages encryption keys and performs crypto operations; the key material never leaves the KMS. A secret manager stores and distributes arbitrary secret values (and usually uses a KMS under the hood to encrypt them at rest). Rule of thumb: use a KMS when you need to encrypt/decrypt/sign data and never expose the key; use a secret manager when a workload needs the actual secret value (a password, an API token). Vault’s transit engine is KMS-like functionality inside Vault.

Key Concepts

Kubernetes Secrets and their limitations

Native K8s Secret objects are only base64-encoded, not encrypted by default — base64 is an encoding, not a cipher, and base64 -d reverses it instantly. They are stored in etcd, in plaintext unless you enable encryption at rest with an EncryptionConfiguration (and even then, prefer a KMS provider over the static aescbc key, whose key otherwise sits on the API server disk). Beyond storage, the real exposure surfaces are:

• Anyone with a broad RBAC role (get/list on Secrets) reads them — and many default roles are too broad.
• Anyone with node access reads secrets mounted into pods on that node (/var/run/secrets/...).
• Secrets committed in Git-managed manifests are just plaintext in Git — the GitOps anti-pattern.
• Secrets injected as environment variables leak via crash dumps, /proc, and child processes.

Prefer mounting secrets as files (tmpfs volumes) over env vars, scope RBAC tightly, and use one of the patterns below to keep plaintext out of Git.

# External Secrets Operator example
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: myapp-db
spec:
  refreshInterval: 1h                 # re-sync so rotations propagate
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: myapp-db                    # resulting K8s Secret name
    creationPolicy: Owner
  data:
    - secretKey: password             # key in the K8s Secret
      remoteRef:
        key: secret/myapp/db          # path in Vault
        property: password

Decision guide: No external store and just want secrets out of Git → Sealed Secrets or SOPS. Already run Vault/cloud secret manager and want K8s to mirror it → ESO or Vault/Secrets CSI. Need dynamic (leased) secrets in pods → Vault (Injector/VSO/CSI). Want to never persist to etcd → CSI driver with file mounts.

Example flow: Vault + Kubernetes auth

1. Enable the Kubernetes auth method in Vault and configure it with the cluster’s API server address and CA (Vault uses these to validate tokens).
2. Bind a Vault role to a specific namespace + service account, attaching a policy and a TTL.
3. The pod (via Vault Agent, VSO, or the app directly) sends its projected service account JWT to Vault’s login endpoint.
4. Vault validates the JWT against the Kubernetes TokenReview API — the cluster itself vouches that the token belongs to that pod’s service account.
5. Vault returns a short-lived Vault token scoped to the role’s policy.
6. The workload reads its static secrets, or requests dynamic DB credentials and receives a leased username/password.
7. Vault Agent/VSO renews the token and any leases in the background; on pod termination, leases are revoked.

vault auth enable kubernetes
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

vault write auth/kubernetes/role/myapp \
    bound_service_account_names=myapp \
    bound_service_account_namespaces=prod \
    policies=myapp-read \
    ttl=1h

No secret zero: the pod’s identity (its Kubernetes-signed service account token, which Kubernetes rotates automatically) is the credential. Nothing long-lived was pre-shared.

Detecting leaked secrets

• gitleaks — fast regex + entropy scanner for repos, full history, and pre-commit hooks. Configurable rules and allowlists: gitleaks git -v . or gitleaks dir ./config.
• trufflehog — scans Git history, S3 buckets, Docker images, filesystems, and more; its killer feature is verification — it actively checks whether a found credential is live against the provider’s API, cutting false positives dramatically: trufflehog git file://. --results=verified.
• detect-secrets (Yelp) — baseline-oriented scanner designed to be adopted on existing repos without drowning you in historical findings.
• GitHub secret scanning + push protection — GitHub matches known credential patterns from 100+ partners; push protection blocks the push before the secret ever lands. Enable both org-wide; partners are auto-notified of verified leaks of their tokens.
• Cloud-native scanners — AWS also flags exposed access keys, and providers like GitGuardian offer continuous org-wide monitoring.

Run scanners in three places: a pre-commit hook (catch it before it leaves the laptop), the CI pipeline (enforce so nothing merges), and scheduled full-history scans (catch old leaks and new detector rules). Defense in depth, because humans will slip past any single gate.

If a secret leaks — rotate first, then clean up. The order matters:

1. Revoke/rotate the credential immediately. The leaked string is compromised the instant it’s public; a bot may already have it.
2. Audit for malicious use. Check CloudTrail/audit logs for access from unexpected IPs or at odd times.
3. Only then rewrite history (git filter-repo, BFG) — and understand this is hygiene, not remediation. Forks, clones, and caches still hold the old blob, which is exactly why step 1 is what actually protects you.
4. Post-incident: figure out how it leaked and close that path (add a pre-commit hook, tighten CI, fix the logging that printed it).

Rotation and least privilege

• Rotation limits the useful lifetime of a stolen credential and is the difference between “a bad day” and “a breach”. Prefer automatic rotation (AWS Secrets Manager Lambda rotation, Vault dynamic secrets, cloud-managed identities) over calendar reminders — manual rotation reliably does not happen. Design apps to re-read secrets on a refresh interval or on a signal, so rotation is a non-event rather than a redeploy. For zero-downtime rotation of static secrets, use the two-secret (dual-credential) pattern: keep the previous credential valid while the new one propagates, then retire the old one.
• Least privilege: each workload gets its own identity and can read only its own paths. In Vault this is per-path, per-role policies (ideally templated to the caller’s identity); in AWS, per-secret IAM/resource policies. Avoid a single “app secrets” bucket that every service can read — that turns any one compromise into a total compromise.
• Prefer workload identity over static keys everywhere: IRSA / EKS Pod Identity on AWS, Workload Identity on GKE, Managed Identities on Azure, and OIDC federation for CI (GitHub Actions / GitLab → AWS/GCP/Azure with no long-lived cloud keys stored in the CI system at all). Long-lived static cloud keys should be the rare, well-justified exception.
• Scope, environment-separate, and expire. Separate secrets per environment (never share prod secrets into staging). Give every credential the narrowest scope and the shortest TTL that still works.

Encryption in transit and at rest

Secrets must be encrypted both at rest (in the store’s backend, in etcd, in backups) and in transit (TLS everywhere — client to Vault, Vault to database, ESO to cloud API). A secret manager that encrypts at rest but is queried over plaintext HTTP is only half-protected. Envelope encryption (a data key encrypts the secret, a KMS root key encrypts the data key) is the standard pattern that lets you rotate the root key without re-encrypting every secret.

Best Practices

1. Never commit secrets — and enforce it with tooling, not trust. Pre-commit gitleaks hooks + CI scanning + GitHub push protection. Humans will slip; layered tooling must catch it. A policy that relies on developers “being careful” is not a control.
2. Centralize in a dedicated store. One system of record (Vault or a cloud secret manager) with encryption, ACLs, versioning, and audit logs — not .env files, wikis, ticketing systems, or chat. Fragmentation is the enemy of control.
3. Prefer dynamic, short-lived credentials. A database or cloud credential with a 1-hour lease turns a breach from a catastrophe into a nuisance and gives you per-consumer attribution for free.
4. Use workload identity, not shared bootstrap secrets. Kubernetes service accounts, cloud IAM roles, and OIDC federation solve the secret zero problem — the platform proves identity so you don’t store a bootstrap password.
5. Automate rotation and design apps to tolerate it. Manual rotation doesn’t happen. Wire rotation into the store and make apps re-read secrets (or use the dual-credential pattern) so rotation causes no downtime.
6. Apply least privilege per workload. One identity, one policy, one narrow path per app; never a shared “everything” bucket. Audit policies regularly for scope creep and revoke access on offboarding.
7. Encrypt Kubernetes secrets at rest and restrict RBAC. Enable etcd encryption with a KMS provider, mount secrets as files rather than env vars, and tightly limit who can get/list Secrets cluster-wide.
8. Keep secrets out of images, logs, and CI output. Never ENV/ARG a secret in a Dockerfile (it persists in image layers); mask secrets in CI; beware set -x, verbose error messages, and error-tracking tools that capture environment.
9. Encrypt in transit, always. TLS from client to store and from store to backing systems. At-rest encryption is meaningless if the value crosses the network in plaintext.
10. Audit everything and alert on anomalies. Enable Vault audit devices / CloudTrail / Key Vault diagnostics, and alert on the unusual — a CI token reading production paths at 3 a.m., a spike in reads, access from a new IP.
11. Treat any exposed secret as compromised — rotate first. Don’t rely on “it was only public for a minute” or a history rewrite. Revoke immediately, then investigate and clean up.
12. Version secrets and plan for rollback. KV v2 / cloud secret versions let you recover instantly from a bad rotation instead of scrambling to reconstruct the old value.
13. Have a break-glass procedure. Document how humans access critical secrets when the automation is down — gated by MFA, requiring approvals, and reviewed after the fact. Test it before you need it.
14. Separate secrets per environment and per tenant. Prod, staging, and dev must not share credentials; a leak in a low-trust environment must never grant prod access.
15. Minimize the number of humans who ever see plaintext. Prefer store-generated secrets so the plaintext never exists on a laptop; use approvals and time-boxed access for the rare cases a human must read one.
16. Give every secret a documented owner and expiry. Ownerless, never-expiring secrets accumulate into an unmanageable liability. Inventory them and retire the unused.
17. Back up and DR-test your secret store. A secret manager is a single point of failure; run it HA, back up its storage and unseal/root keys securely, and rehearse recovery.

References

• roadmap.sh — DevOps Roadmap
• HashiCorp Vault documentation
• Vault concepts: seal/unseal · leases
• Vault dynamic secrets tutorial
• Vault Kubernetes auth method
• Vault transit (encryption-as-a-service) engine
• AWS Secrets Manager documentation
• AWS Secrets Manager rotation
• AWS Systems Manager Parameter Store
• Azure Key Vault documentation
• GCP Secret Manager documentation
• Kubernetes — Secrets · Encrypting Secret Data at Rest
• External Secrets Operator
• Bitnami Sealed Secrets
• SOPS
• Secrets Store CSI Driver
• gitleaks · trufflehog · detect-secrets
• GitHub secret scanning & push protection
• Configuring OpenID Connect in cloud providers (GitHub Actions)
• OWASP Secrets Management Cheat Sheet