Secrets Management for Platform Teams

The security audit found API keys in three Git repos, two Slack messages, and a shared Google Doc titled "Prod Credentials DO NOT SHARE." Twelve people had access. Nobody remembered who added the Razorpay key or when it was last rotated.

Secrets management is not a tool decision - it is an operational discipline. The tool (AWS Secrets Manager, Vault, SSM Parameter Store) matters less than the rules: no secrets in Git, no secrets in env files on developer laptops, automatic rotation where possible, and audit logs for every access.

The Secrets Hierarchy

Not all secrets are equal. I classify them into three tiers with different storage, access, and rotation policies.

Critical secrets never appear in developer environments. Developers use local .env files with dummy values and connect to staging services for integration testing.

AWS Secrets Manager as the Default

For teams on AWS, Secrets Manager is my default for critical and operational secrets. It integrates with RDS for automatic database password rotation, IAM for access control, and CloudTrail for audit logging.

Developer Laptop          CI (GitHub Actions)         Production (ECS/EKS)
──────────────          ───────────────────         ────────────────────
.env (dummy values)     OIDC → IAM role              Task role → Secrets Manager
                        Secrets Manager read          Inject at startup
                        (test keys only)              (never in env vars on disk)
                            
                        ✗ No secrets in Git
                        ✗ No secrets in build logs
                        ✗ No secrets in Docker layers

The rule is simple: secrets enter the runtime at startup from Secrets Manager, live in memory, and never touch disk or logs.

Injecting Secrets at Runtime

I avoid baking secrets into Docker images or ECS task definitions as plaintext environment variables. Instead, applications fetch secrets at startup from Secrets Manager using the task's IAM role.

// Node.js startup secret loading
import { SecretsManagerClient, GetSecretValueCommand } from "@aws-sdk/client-secrets-manager";

const client = new SecretsManagerClient({ region: "ap-south-1" });

async function loadSecrets() {
  const command = new GetSecretValueCommand({ SecretId: "prod/api/database" });
  const response = await client.send(command);
  const secrets = JSON.parse(response.SecretString);
  process.env.DB_HOST = secrets.host;
  process.env.DB_PASSWORD = secrets.password;
  // Secrets exist in memory only - never written to disk
}

await loadSecrets();
// Now start the application

For GitHub Actions, I use OIDC federation to assume an IAM role - no long-lived AWS access keys stored as GitHub secrets. The workflow assumes a role scoped to reading staging secrets only.

Rotation and Emergency Access

Automatic rotation for RDS credentials via Secrets Manager Lambda rotation function. For API keys (Razorpay, Stripe, SendGrid), I set calendar reminders at 90 days and document the rotation runbook: generate new key → update Secrets Manager → deploy → verify → revoke old key.

Break-glass access is for emergencies only. One IAM role with Secrets Manager read access, requiring MFA, logged in CloudTrail, with an alert to the security channel on every use. I review break-glass usage monthly - any use outside a documented incident is a process failure.

When a secret is compromised, rotation speed matters more than rotation frequency. Have a runbook that revokes the old secret and deploys the new one in under 15 minutes.

Preventing Secret Leakage

Git hooks: pre-commit hook scans for patterns (API keys, passwords, AWS access keys) using gitleaks or trufflehog. CI gate: same scan on every PR. These catch 95% of accidental commits.

Structured logging must redact secrets. I configure log formatters to mask fields matching password, secret, key, token, authorization. A Razorpay key in a debug log is a credential leak even if the log is "internal."

Docker build args are not secret. Never pass credentials as ARG or ENV in Dockerfile. Multi-stage builds do not hide them - they remain in image layers. Fetch at runtime instead.

Engineering Takeaways

• Classify secrets into tiers: critical (auto-rotate, no human access), operational (manual rotate, break-glass), configuration (SSM String).
• Use AWS Secrets Manager for critical and operational secrets with IAM task roles - never env files in production.
• Fetch secrets at application startup into memory; never bake into Docker images or task definitions.
• Use GitHub Actions OIDC for CI secret access - no long-lived AWS keys in GitHub secrets.
• Scan Git commits and PRs for secret patterns with gitleaks/trufflehog.
• Maintain a 15-minute rotation runbook for compromised credentials.

Trade-offs I Would Consider

Option: AWS Secrets Manager (centralized, managed)

• Pros: Automatic RDS rotation, CloudTrail audit, IAM integration, no ops overhead.
• Cons: $0.40/secret/month + API call costs, AWS lock-in, latency on cold start.

Option: HashiCorp Vault (self-hosted or HCP)

• Pros: Multi-cloud, dynamic secrets, advanced policies, PKI engine.
• Cons: Operational burden (HA cluster, unsealing, upgrades), team needs Vault expertise.

What I Would Do in Production

Audit all repos for existing secrets in the first week. Rotate anything found - consider it compromised.

Set up gitleaks in pre-commit hooks and CI within the first sprint. Block merges on detected secrets.

Migrate one service at a time to runtime secret loading. Start with the highest-risk (payment, database).

Create a secrets inventory spreadsheet: name, tier, owner, last rotated, rotation method. Review quarterly.

Run a quarterly "secret drill": pick a random critical secret, rotate it, verify services recover, measure time-to-recovery.

FAQ

Q: SSM Parameter Store or Secrets Manager?

Secrets Manager for credentials that need rotation and audit (DB passwords, API keys). SSM Parameter Store for non-sensitive config and values that change rarely. Do not put DB passwords in SSM String parameters.

Q: How do developers access secrets locally?

They do not access production secrets locally. Development uses .env with dummy values. Staging secrets are accessible via a CLI tool (aws secretsmanager get-secret-value) with dev IAM credentials scoped to staging only.

Q: What about secrets in Kubernetes?

External Secrets Operator syncs Secrets Manager to K8s Secret objects. Pods mount them as volumes or env vars. Prefer volume mounts - env vars appear in /proc and crash dumps.

Q: How do I handle secrets shared across multiple services?

One secret per logical credential (one DB secret, one Razorpay secret), accessed by multiple task roles. Never duplicate secrets across Secrets Manager entries - rotation becomes error-prone.

Q: Should I encrypt secrets at the application level too?

KMS encryption at rest in Secrets Manager is sufficient for most cases. Application-level encryption adds complexity without meaningful security gain if the decryption key is in the same secret store.