Secrets Management

Why Secrets Management Matters

Every application has secrets. Database passwords. API keys. OAuth tokens. Encryption keys. Webhook signing secrets. The average production application has dozens of secrets, and each one is a potential security incident if exposed.

AI-generated code is particularly bad at secrets handling. Prompts produce code with hardcoded API keys, .env files that get committed to git, and secrets passed as command-line arguments visible in process lists. AI assistants optimize for “it works” not “it’s secure.” The application functions perfectly while broadcasting credentials to anyone who can read the repository history.

The problem compounds with team growth. When secrets live in .env files, every developer has a copy. When someone leaves the team, you should rotate every secret they had access to. But nobody tracks which secrets each person accessed. Nobody has a rotation procedure. The departed employee’s local .env file still has valid production credentials weeks later.

What We Build

Secret Storage:

• HashiCorp Vault for teams that need full control over secrets lifecycle
• AWS Secrets Manager or GCP Secret Manager for cloud-native applications
• 1Password or Doppler for smaller teams that need simplicity
• Proper encryption at rest and in transit for all stored secrets

Access Controls:

• Principle of least privilege - services only access the secrets they need
• Role-based access that maps to your team structure
• Audit logging for every secret access
• Separate secret stores for development, staging, and production

Key Rotation:

• Automated rotation schedules for database passwords and API keys
• Zero-downtime rotation procedures
• Rotation runbooks for secrets that require manual steps
• Expiration alerts before secrets become invalid

Application Integration:

• Secrets injected at runtime, never baked into images or artifacts
• Environment variable injection from secret stores
• Kubernetes External Secrets Operator for container workloads
• CI/CD pipeline secrets management without exposing values in logs

Incident Response:

• Procedures for when a secret is exposed
• Rapid rotation capabilities
• Blast radius assessment - what can the exposed secret access?
• Post-incident verification that rotation was complete

Our Experience Level

We’ve implemented secrets management for applications handling sensitive customer data, payment processing, and healthcare information. We’ve cleaned up repositories where API keys were committed to git history and needed to be rotated across dozens of services.

We’ve deployed HashiCorp Vault in production with auto-unseal, high availability, and audit logging. We’ve configured AWS Secrets Manager with automatic rotation for RDS credentials. We’ve integrated Doppler into development workflows so developers have correct secrets without manual .env file management.

We’ve responded to secret exposure incidents - API keys posted to public repositories, database credentials in Docker images pushed to public registries, and OAuth tokens logged to monitoring systems. Each incident taught us what goes wrong and how to prevent it.

When to Use It (And When Not To)

Every application needs some level of secrets management. The minimum: secrets in environment variables (not code), .env files in .gitignore, and a documented list of what secrets exist.

For applications with a small team and limited secrets, Doppler or 1Password’s secrets automation provides excellent developer experience with minimal setup. Developers get the right secrets automatically. Rotation is manual but tracked.

For applications handling sensitive data or operating under compliance requirements, invest in proper secret stores. HashiCorp Vault or cloud-native secret managers provide access controls, audit logging, automatic rotation, and encryption that compliance frameworks require.

For microservice architectures with dozens of services, centralized secrets management becomes essential. Services authenticate to the secret store and receive only their authorized secrets. Rotation happens once and propagates everywhere. No more updating .env files across twenty repositories.

Common Challenges and How We Solve Them

Secrets committed to git. The most common problem. API keys in source code, .env files in repository history. We scan repositories with tools like truffleHog or gitleaks, rotate every exposed secret, and add pre-commit hooks that prevent future commits containing secret patterns.

No rotation procedures. Secrets are set once and never changed. Database passwords are the same ones created two years ago. We establish rotation schedules, automate what can be automated, and document manual rotation procedures with step-by-step instructions.

Developer experience friction. Proper secrets management adds steps to the development workflow. Developers bypass security because it’s inconvenient. We implement solutions that are as easy as .env files but actually secure - Doppler pulls, Vault agents, or secret store CLIs that integrate with existing workflows.

Secret sprawl across services. The same API key exists in twelve different services’ configuration. When it rotates, eleven of them break. We centralize secrets in a single source of truth and have services reference them rather than copy them.

Secrets in CI/CD logs. Build pipelines print environment variables during debugging, exposing secrets in build logs. We configure pipelines to mask secret values, use pipeline-native secret management, and audit logs for accidental exposure.