Secrets Management for Developers

Charlie Belmer -

One thing that comes up frequently in security is how to deal with application secrets. There are many methods for managing secrets, from hard coding into source code to using a credential manager to environment variables.

In this post, I'll go through some Do's and Don'ts for managing secrets securely, both for web services and for client applications like mobile apps that need embedded API keys.

The Best Practices for Managing Secrets

If you just want to go straight to the most secure practices, here they are.

  • Don't store any secrets on client applications or devices. If you need something like an auth token for a service, generate one token per user or device, and treat it as you would a user password.
  • Use a secret manager like Hashicorp Vault, AWS Secrets Manager or other platform specific credential management technology like kubernetes secrets. Store all secrets here: API keys, service passwords, secret keys, etc.
  • Vault access should use a management system (IAM for AWS or the platform equivalent). If IAM solutions aren't available, use a CI/CD system to encrypt the vault credentials and inject them at deployment. The credentials themselves can be stored in the Vault and rotated by the vault admins.
  • Create separate sets of credentials for production at a minimum, ideally for each environment.

This does requires running a separate piece of infrastructure (the vault), generally means more complex code has to be written to retrieve secrets, along with all the various failure modes that might be encountered, and ties your application more closely to the hosting solution. Not all organizations or teams are willing to do everything here.

Slightly Less Secure, but Still Pretty Good, Secret Management.

For some systems, the overhead of maintaining a vault is too great. you can get most of the way there by managing credentials in a CI/CD pipeline and injecting them into the deployment environment. You can do this a few ways - by having a build step that injects encrypted credentials into a file (preferred), or directly setting environment variables on deploy (only if a file won't work for some reason, to avoid accidental disclosure of environment variables from things like phpinfo).

Alternately, if not using a CI/CD platform, have the production support team deploy environment variables or update credential files manually on production systems. Ensure that separate passwords are used in each environment.

Don't Do These Things

Here's a handy list of things to avoid doing where possible.

  • Don't store credentials in your source code management system, like github, even if internal or private. It's hard to remove them from the history and too easy to leak.
  • Don't use the same set of credentials or API keys in production and other environments. Keep production credentials limited to a very small set of owners.
  • Don't give everyone read access to a credential store. Instead, use granular control, so users can only see credentials they reasonably need.
  • Don't put secrets in client applications. Every secret on a client can (and likely eventually will) be found and made public. This includes encryption keys, API keys, database credentials, and anything else that you wouldn't want users having access to.

    Client applications include front end JavaScript on a web page, downloadable binaries, and mobile applications.
  • Don't share credentials between different production systems (have separate API keys, passwords, user accounts, etc. for every application)

Conclusions

Managing secrets is hard, which is why searching for "credential leaks" finds multiple recent news stories of data breaches, ransoms, and other problems from mismanaging credentials.

Though it can be involved to get to the best practices, every step in that direction helps!