Sometimes bad things just happen, or perhaps they happen because of our neglect. Bad things like committing an application secret into our repository. The problem with doing that is that if that happens, deleting the secret and committing will not suffice. We need to purge that commit completely from our repository and its history. But there are loads of other problems surrounding that. For instance, if we don’t want to commit our application secret in our repository, where should we put it then? 

Another relevant question is that, how can we store our application secret in a way that is accessible in different environments? Whether it would be the production environment or various different types of deployments or information regarding our identity and auth servers. The answer is, there are options, free and premium that can helps us achieve that in a secure fashion. There are also ways to store user secrets outside our repo in our local environment.

So, in this post, I’m going to assume we have a repository that currently contains application secrets which are committed into the repository by mistake. Then we’re going to see the steps we need to take to first remove those secrets from our repo history. Also, we’re going to make sure there are no more security concerns from this point on.

Next, we’ll see how we can start fresh and think about an application secret management strategy for our git repository. We’re going to make sure this kind of security related issues cannot happen again by implementing preventative measures. So, we’re going to see what solutions are available to be able to achieve those steps.

What is Application Secret and User Secret?

First thing first, let’s define our terms. What is application secret (service secret) and what is user secret, what are the differences between the two?

User secret

User secrets are often things like the login passwords for individual user accounts (email, cloud consoles, personal keys etc), as well as shared passwords for group/team accounts. For that we can use a password manager with organization support such as LastPass and 1password. That part is not difficult, but it’s something to have in mind, nevertheless.

Application secret (service secret)

Application secrets are things that need to persist regardless of any individual user’s employment status, e.g., they must not be affected by user off boarding, shouldn’t be stored on local machines, and should be accessible through automation. This is a more difficult subject and there are many steps that needs to be taken to achieve this. Here I will describe these steps and then we’ll see what options are available in order to be able to complete them.

Steps we Need to Take to Recover from Bad Application Secret Management

1 – Scanning the Repository for Secrets and potential sensitive information

I think a first good step is to scan the repository to find if they contain secret and other sensitive information. There are tools that can help us do that. After that we can start to come up with strategies to remove any secret or encrypt the files containing secret information. But of course, that’s not all because they still will exist in our git history. Below I listed some tools that can help us achieve that.

gitleaks: Protect and discover secrets using Gitleaks 🔑

repo-security-scanner: CLI tool that finds secrets accidentally committed to a git repo, eg passwords, private keys

Yelp/detect-secrets: An enterprise friendly way of detecting and preventing secrets in code.

2 – Make sure secrets are not leaked anywhere on the internet

The next step would be to look to see if any our secrets are leaked anywhere on the internet. If so, we need to change those secrets immediately and make sure our resources are not compromised. There are tools that can search in many places on the internet to make sure that didn’t happen. There are tools that can helps us achieve that. One of them is called trufflehog. This tool has other capabilities also, but we can use it for this purpose. Of course, there are also other tools that we can use for this purpose. But it is important to at least think about searching on the internet to make sure our sensitive information is not leaked.

3 – Remove the application secret from our git history

Our next important step is to scan the repository history for any secret. Because even though we remove the secrets that we’ve found in our repo, all of our secrets still exist in our repository history. So, we are still vulnerable because they can be easily retrieved by going back in history. There are tools that can helps us achieve that, or we can the built-in options that exist for that purpose by git itself.

If you commit sensitive data, such as a password or SSH key into a Git repository, you can remove it from the history. To entirely remove unwanted files from a repository’s history you can use either the git filter-repo tool or the BFG Repo-Cleaner open-source tool. Basically, these tools are going to rewrite the entire commit graphs while keeping the history with the things we want removed. Here’s also a good article explaining how to exactly do that with our git repository.

Removing sensitive data from a repository – GitHub Docs

4 – Encrypt any file that contain secret but cannot be removed from our repository

Sometimes there are files that even though they have sensitive information, we still cannot remove them completely. Perhaps because there might be a high refactoring cost in changing and moving them at the present time or those files have become an important part of our system and we still don’t have the necessary infrastructure to be able to move them.

For those files we can use some kind of encryption, without actually deleting them or changing how our other assets using those files. In this case we have some options, one of which is to simply encrypt those files. This gives us a little more security, albeit temporary, until we can come up with a complete solution. Here I list some of the tools that can be used to be able to encrypt files.

FiloSottile/age: A simple, modern and secure encryption tool (and Go library) with small explicit keys, no config options, and UNIX-style composability.

PGP Tool · GitHub

Steps We Can Take to Implement a Good Application Secret Management Strategy

1 – Create pre-commit hooks to check for sensitive information before them even being committed to our repository

Even though we can be watchful not to commit sensitive information in our repository, there are also tools that can helps us achieve that. They can help us detect those kinds of issue, before they even being committed to our repository. These tools are normally pre-commit hooks that check the code that is currently being committed. If any of the files being committed contained secrets, the commit will be rejected and will not be allowed in our repository. So basically, any secrets cannot be committed in our git repository. This works as a preventative measure that can stop insecure practice such as this in its track. Below I listed some of those tools and the link to their git repo.

thoughtworks/talisman: Using a pre-commit hook, Talisman validates the outgoing changeset for things that look suspicious — such as tokens, passwords, and private keys.

awslabs/git-secrets: Prevents you from committing secrets and credentials into git repositories

2 – Include a security check for secrets in our build process and CI/CD

A good next line of defense would be to make sure our infrastructure such as our build system and deployment processes becomes aware of this risk. That is to say the build is going to fail or our deployment process is going to be cancelled if such security risk has existed in our latest changes.

I think we can create a script which runs something like git-secrets as a build step to find out if there are any secret inside our latest commits. As I said subsequently, we can fail the build or cancel the deployment if there was any commit that contained secrets, with appropriate message.

3 – Use a secret repository and read our application secret from there instead of storing them on our file system

Now we need to find a better way to store and use secrets in our application. The correct way starts by not storing any secrets in our repositories at all, but to store them somewhere other than our repo.

We should have a dedicated secret storage mechanism. There are free and premium options for that which handles all of this for us. Not only they do provide a single place for developers to store and see their app secrets, but it also provides access control, access logging, and versioning of secret values. Also, it can be integrated into our pipelines and our deployments to production to automatically deliver secrets to running applications.

There are many tools for managing secrets with different characteristic and features, free and paid. But the ones that were most prominent and people on the internet were using them with the most communities around them are the ones that I’m going to list down below.

HashiCorp

It has a community version which is free to use, it can be a good start.

Mozilla SOPS

It might be more open and cheaper than HashiCorp, but harder to implement.

It allows to encrypt the YAML values (as opposed to keys) but leave the keys readable. Decrypt function native in Terragrunt, ArgoCD, Flux, Helm, and many others.

Azure Key Vault

If you’ve already use azure Devops, then integrating azure key vault into your application is not going to be that difficult because many needed prerequisites are already there. if you want to know more about their differences, here’s a Stack overflow answer that goes into more detail.

4 – Find a way to integrate our secret repository with our CI/CD pipeline

After all of the above steps have taken, finally we can move to our final step. Namely, to read our application secret needed to run our services from our secret repository. We need to do that for different environments, because some of them might have different requirement in terms of supplied secrets. Here’s an article that explains how we can integrate HashiCorp into our CI/CD pipeline if we opt to use it for our application secret repository.

More Readings

Encrypting your YAML files using Mozilla SOPS and GCP KMS | by Pascal Zwikirsch | Level Up Coding

If you don’t use a secret management tool, you’re doing it wrong

Sensu | What is HashiCorp Vault and how does it work?

Various Tools Helping in Application Secret Management

There are numerous tools available to search a Git repository for potentially sensitive information. Each of these may have distinct capabilities and employ distinct detection methods. Here’s a curated list of secret detection tools I’ve discovered:

Summary

In this post we’ve taken a look at a hypothetical scenario where we committed sensitive data in our git repository by mistake. Then we saw all the potential steps that can be potentially taken, to be able to recover from that mistake and have a more secure way of managing application secret. We also introduced the tools and resources that can be used in order to achieve those steps. I hope this post gave you a glimpse into managing application secret. If you have any question or suggestion that completes the steps I described above, please share it with my readers in the comment section.