Uh-oh - I've just shared my secrets with the Internet.
It's an easy mistake, but you'll only make it once: accidentally uploading your secrets to Github. What do you do now?
Why does this happen?
Sometimes as developers we want to put secrets (credentials, usernames, passwords, license keys, etc) in our code. We know we should follow best practices but sometimes we cut corners. This is what leads to many breaches.
An example: Your app in AWS needs read-only access to content in an AS3 bucket. You may be tempted to create an IAM user and put the credentials of this user into your app's code. Maybe you put them in a config file outside of your project. Or maybe you use .gitignore so that this file is never committed.
This is a dangerous practice because as long as your secret is not encrypted, you're in danger of accidentally committing it, and accidentally sharing that with your team - or worse, the whole Internet. There's a ton of examples of devs accidentally uploading their AWS or Azure keys to Github - guess what happens next? Hackers log into your AWS or Azure account using your credentials.
How does this happen?
How do you "accidentally" upload your secrets by committing them in your code? It's easier than you think.
Here's an example from my own life: I was tasked with deploying VM's to Azure. My automated scripts to deploy required a Service Principal. It was very tempting to put the secret for this Service Principal in my script, since it was normally run locally on my machine. This meant that if I ever shared my script, I would also be sharing my secrets. Not good.
However, I can also store secrets securely and automate my deployments. In my case, I used Azure's KeyVault service, where I would store and then retrieve a secret at the time of deployment. With this configured, I could confidently share my code without fear, because to run the script successfully, access to the secret in KeyVault was required. Another user could develop with the script, using their own secrets. Much safer.
Here's some other common ways that secrets are accidentally shared with the world:
- Standards for production, but not for dev. Promoting bad practices from Dev to Prod, like secrets in your code, is easier than you might think.
- Move/rename a file after configuring .gitignore. This one is frustrating but easy to commit accidentally. You deliberately keep a config file out of your repo, but then you move or rename it and commit your changes. It's now committed.
- Plain old forgetting. Mistakes happen. People rush. Beginners learn. Sometimes the secret doesn't "feel" like sensitive data, for example, a license key. You might put this in your code, and then realize afterward that it's private data you didn't want to share.
- Shared passwords or a shared file. If your team has a shared known password, you have no control over other people's management of it. If it's in a shared location like Dropbox, the same risks apply. Eventually, someone's going to share it accidentally.
- Passwords over email/Slack/chat/etc. Pretty obvious and similar to above. But this never happens, right?
Hackers scan sites like GitHub and pastebin and look for well-known types of secrets, like AWS keys and Azure ServicePrincipal secrets. Hackers will scan, discover, and attempt to exploit your shared passwords in seconds or minutes after you accidentally upload them.
How can we protect ourselves?
Below I have collated a few recommendations to remember when dealing with secrets in your application.
- Good coding practices.
- Never store secrets in code. If you have a config file that expects a variable that should be secret, encrypt that value.
- Use a least-privilege approach for service accounts. If you have an application that needs to access data, make it Read-Only. If your application needs write access, see if you can limit that write access to only what is required - not a global permission.
- Use tools designed to store secrets. Hashicorp Vault is a popular tool for storing passwords, and if you'd like to use a cloud vendor-native solution, AWS Key Management Service and Azure Key Vault are both popular, and relatively easy, tools to use.
- Use cloud providers' tools and methods for your cloud-based environments.
- Azure managed identities is a way to keep credentials out of your code. It allows you to assign an identity to a VM, much like a user in a directory. Then your code running on this VM can request a token from AzureAD, and then access services that use AzureAD as their authentication point.
- AWS IAM role permissions assigned to EC2 instances work in a similar way. Your code running on an AWS VM can access temporary credentials that can be used to access resources and scoped to permissions you determine.
- gitignore. This is the method of intentionally not tracking files in your git repo. You should use this, but it should not be your only line of defense. If you move or rename a file, it may accidentally end up committed in your repo. This is an easy and common mistake to make.
- Use development tools while you work. Personally, I've installed git-secrets which uses hooks to ensure that you do not include secrets before you commit your work. Depending on your environment and tools, you may have others you want to install at stages of your pipeline. Please share your recommendations!
- Governance of your environment. This is something you'll be extremely thankful for if you ever get breached, because it may limit your exposure. It may also prevent your breach. AWS CloudTrail and Azure Policy are good examples of this.
- Policies. For example, I only allow public IP addresses to exist in one Resource Group (this example is Azure-specific, but you can apply the idea to other environments).
- RBAC. An obvious one - least permissions required for everyone.
- Automation Accounts. Service Principals (Azure) or IAM users (AWS) need to be governed in your organization. Creating them, storing them, and the permissions they have.
- Monitoring. Turn on logging, such as AWS CloudTrail, and analytics services in your AWS or Azure accounts. You can get logs of your AWS and Azure events, and you can perform reporting and diagnostics too.
- Organizational review. A great starting point, even if you aren't in public cloud, is a best practice guide like this one from AWS or this one from Azure.
- Clean up. Here's how to remove secrets from your code, if you're using git. But more than that, you should have an automated and tested way to revoke access keys, roll app credentials, and perform an audit of your environment. This needs to be an organizational priority.
I'd love to hear what you are doing to protect yourself against your own future accidents - specifically at the intersection of secrets in code, source control, and accidents. Please share tips, advice, and recommendations!
