Secrets In GitOps That's Used By Governments And The Military

How to manage Kubernetes secrets in GitOps

One of the mistakes that even some senior developers sometimes still make is committing sensitive data to Git repository services, like GitHub or GitLab. I can't personally confirm this, since the senior developers I've worked with so far haven't made this mistake, but I have heard stories from other teams. What I can confirm is that this is a common mistake among junior developers, myself included. Luckily, I made this mistake during my traineeship in a non-production environment, so the implications weren't significant. It did teach me early on to be aware of this.

Recently I got introduced to GitOps. Now, I don't want to make this post heavily focused on GitOps, but in short, GitOps is a modern approach to managing infrastructure and applications using Git as the single source of truth. While I really see the benefits of using GitOps, it does bring some challenges that need to be addressed. One of those challenges is how to work with secrets. Especially in a production environment where you may have multiple applications running in your cluster, each with its own database and secrets. Ideally, you want these secrets to be in one place, so that you could easily pull those secrets in the likely event that you need to rebuild your cluster. In other words, you don't want to manually create a new secret for each application, but you want this to be available with all of your other code in a Git repository service, and therein lies the rub. So in this post, I want to share one of many ways you can achieve this, while not exposing your secrets in public.

Manage Kubernetes secrets with SOPS

SOPS is a CLI-tool that helps to encrypt Kubernetes secrets. The only prerequisite you'll need is a Kubernetes cluster with the GitOps toolkit controllers installed on it.

You could encrypt your secrets using OpenPGP, but the official documentation recommends using age, which is an alternative that is more simple and modern.

So we will need to install both SOPS and age, and we can do that by running the following command:

brew install sops age

Once this is installed, we will need to create what's called an agekey, which is similar to SSH-keys. So it will have a public and private key pair. The command also looks very similar, and is as follows:

age-keygen -o age.agekey

You can name the key however you want, but it must have the extension .agekey. If the command ran successfully, then you should be able to view the content of the keys, which consist of a public and private key. Similarly to SSH-keys, it's perfectly fine to share your public key to the world, but you should never ever share your private key with anyone.

As mentioned above, ideally we want to be able to push all of our manifests.yaml to a Git repository service, including our secret.yaml. However, if we would push a vanilla Kubernetes secret.yaml, then anyone could simply decode the data from that manifest, since it's simply base64 encoded. And this is where we use SOPS and age to encrypt the manifest, so that we can safely push our manifest to a Git repository service.

In order to do this, we will need our age public key, which we can either copy/paste from the .agekey file or export as an environment variable. But once you've done that, we can simply run the command:

sops --age=$AGE_PUBLIC_KEY \
--encrypt --encrypted-regex '^(data|stringData)$' --in-place secret.yaml

Assuming the command ran successfully, when you now view the content of secret.yaml, you will see that the data is completely encrypted. And not just encrypted, but AES 256 encrypted, which is virtually impenetrable. It's that sophisticated and effective that even governments and the military uses it. But this also means that it's perfectly safe now to push your manifests to a Git repository service. Just make sure that no one else have access to your private agekey.

Now the last thing we need to do is to set the decryption secret in the Flux Kustomization to sops-age. So basically we need to add our private agekey to the cluster, which if you remember is in the age.agekey, and we can do this by running the following command:

cat age.agekey | kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

With this Flux can now access our private agekey, and all we need to do is tell Flux where to find this key. And we do this by creating a .sops.yaml file in the same directory where the flux-system directory is located with the following content:

creation_rules:
  - path_regex: .*.yaml
    encrypted_regex: ^(data|stringData)$
    age: <AGE_PUBLIC_KEY>

And finally, we will need to add one more code to the Kustomization file that is pointing to where the secret.yaml is located. So if you're following the Flux' way of structuring your repositories, i.e. using a monorepo, then this Kustomization file(s) would typically live in the same directory where you have just created the .sops.yaml file and where the flux-directory is located, which is ./clusters/staging/apps.yaml for example. And in this file you will need to add the following code to let Flux know to use the SOPS provider for the decryption:

decryption:
    provider: sops
    secretRef:
      name: sops-age

And the above code is part of the spec section in that Kustomization file in the flux-system namespace.

And with that you have a relatively easy and safe approach to managing secrets in Kubernetes with SOPS in GitOps. I hope this is helpful, if not feel free to connect with me. Bye for now.