Under the Covers of SOPS for Codifying CI/CD and IaC Secrets

Published on Jun 10, 2024 by Ian Homer

SOPS (Secrets OPerationS) is a command line tool that encrypts and decrypts files in a way that allows you to codify CI/CD and dev processes that require secrets. In an encrypted form, secrets can be stored in a Git repository, with appropriate access control, in the knowledge that it is hard to decrypt the secrets without the authorisation to decrypt. Codifying of secrets and SOPS tooling, helps make rotation of secrets easier, and hence encourages us to become better at timely rotations, in turn de-risking exposure of historical secrets.

Team members, or services, can be given the appropriate permissions to handle different groups of secrets. The SOPS command line decrypts the file when loaded and encrypts the file when saved.

A good way to get hands on with SOPS is with local encryption tooling. We can do this with age. Age is a file encryption library that, roughly speaking, is easier to operate than GPG. Typically we would delegate such encryption processes to cloud tooling such as AWS KMS, Azure Key Vault or GCP KMS. Starting with age encryption, we can gain an understanding of the fundamentals. This also allows us to look under the covers of how SOPS encrypts secrets.

Installing SOPS and age

We'll need the sops and age command line tools installed. Go ahead and install these, for example on macOS we can install these with brew.

brew install sops
brew install age

Generate our encryption key with "age"

To give us a local set up to try out the SOPS tooling, we can create an age key for SOPS to encrypt and decrypt our files. SOPS uses this key to encrypt the data key, which is then used to encrypt our secrets.

age-keygen > key.age

We can set environment variables for the location of key file and the age recipient (public key) which SOPS will pick up on and use for the for the encryption and decryption process.

export SOPS_AGE_KEY_FILE=./key.age
export SOPS_AGE_RECIPIENTS=age...

Encrypt the secret with SOPS

Let's create an encrypted secret file using the age recipient we have set up.

echo "{'password': 'top-secret'}" |
  sops --input-type yaml --output-type json --encrypt /dev/stdin > secret.json

If we look in the encrypted file secret.yaml we see how it stores the encrypted data.

{
  "password": "ENC[AES256_GCM,data:...,iv:...,:tag:...,type:str:str]",
  "sops": {
    "kms": null,
    "gcp_kms": null,
    "azure_kv": null,
    "hc_vault": null,
    "age": [
      {
        "recipient": "age1hh2ejk5kgcvsgrlkjqjhlgm9qnqwmuxzrvshswjtw
l7cr7zkfs7q7qyvrx",
        "enc": "-----BEGIN AGE ENCRYPTED FILE-----\n...==\n-----END AGE ENCRYPTED FILE-----\n"
      }
    ],
    "lastmodified": "2024-06-08T06:22:26Z",
    "mac": "ENC[AES256_GCM,data:...:str]",
    "pgp": null,
    "unencrypted_suffix": "_unencrypted",
    "version": "3.8.1"
  }
}

The password field includes the encrypted data. The data has been encrypted using the data key that is stored in the sops.age[0].enc field. That data key is encrypted for our age recipient. Note that to encrypt the data for an age recipient I do not need to have the key. I only need the key when it comes to reading secret.

How does SOPS encrypt our secret?

I'm naturally curious in what is happening. To help understand a little more, I thought I'd have a go at decrypting this secret away from the SOPS tooling. We are working in with local tooling, we have all the keys and we know the encryption algorithm, so it should be possible.

With cloud based tooling, the data key would have been encrypted with master keys that we don't have access to. We may have permission to encrypt and decrypt (with the master key), but we shouldn't at any time be exposed to or have access to the master key.

Now we can't use openssl to decrypt the data. If we try with openssl enc -aes-256-gcm we get an error stating enc: AEAD ciphers not supported. See the OpenSSL docs for background on why.

Let's instead create a small python script decrypt.py to decrypt the data using the cryptography module.

pip install cryptography

Extract and decrypt the data key with age. We'll use this to decrypt the data. Recall that the data key is stored in our secrets file and has been decrypted using age, so we can readily reverse this.

cat secret.json | jq -r '.sops.age[0].enc' |
  age --decrypt -i key.age > data.key

The ENC field in the secret file is encoded with a specific format

ENC[AES256_GCM,data:...,iv:...,:tag:...,type:str:str]

We can see that it is encoded with AES GCM encryption. See How does AES GCM encryption work for a good overview on how this encryption works. The cryptography python module provides a function to decrypt with this cypher. We get the nonce from the Base64 decoded value of the iv (Initialization vector) field in the ENV block. The data we pass in to the API is the data field appended with the tag, both of which need to be Base64 decoded first. Finally we need to set the associated data to authenticate the encryption. When we decrypt, this has to be the same as the value that was used by SOPS to encrypt the data, otherwise the library will reject the decryption. For SOPS the associated data is a colon separated path (with a trailing colon) of the data in the secret file. In our case this is password:.

With all that in mind we can proceed with decrypting the secret.

import re
import base64
import json

from cryptography.hazmat.primitives.ciphers.aead import AESGCM

with open("./data.key", "rb") as infile:
    key = infile.read()

with open("./secrets.json") as infile:
    enc = json.load(infile)["password"]

match = re.match("^ENC\\[AES256_GCM,data:(.+),iv:(.+),tag:(.+),type:(.+)\\]", enc)

if not match:
    raise Exception("encrypted data is not in expected format")

data = base64.b64decode(match.group(1))
iv = base64.b64decode(match.group(2))
tag = base64.b64decode(match.group(3))

cyphertext = data + tag

aesgcm = AESGCM(key)

additional_data = b"password:"
plaintxt = aesgcm.decrypt(iv, cyphertext, additional_data)
print(plaintxt)

Which when we run the script we see the decrypted secret.

❯ python decrypt.py
b'top-secret'

So there we have it, we can see how the secret file is structured and how the data is encrypted. Next, we'll explore more the SOPS tooling to see how it can with codifying secrets.

Editing the secret

Let's create another secret file, this time we'll start with a YAML plain text file that we want to encrypt before sharing.

# secret.yaml
my-service:
  password: this-is-top-secret

We've got a secret in a plain text file, and we don't want that. We can encrypt it with SOPS. We'll use the -i option to update the file in place, and in effect removing the secret in plain text.

sops -i --encrypt secret.yaml

We now a have a secrets file like before, although now in YAML format.

my-service:
  password: ENC[...]
sops:
  kms: []
  gcp_kms: []
  azure_kv: []
  hc_vault: []
  age:
    - recipient: age...
      enc:

We had the secret in a plain text file before, so probably best if we rotate the secret now. With the secret file set up, we can edit it with the sops command. SOPS will jump into your editor configured with $EDITOR environment variable, allow you to edit the decrypted file and then when you save, SOPS will re-encrypt it for you. Go ahead and edit the secret to a new secret, save, and we're good.

sops secret.yaml

Rotate the data keys

As a general house keeping practice we should rotate data keys. This mitigates against any rogue process that might have cached, or extracted, a data key.

sops --rotate -i secret.yaml

We could put this in a Git hook to rotate before commit. We could also set this up on automated process to run on update or periodically.

Adding new keys

We can add new keys to set up access for other actors. To see this in practice we can create ourselves another age key.

age-keygen > key-alt.age

Then add that key to the secrets file with a key rotation.

sops --rotate --add-age age... -u secret.yaml

We now have two keys defined in our secrets file

sops:
  kms: []
  gcp_kms: []
  azure_kv: []
  hc_vault: []
  age:
    - recipient: age1hh2ejk5kgcvsgrlkjqjhlgm9qnqwmuxzrvshswjtwl7cr7zkfs7q7qyvrx
      enc: ...
    - recipient: age19vm5akd0y72mj36fk3dra63d60zg9t4g3sz2pcnd0urdmck6833qds8twl
      enc: ...

To remind ourselves on how SOPS work, we can see that both these enc blocks provide the same data key, albeit encrypted for different recipients. We can see this when we decrypt the data with the two different age keys that we have created.

cat secret.yaml | yq -r '.sops.age[0].enc' |
  age --decrypt -i key.age | base64
cat secret.yaml | yq -r '.sops.age[1].enc' |
  age --decrypt -i key-alt.age | base64

Also, as mentioned before, we do not need the age key to add this other recipient. If we were adding access for some other party we are not likely to have the key.

Although I'm not going to go into the cloud provider support in this article, this process naturally applies when we delegate encryption to cloud services like AWS KMS, Azure Key Vault or GCP KMS. For those service access control, to the process to encrypt or decrypt the data key, can be centrally managed.

Codifying secrets and access control

SOPS comes incredibly powerful when we want to codify our secrets and tightly control who or what can access each secret, or groups of secrets. The cloud encryption services can provide audits on who decrypts a data key to access a secret. We can set up processes that rotate secrets and allow intended recipients to be able to read updates to secrets.

We can store these secrets in a private Git repository, control who or what can access the encrypted secret files. When we update secrets we could automate the rotation of the data keys, so that any actor that may have a copy of the old data key, will need to decrypt the new data key before accessing secrets. A Git repository gives us a natural audit of when secrets were changed and who changed them. This can help guide us when secrets are due a rotation.

Codification of secrets makes our secret rotation processes easier, so we can force any actor who might have retained an old copy of the repository, to pull updates to the repository and decrypt the new secret. Such actions require the appropriate access control and can provide the desired access audit.

On a final note, it is worth noting that "CNCF accepted SOPS on May 17, 2023 at the Sandbox maturity level.". SOPS is definitely worth looking at if you have not yet had the chance.