The Joys of SOPS — managing secrets and making software teams happy.

Author Tim Kaczynski

Organizations struggle with secrets management. I’ve seen a wide range of secret management; plaintext passwords living on jumpboxes, secrets committed directly to code, shared keepass vaults in network storage, managed solutions like LastPass and OnePassword, enterprise grade tools like KMS and HashiCorp vault.

Some organizations are perpetually stuck between solutions owing to technical debt and broken systems of work. Luckily, I have never seen enterprise sticky note secrets management (whew!).

If you’re researching your next solution, you should consider Secrets OPerationS — aka SOPS [https://github.com/getsops/sops]. From a stability perspective, SOPS has a lot going for it:

• Created in 2015, it has nearly 10 years of active production use
• Open source and well maintained
• Owned and licensed by a beneficent organization, Mozilla
• Wide adoption between corporations and government entities
• Easy to configure and use
• Good documentation to figure out edge cases
• Wide feature set for integration with multiple services and tools
• Owing to these properties, from a software quality and sustainability perspective, SOPS checks many boxes for a secrets solution.

That’s not joy though. That’s survival. The minimum. Let’s move beyond that — how does SOPS bring us joy?

SOPS allows us to treat secrets very nearly the same as all of our other variables. Let’s run through an example of daily use. We have our unencrypted example file as so.

If we want to encrypt this file, it’s as simple as having a key present in the SOPS config file, and running

“`sops -config ./environments/medium-article/config.sops.yaml –encrypt –in-place./environments/medium-article/medium.sops.yml“`

We can see a few things here — our keys have remained unencrypted, our values are encrypted, and we have a bunch of “stuff” at the bottom of the file. This “stuff” includes the types of keys used to encrypt the file (kms, gcp_kms azure_kv, hc_vault, age, pgp). Data about the age key used to encrypt this file (public key and ID), when the file was modified, a Message Authentication Code (MAC) to ensure that no value has been added or removed in a sketchy way, a suffix that could be used to keep just some values unencrypted if you so chose (for example ansshkey_unencrypted) and the version of SOPS that was used.

At this point, the file could safely be pushed remote and used in whatever workflow you could realistically imagine. Encrypt with as many keys and key types as you want and you can retrieve your information. You could also use a combination of key types for enterprise setups — like one KMS in one region, another KMS in another region, and a PGP/ AGE key saved in a physical vault or safe just in case.

Use in CI/CD is simple — create an AGE or PGP key as a secret in your GitLab or GitHub CI/CD secrets store for decryption and you’re off to the races. Adding your teammates is easy as well, just get their public key, decrypt the file, add the key to the config, reencrypt, and push back to remote. This becomes a functional form of access control when using issues to track who was given access.

There are two potential issues that are easily managed: The first is accidentally pushing your secrets remote. This can be accounted for with pre-commit hooks looking at the SOPS file and ensuring the MAC field — or AES in the value is present. The second issue lay with the issues inherent to SSH keys when using PGP/ AGE. SSH keys can sometimes become a hassle if the developer or engineer using them doesn’t treat them appropriately. The same can be true of AGE/ PGP. It’s possible to mitigate this issue multiple ways, but ultimately comes down to system level thinking — what is the root cause of the issue for your org if this problem exists and how can we make things easier to remove the issue.

The value of being able to handle secrets using version control has been a blessing, saved us countless hours of troubleshooting a neglected portion of many environments, and enabled collaboration while reducing overhead. SOPS has allowed us to manage secrets easily. With zero technical issues encountered by us since employing the tool, it has made operations even smoother. Of course a tool is only as good as the systems that support the smooth flow of work. GitOps, trunk based development, monorepo use, and Test Driven Development have been the support that SOPS has needed to truly flourish in our environment — and allow us to sustainably deliver value.

Sops