• Background
• Overview
• Requirements
• Installation Options
• Getting Started
• Generator Options
• Development and Testing
• Legacy Exec Plugin
• Argo CD Integration 🤖

At Viaduct, we manage our Kubernetes resources via the GitOps pattern; however, we could not find a solution compatible with our stack for managing secrets via the GitOps paradigm. We built KSOPS to connect kustomize to SOPS and integrated it with Argo CD to safely manage our secrets the same way we manage the rest our Kubernetes manifest.

KSOPS, or kustomize-SOPS, is a kustomize KRM exec plugin for SOPS encrypted resources. KSOPS can be used to decrypt any Kubernetes resource, but is most commonly used to decrypt encrypted Kubernetes Secrets and ConfigMaps. As a kustomize plugin, KSOPS allows you to manage, build, and apply encrypted manifests the same way you manage the rest of your Kubernetes manifests.

• kustomize

Using curl

curl -s https://raw.githubusercontent.com/viaduct-ai/kustomize-sops/master/scripts/install-ksops-archive.sh | bash

Or using wget

wget -qcO - https://raw.githubusercontent.com/viaduct-ai/kustomize-sops/master/scripts/install-ksops-archive.sh | bash

If you receive a Permission denied error, try running the command with sudo

wget -qcO - https://raw.githubusercontent.com/viaduct-ai/kustomize-sops/master/scripts/install-ksops-archive.sh | sudo bash

Note: Installing from source requires Go

# Optionally, install kustomize via
# make kustomize
# Verify the $GOPATH environment variable exists
make install

Before continuing, verify your installation of kustomize and gpg. Below are a few non-comprehensive commands to quickly check your installations:

# Verify kustomize is installed
kustomize version

# Verify gpg is installed
gpg --help

Make KSOPS available to your shell's PATH

source <(curl -s https://raw.githubusercontent.com/viaduct-ai/kustomize-sops/master/scripts/install-ksops-archive.sh)

To simplify local development and testing, we use PGP test keys. To import the keys, run the following command from the repository's root directory:

make import-test-keys

If you are following this tutorial, be sure to run this before the following steps. The PGP keys will also be imported when you run make test

See SOPS for details.

For this example and testing, KSOPS relies on the SOPS creation rules defined in .sops.yaml. To make encrypted secrets more readable, we suggest using the following encryption regex to only encrypt data and stringData values. This leaves non-sensitive fields, like the secret's name, unencrypted and human readable.

Note: You only have to modify .sops.yaml if you want to use your key management service in this example instead of the default PGP key imported in the previous step.

creation_rules:
  - unencrypted_regex: "^(apiVersion|metadata|kind|type)$"
    # Specify kms/pgp/etc encryption key
    # This tutorial uses a local PGP key for encryption.
    # DO NOT USE IN PRODUCTION ENV
    pgp: "FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4"
    # Optionally you can configure to use a providers key store
    # kms: XXXXXX
    # gcp_kms: XXXXXX

# Create a local Kubernetes Secret
cat <<EOF > secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: mysecret
type: Opaque
data:
  username: YWRtaW4=
  password: MWYyZDFlMmU2N2Rm
EOF

# Encrypt with SOPS CLI
# Specify SOPS configuration in .sops.yaml
sops -e secret.yaml > secret.enc.yaml

# Create a local Kubernetes Secret
cat <<EOF > secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  # Specify a name
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          # if the binary is in your PATH, you can do
          path: ksops
          # otherwise, path should be relative to manifest files, like
          # path: ../../../ksops
files:
  - ./secret.enc.yaml
EOF

Read about kustomize plugins

cat <<EOF > kustomization.yaml
generators:
  - ./secret-generator.yaml
EOF

# Build with kustomize to verify
# In kustomize v2 and v3 the command is
# kustomize build --enable_alpha_plugins .
kustomize build --enable-alpha-plugins --enable-exec .

• Validate ksops command is in your path

# Should output a path to KSOPS executable
command -v ksops

• If you prefer to not install ksops to your path, make sure the path to the executable in the generator manifest is relative to the manifests files

Someone might have already encountered your issue.

https://github.com/viaduct-ai/kustomize-sops/issues

KSOPS can also generate a Kubernetes Secret directly from encrypted files or dotenv files.

cat <<EOF > secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
secretFrom:
- metadata:
    name: secret-name
    labels:
      app: foo
    annotations:
      kustomize.config.k8s.io/needs-hash: "false"
  type: Opaque
  files:
  - ./secret.enc.conf
  - secret.yaml=./secret.enc.yaml
EOF

cat <<EOF > secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
secretFrom:
- metadata:
    name: secret-name
    labels:
      app: foo
    annotations:
      kustomize.config.k8s.io/needs-hash: "false"
  type: Opaque
  binaryFiles:
  - ./secret.enc
  - secret=./secret.enc
EOF

cat <<EOF > secret-generator.yaml
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
secretFrom:
- metadata:
    name: secret-name
  envs:
  - ./secret.enc.env
EOF

KSOPS supports kustomize annotation based generator options. At the time of writing, the supported annotations are:

• kustomize.config.k8s.io/needs-hash
• kustomize.config.k8s.io/behavior

For information, read the kustomize generator options documentation.

Sometimes there is a default secret as part of a project's base manifests, like the base Argo CD secret, which you want to replace in your overlay. Other times, you have parts of base secret that are common across different overlays but you want to partially update, or merge, changes specific to each overlay as well. You can achieve both of these goals by simply adding the following annotations to your encrypted secrets:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  annotations:
    # replace the base secret data/stringData values with these encrypted data/stringData values
    kustomize.config.k8s.io/behavior: replace
type: Opaque
data:
  # Encrypted data here
stringData:
  # Encrypted data here

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  annotations:
    # merge the base secret data/stringData values with these encrypted data/stringData values
    kustomize.config.k8s.io/behavior: merge
type: Opaque
data:
  # Encrypted data here
stringData:
  # Encrypted data here

Before developing or testing KSOPS, ensure all external requirements are properly installed.

# Setup development environment
make setup

KSOPS implements the kustomize plugin API in ksops.go.

KSOPS's logic is intentionally simple. Given a list of SOPS encrypted Kubernetes manifests, it iterates over each file and decrypts it via SOPS decrypt library. KSOPS assumes nothing about the structure of the encrypted resource and relies on kustomize for manifest validation. KSOPS expects the encryption key to be accessible. This is important to consider when using KSOPS for CI/CD.

Testing KSOPS requires:

Everything is handled for you by make test. Just run it from the repo's root directory:

make test

If you are on an older version (<=v3.x.x) of KSOPS or want to use legacy exec style plugin, the then read the Legacy README for instructions.

KSOPS becomes even more powerful when integrated with a CI/CD pipeline. By combining KSOPS with Argo CD, you can manage Kubernetes secrets via the same Git Ops pattern you use to manage the rest of your kubernetes manifests. To integrate KSOPS and Argo CD, you will need to update the Argo CD ConfigMap and create a strategic merge patch or a custom Argo CD build. As an alternative you can also use the Argo CD Helm Chart with custom values. Don't forget to inject any necessary credentials (i.e AWS credentials) when deploying the Argo CD + KSOPS build!

KSOPS Docker Image

KSOPS Quay.io Image

As of now to allow Argo CD to use kustomize plugins you must use the --enable-alpha-plugins and --enable-exec flags. This is configured by the kustomize.buildOptions setting in the Argo CD ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

The simplest way to integrate KSOPS with Argo CD is with a strategic merge patch on the Argo CD repo server deployment. The patch below uses an init container to build KSOPS and volume mount to inject the KSOPS plugin and, optionally, override the kustomize executable.

# argo-cd-repo-server-ksops-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      # 1. Define an emptyDir volume which will hold the custom binaries
      volumes:
        - name: custom-tools
          emptyDir: {}
      # 2. Use an init container to download/copy custom binaries into the emptyDir
      initContainers:
        - name: install-ksops
          image: viaductoss/ksops:v4.4.0
          command: ["/bin/sh", "-c"]
          args:
            - echo "Installing KSOPS...";
              mv ksops /custom-tools/;
              mv kustomize /custom-tools/;
              echo "Done.";
          volumeMounts:
            - mountPath: /custom-tools
              name: custom-tools
      # 3. Volume mount the custom binary to the bin directory (overriding the existing version)
      containers:
        - name: argocd-repo-server
          volumeMounts:
            - mountPath: /usr/local/bin/kustomize
              name: custom-tools
              subPath: kustomize
            - mountPath: /usr/local/bin/ksops
              name: custom-tools
              subPath: ksops
        ## If you use AWS or GCP KMS, don't forget to include the necessary credentials to decrypt the secrets!
        # env:
        #  - name: AWS_ACCESS_KEY_ID
        #    valueFrom:
        #      secretKeyRef:
        #        name: argocd-aws-credentials
        #        key: accesskey
        #  - name: AWS_SECRET_ACCESS_KEY
        #    valueFrom:
        #      secretKeyRef:
        #        name: argocd-aws-credentials
        #        key: secretkey

1. Install the Age tool and run the below command to generate a new key:

age-keygen -o age.agekey

2. Create a secret in the namespace where your ArgoCD-instance is running:

cat age.agekey | oc create secret generic sops-age --namespace=openshift-operators \
--from-file=keys.txt=/dev/stdin

3. Add the following in your ArgoCD-CRD:

kind: ArgoCD:

  kustomizeBuildOptions: --enable-alpha-plugins --enable-exec
  repo:
    env:
    - name: XDG_CONFIG_HOME
      value: /.config
    - name: SOPS_AGE_KEY_FILE
      value: /.config/sops/age/keys.txt
    initContainers:
    - args:
      - echo "Installing KSOPS..."; mv ksops /custom-tools/; mv kustomize /custom-tools/;
        echo "Done.";
      command:
      - /bin/sh
      - -c
      image: viaductoss/ksops:v4.4.0
      name: install-ksops
      volumeMounts:
      - mountPath: /custom-tools
        name: custom-tools
    volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age
      name: sops-age
    volumes:
    - emptyDir: {}
      name: custom-tools
    - name: sops-age
      secret:

4. Encrypt your secrets with sops and commit it to your repository: (all changes to the secret after encryption will give data integrity errors, so re-encrypt if you want to add stuff!)

sops --encrypt --in-place secret.sops.yaml

5. Point to your encrypted secret(s):

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  # Specify a name
  name: example-secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - secret.sops.yaml

6. Test locally if your setup works(sops/ksops/kustomize binaries are required):

kustomize build --enable-alpha-plugins --enable-exec| oc diff -f -

cp age.agekey ~/.config/sops/age/keys.txt
sops --decrypt secret.sops.yaml

7. Test to rollout your encrypted Secret w/ ArgoCD

Alternatively, for more control and faster pod start times you can build a custom docker image.

ARG ARGO_CD_VERSION="v2.6.7"
# https://github.com/argoproj/argo-cd/blob/master/Dockerfile
ARG KSOPS_VERSION="v4.4.0"

#--------------------------------------------#
#--------Build KSOPS and Kustomize-----------#
#--------------------------------------------#

FROM viaductoss/ksops:$KSOPS_VERSION as ksops-builder

#--------------------------------------------#
#--------Build Custom Argo Image-------------#
#--------------------------------------------#

FROM argoproj/argocd:$ARGO_CD_VERSION

# Switch to root for the ability to perform install
USER root

ARG PKG_NAME=ksops

# Override the default kustomize executable with the Go built version
COPY --from=ksops-builder /usr/local/bin/kustomize /usr/local/bin/kustomize

# Add ksops executable to path
COPY --from=ksops-builder /usr/local/bin/ksops /usr/local/bin/ksops

# Switch back to non-root user
USER argocd

We can setup KSOPS custom tooling in the Argo CD Chart with the following values:

# Enable Kustomize Alpha Plugins via Argo CD ConfigMap, required for ksops
configs:
  cm:
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

repoServer:
  # Use init containers to configure custom tooling
  # https://argoproj.github.io/argo-cd/operator-manual/custom_tools/
  volumes:
    - name: custom-tools
      emptyDir: {}

  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.4.0
      command: ["/bin/sh", "-c"]
      args:
        - echo "Installing KSOPS...";
          mv ksops /custom-tools/;
          mv kustomize /custom-tools/;
          echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops