• Fabrice Vergnenegre

Lightweight Kubernetes GitOps Secrets


Photo by Bharat Patil on Unsplash

Every day at Sokube we evangelize GitOps as a way to simplify, improve and accelerate digital operations, and we support our customers on this journey. With the abundance of technology bricks that now compose the "Cloud Native" landscape, something that years ago looked difficult and hazardous is now easily reduced to choosing the combination of products / platforms that best fits a use-case.


However, when it comes to keeping sensitive information confidential inside a GitOps approach, choices are neither multiple nor simple. With its reversible base64 encoding, Kubernetes Secrets has been the Achille's heel of the platform regarding information security or confidentiality whether or not RBAC has been correctly configured. Native Kubernetes Secrets to store confidential or sensitive information should be purely prohibited on production systems, at least.


Couple of GitOps-friendly alternatives that preserve confidentiality have naturally emerged as workarounds: the first part of this article will be focusing on a short overview of what's existing. While Hashicorp Vault remains a truly secure, feature-rich and widely adopted solution, we believe it is positioned rather as an enterprise-wide or cross-organizational solution for confidential information. In the context of GitOps however, it would distort the “git as single source of truth” principle. Finally, a pertinent deployment of HashiCorp Vault often requires the appropriate organizational changes as well as a solid budget for its operations.


In the second part of the article, we will focus on two lightweight solutions which provide exactly what is needed for GitOps without disrupting the organization, the project and the budgets : Bitnami Sealed Secrets and Soluto's Kamus.


Kubernetes Secrets Landscape


Most of our use-cases involve on-premise only installations, so while Amazon, Azure, and Google usually provide their secret management for their platform, we've focused primarily on the on-premise solutions, and we have evaluated:

  • Complexity: Overall we have estimated the efforts to get a grasp on the solution, to install and operate it.

  • Maturity: Based on the number and frequency of the releases, the age of the solution, the practical examples found on the web that demonstrate its adoption

  • Scope: Classify the scope of validity of the secrets, organization meaning that there is no specific restriction of validity of the secret, contrary to cluster or service account where the secret can’t be decrypted on another specific cluster or with another service account.

  • Dependency: This is an important aspect of the solution that makes it more or less applicable in particular contexts



Bitnami Sealed Secrets


The Bitnami Sealed Secrets solution is based on a controller that runs in the target cluster, where a cluster-specific pair of private/public keys is generated.


Encryption


The public key is used by a client binary (kubeseal) to live-encrypt data (in the form of custom resources named SealedSecret). This operation can also be done offline (without the need for a cluster with a bitnami controller) by using the controller certificate (public key) that can be downloaded from the cluster. The resulting resource is a SealedSecret that is safe to store inside a git repository.


Decryption


The controller detects the presence of SealedSecret resources and automatically decrypts the content in an equivalent standard Secret resource.


Installation


Installation of the solution is relatively easy:

$ kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.13.1/controller.yaml

The client tool (kubeseal) to interact with the controller inside the target cluster is a standalone binary that can be downloaded separately:

$ wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.13.1/kubeseal-linux-amd64 -O kubeseal

Usage


We will use a specific namespace:

$ kubectl create namespace my-namespace
namespace/my-namespace created

Bitnami's kubeseal can be used as a pipe command, and when it's combined with the dry-runs capabilities of kubectl, we can generate sealed secrets without even intermediate resources stored locally as files:

$ kubectl create secret generic db-credentials \
  -n my-namespace \
  --from-literal=DBUSER=mydbuser \
  --from-literal=DBPWD=highlysecret \
  --dry-run=client \
  -o yaml \
  | kubeseal -o yaml > db-credentials-sealed-secret.yaml

The resulting SealedSecret can be stored safely inside a git repository:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: db-credentials
  namespace: my-namespace
spec:
  encryptedData:
    DBPWD: <ENCRYPTED DATA>
    DBUSER: <ENCRYPTED DATA>
  template:
    metadata:
      creationTimestamp: null
      name: db-credentials
      namespace: my-namespace

Applying the resource:

$ kubectl apply -f db-credentials-sealed-secret.yaml

Checking the resource:

$ kubectl get sealedsecrets -n my-namespace db-credentials 
NAME             AGE
db-credentials   2m3s
​
$ kubectl get events -n my-namespace
LAST SEEN   TYPE     REASON     OBJECT                        MESSAGE
2m9s        Normal   Unsealed   sealedsecret/db-credentials   SealedSecret unsealed successfully

Checking the equivalent Secret (and data) that has been generated by the controller:

$ kubectl get secrets -n my-namespace
NAME                  TYPE                                  DATA   AGE
...
db-credentials        Opaque                                2      3m18s
​
$ kubectl get secrets -n my-namespace db-credentials -o jsonpath='{.data.DBUSER}' | base64 --decode
mydbuser
​
$ kubectl get secrets -n my-namespace db-credentials -o jsonpath='{.data.DBPWD}' | base64 --decode
highlysecret


Notes


Important to know: Each sealed secret is encrypted with its own random asymmetric key that is specific to the sealed secret name and namespace. Copy-pasting the encrypted data for another secret or in another namespace won't work.


By default, SealedSecrets generated for a cluster won't work with another cluster (the installation of the Bitnami Controller creates a new pair of public/private keys), which could make the overall GitOps staging operations cumbersome (each environment would need to have its custom SealedSecrets). One solution is to export/import the same set of key pairs from one cluster to another. However a compromised key on a cluster would also grant access to all sealed secrets on all clusters with the same key pair.


Summary


➕ Simplicity of setup and usage

➕ Mature and well maintained solution

➖ Regular Secrets are still exposed

➖ No usage outside Secrets (e.g. ConfigMap)



Kamus


The Kamus solution offers two different mechanisms for secrets generation. One that is similar to Bitnami Sealed Secrets, another which is a true zero-trust secrets management solution.


First mode: KamusSecret


KamusSecret tend to mimic the Bitnami Sealed Secrets : a controller inside the cluster detects KamusSecrets and provide a decrypted version as a Secret.


The encoding part is a bit more tedious, as the kamus-client doesn't support passing a whole Secret resource but rather works as a raw encryption service: individual items need to be separately encrypted with the client, and aggregated manually inside an “envelope” KamusSecret definition.


As with Bitnami, the controller is responsible to decrypt a KamusSecret as a regular Secret.



Second mode: Zero-Trust Secrets


Contrary to a Bitnami Sealed Secret or a KamusSecret, Kamus zero-trust mode has the huge advantage of never revealing the decrypted data except for the intended service at runtime.


The encryption part of the zero-trust secrets mode is very similar to the previous example: instead of wrapping encoded data in a KamusSecret, a ConfigMap will be used to store secrets as encrypted files in a mounted ConfigMap volume.



For consumption of secrets, an initContainer is invoking the kamus-decryptor service and decrypted results are stored in an ephemeral emptyDir volume accessible only for the application container in the pod.



Installation


Installation of the Controller and its CustomResourcesDefinitions is relatively easy:

$ helm repo add soluto https://charts.soluto.io
​
$ key=$(openssl rand -base64 32 | tr -d '\n')
$ helm upgrade --install kamus soluto/kamus --set keyManagement.AES.key=$key

We've added here the random key generation as the solution comes (dangerously) with a predefined key. Also, contrary to Bitnami, the solution installs its workloads in the default namespace so we recommend tweaking the helm release accordingly. Finally, replicas in the encryptor/decryptor deployments are set to 2 by default, which might be unnecessary.


The Kamus client is unfortunately an npm package (a debatable choice that forces users to install npm package manager). A Docker container image seems to be available, however it’s not mentioned on Kamus website so we have chosen to stick to the documented approach:

$ npm install -g @soluto-asurion/kamus-cli

We will use a specific namespace and a specific service account:

$ kubectl create namespace my-namespace
namespace/my-namespace created
​
$ kubectl create serviceaccount dummy-sa -n my-namespace
serviceaccount/dummy-sa created

The kamus-cli also needs an exposed encryptor pod as an URL:

$ export POD_NAME=$(kubectl get pods --namespace default -l "app=kamus,release=kamus,component=encryptor" -o jsonpath="{.items[0].metadata.name}")$ kubectl port-forward $POD_NAME 8880:9999 &


KamusSecret Usage


First scenario, setting up a KamusSecret containing our sensitive information:

$ kamus-cli encrypt --secret mydbuser --service-account dummy-sa --namespace my-namespace --allow-insecure-url --kamus-url http://localhost:8880
[info  kamus-cli]: Encryption started...
[info  kamus-cli]: service account: dummy-sa
[info  kamus-cli]: namespace: my-namespace
[warn  kamus-cli]: Auth options were not provided, will try to encrypt without authentication to kamus
Handling connection for 8880
[info  kamus-cli]: Successfully encrypted data to dummy-sa service account in my-namespace namespace
[info  kamus-cli]: Encrypted data:
0XZm9vDLVvjNGCgeFZqNnA==:GoaTZCXkwyfoBNRwjtgsOQ==$ kamus-cli encrypt --secret highlysecret --service-account dummy-sa --namespace my-nymespace --allow-insecure-url --kamus-url http://localhost:8880
[info  kamus-cli]: Encryption started...
[info  kamus-cli]: service account: dummy-sa
[info  kamus-cli]: namespace: my-nymespace
[warn  kamus-cli]: Auth options were not provided, will try to encrypt without authentication to kamus
Handling connection for 8880
[info  kamus-cli]: Successfully encrypted data to dummy-sa service account in my-nymespace namespace
[info  kamus-cli]: Encrypted data:
NlHRuvaqFlip5mDP6d5cDw==:OVW3C2hY/wZW6o2I+YjL5Q==

Then we need to create a KamusSecret envelope object based on the following structure (please note that documentation refers to a data section which doesn't seem to be relevant or working, so we've used the stringData syntax).

$ cat <<EOF >kamus.yaml
apiVersion: "soluto.com/v1alpha2"
kind: KamusSecret
metadata:
  name: db-credentials
  namespace: my-namespace
type: Generic
stringData:
  DBUSER: 0XZm9vDLVvjNGCgeFZqNnA==:GoaTZCXkwyfoBNRwjtgsOQ==
  DBPWD: NlHRuvaqFlip5mDP6d5cDw==:OVW3C2hY/wZW6o2I+YjL5Q==
serviceAccount: dummy-sa
EOF

This KamusSecret resource can now be safely stored under a source code versioning system like git.


Apply the new resource:

$ kubectl apply -f kamus.yaml

Check the resources created:

$ kubectl get kamussecrets.soluto.com -n my-namespace 
NAME             AGE
db-credentials   20s
​
$ kubectl get secrets -n my-namespace 
NAME                  TYPE                                  DATA   AGE
default-token-sc4ds   kubernetes.io/service-account-token   3      9m3s
db-credentials        Generic                               2      53s

Verify the decryption:

$ kubectl get secrets -n my-namespace db-credentials -o jsonpath='{.data.DBUSER}' | base64 --decode
mydbuser
​
$ kubectl get secrets -n my-namespace db-credentials -o jsonpath='{.data.DBPWD}' | base64 --decode
highlysecret


Zero-Trust Secrets Usage


In this mode, we will reuse the encrypted data of the first example, but this time we'll use a ConfigMap that we later will use as a mounted volume of encrypted files (DBUSER and DBPWD)

$ cat <<EOF >kamus-encrypted-secrets-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: kamus-encrypted-secrets-cm
  namespace: my-namespace
data:
  DBUSER: 0XZm9vDLVvjNGCgeFZqNnA==:GoaTZCXkwyfoBNRwjtgsOQ==
  DBPWD: NlHRuvaqFlip5mDP6d5cDw==:OVW3C2hY/wZW6o2I+YjL5Q==
EOF

This config map can be safely stored under a git repository.


Next, we'll use the initContainer from Kamus solution which will invoke the decryption services on mounted files, and results will be stored on a common emptyDir volume for our "application" container to use (our application is a sleepy container that we'll consult later on)

$ cat <<EOF >kamus-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  namespace: my-namespace
  name: kamus-zero-trust-example
spec:
   serviceAccountName: dummy-sa
   automountServiceAccountToken: true
   initContainers:
   - name: "kamus-init"
     image: "soluto/kamus-init-container:latest"
     imagePullPolicy: IfNotPresent
     env:
     - name: KAMUS_URL
       value: http://kamus-decryptor.default.svc.cluster.local/ 
     volumeMounts:
     - name: encrypted-secrets
       mountPath: /encrypted-secrets
     - name: decrypted-secrets
       mountPath: /decrypted-secrets
     args: ["-e","/encrypted-secrets","-d","/decrypted-secrets", "-n", "config.json"]
   containers:
   - name: app
     image: busybox
     imagePullPolicy: IfNotPresent
     args: ["sleep", "3600"]
     volumeMounts:
     - name: decrypted-secrets
       mountPath: /secrets
   volumes:
   - name: encrypted-secrets
     configMap: 
       name: kamus-encrypted-secrets-cm
   - name: decrypted-secrets
     emptyDir:
       medium: Memory
EOF

Create our application pod:

$ kubectl create -f kamus-pod.yaml
pod/kamus-zero-trust-example created

And check the decrypted /secrets/config.json file available for our application container:

$ kubectl exec -n my-namespace kamus-zero-trust-example app -- more /secrets/config.json 
{ 
    "DBPWD":"highlysecret",
    "DBUSER":"mydbuser"
}

Notes


Please note that:

  • in the case of a KamusSecret (first method), the --service-account and --namespace arguments for the kamus-cli command line, although mandatory, are purely arbitrary: the encrypted data with a KamusSecret is not linked to a particular service account and/or namespace, and their existence is not a requirement at encryption time.

  • On the contrary, in the case of a Zero-Trust mode (second method), having the service accounts and namespaces consistent between the encryption steps and the ones used inside the initContainer are fundamental. Another service account, or another namespace won't trigger a proper decryption.


Summary


➕ Multiple operating modes

➕ Working in zero-trust environments

➕ Can use a true KMS backend

➖ Confusing documentation

➖ Still early stages

➖ Its npm based client


Conclusion


While Kamus offers more options than Bitnami, it requires substantially more efforts for the most common scenarios. The documentation coming with the solution is also sometimes confusing, although there have been significant efforts on clarity since the early releases. Some customers and audience will be particularly receptive to its ability to operate in a zero-trust environment despite it requires a significant setup and injection of init containers in workloads. It also can use sophisticated and cryptographic secure vault solutions as a backend, which is also a strong argument when this is a requirement.


In most common scenarios however, Bitnami Sealed Secrets remains particularly effective, compact and straightforward to operate. It's been around since quite some time now, is well maintained and is becoming a de-facto answer for lightweight secrets management for GitOps workflows for very good reasons.


As usual, there is no best all-around alternative, but there's always the best answer to a particular use-case with some requirements and constraints from the environment. We hope this article will help you selecting the secrets solution for your GitOps scenarios.

©2020 - SOKUBE SA - GENEVA - SWITZERLAND

linkedin_big.png