Skip to main content

Kubernetes GitOps

In this article I’ll explain how I used Git and Github Actions to manage a Google Cloud hosted Kubernetes (K8s) cluster that powers an e-commerce website producing over $1 million in revenue per year.

Working with a Kubernetes cluster is no simple task for a small team. An application with several services maps to a dozen or more K8s objects and YAML files. Changes to these files need to be code-reviewed, safe to deploy, and quick to roll back. Luckily, Git makes it easy to track revisions. Let’s dive in to how the Git repository is designed.

Repository Structure #

README.md
.github/workflows/kubectl-apply.yml
bin/
  create-secret.sh
  read-secret.sh
cache/
  deployment.yml
  service.yml
image-server/
  deployment.yml
  service.yml
web-server/
  deployment.yml
  service.yml
  ingress.yml
  secrets/
    account/
      secret.yml
      username.enc.b64
      password.enc.b64

Each service has a separate folder containing K8s configuration files named after the object each file represents1. In other words, service.yml holds configuration for a Service. This structure is great because it groups related objects, making it obvious what resources a particular service requires, without introducing any abstraction overhead2. The web-server/secrets/ directory is special, let me explain.

Secrets #

Deploying infrastructure changes often requires hand-crafted environment variables and configuration files, none of which are checked in. In my opinion, this is an anti-pattern. That said, storing plain-text secrets in source control (even with private Git hosting) is much worse. The solution is to encrypt before you commit.

A secret is now a folder containing a YAML file (with a name and other relevant metadata) and encrypted files stitched together into key-value data. I wrote a wrapper script around gcloud kms encrypt and gcloud kms decrypt to create and read these secrets. The script is portable so it can be used by developers and in automation scripts.

Code Review #

This repository structure enables Infrastructure as Code, a powerful model for working with Kubernetes (and other infrastructure). Good engineering hygiene requires reviewing code; Github makes this easy with pull requests.

Sample pull request
A sample pull request on Github

After a change is committed to a feature branch, a pull request into the main branch is created. After the change is reviewed and approved, the pull request is merged which automatically applies the change by way of Github Actions. Reverting the pull request rolls back the change.

Github Actions #

Github Actions is a straightforward way to execute a script when a pull request is merged into main. I wrote a single workflow that did a few things:

  • For each encrypted secret;
    • Base-64 decode and decrypt each encrypted file
    • Produce a well-formed K8s Secret YAML file with key-value data mapping filenames to file contents
  • For each top level directory;
    • Run kubectl apply -f .

I created a Google Cloud Service Account with sufficient permissions to run gcloud kms decrypt and access the K8s cluster. The Service Account key was stored as a Github Actions secret, though it appears there is now better support for authentication3.

Closing Thoughts #

Spending time defining the properties you want to follow when working with K8s (or any infrastructure) is worth the effort. To summarize, this system supports code reviewed configuration, safe deployments with little magic or abstraction overhead. Changes are tracked with Git and are simple to revert and roll back. This system handled dozens of deployments per day and is easily extended to manage pre-production environments.

If you find yourself nodding your head and want more pointers, please reach out! If you’d like to learn how this system translates to Terraform, keep on reading.


  1. This was eventually simplified to a single app.yml file containing all the K8s objects needed to deploy a service. ↩︎

  2. In this case the penalty of abstraction overhead is not performance, its a combination of comprehensibility and configurability. ↩︎

  3. See google-github-actions/setup-gcloud on Github. ↩︎