In this article I’ll explain how I used Git and Github Actions to manage a Google Cloud hosted Kubernetes (K8s) cluster that powers an e-commerce website producing over $1 million in revenue per year.
Working with a Kubernetes cluster is no simple task for a small team. An application with several services maps to a dozen or more K8s objects and YAML files. Changes to these files need to be code-reviewed, safe to deploy, and quick to roll back. Luckily, Git makes it easy to track revisions. Let’s dive in to how the Git repository is designed.
Repository Structure #
README.md .github/workflows/kubectl-apply.yml bin/ create-secret.sh read-secret.sh cache/ deployment.yml service.yml image-server/ deployment.yml service.yml web-server/ deployment.yml service.yml ingress.yml secrets/ account/ secret.yml username.enc.b64 password.enc.b64
Each service has a separate folder containing K8s configuration files named after the object each file represents1. In other words,
service.yml holds configuration for a Service. This structure is great because it groups related objects, making it obvious what resources a particular service requires, without introducing any abstraction overhead2. The
web-server/secrets/ directory is special, let me explain.
Deploying infrastructure changes often requires hand-crafted environment variables and configuration files, none of which are checked in. In my opinion, this is an anti-pattern. That said, storing plain-text secrets in source control (even with private Git hosting) is much worse. The solution is to encrypt before you commit.
A secret is now a folder containing a YAML file (with a name and other relevant metadata) and encrypted files stitched together into key-value data. I wrote a wrapper script around gcloud kms encrypt and gcloud kms decrypt to create and read these secrets. The script is portable so it can be used by developers and in automation scripts.
Code Review #
This repository structure enables Infrastructure as Code, a powerful model for working with Kubernetes (and other infrastructure). Good engineering hygiene requires reviewing code; Github makes this easy with pull requests.
After a change is committed to a feature branch, a pull request into the
main branch is created. After the change is reviewed and approved, the pull request is merged which automatically applies the change by way of Github Actions. Reverting the pull request rolls back the change.
Github Actions #
Github Actions is a straightforward way to execute a script when a pull request is merged into
main. I wrote a single workflow that did a few things:
- For each encrypted secret;
- Base-64 decode and decrypt each encrypted file
- Produce a well-formed K8s Secret YAML file with key-value data mapping filenames to file contents
- For each top level directory;
kubectl apply -f .
I created a Google Cloud Service Account with sufficient permissions to run
gcloud kms decrypt and access the K8s cluster. The Service Account key was stored as a Github Actions secret, though it appears there is now better support for authentication3.
Closing Thoughts #
Spending time defining the properties you want to follow when working with K8s (or any infrastructure) is worth the effort. To summarize, this system supports code reviewed configuration, safe deployments with little magic or abstraction overhead. Changes are tracked with Git and are simple to revert and roll back. This system handled dozens of deployments per day and is easily extended to manage pre-production environments.
This was eventually simplified to a single
app.ymlfile containing all the K8s objects needed to deploy a service. ↩︎
In this case the penalty of abstraction overhead is not performance, it’s a combination of comprehensibility and configurability. ↩︎