K3S + K3D = K8S : a new perfect match for dev and test

Dernière mise à jour : 8 sept. 2021

Working with Kubernetes on a local machine when you are a Dev or an Ops is not as easy as we could think. So, how to easily create a local Kubernetes cluster that would meet these needs ? At SoKube we heavily use k3d and k3s for these purposes.



More than a year ago I presented in a previous blog what is k3d (with k3s) and how to use it. In the meantime, k3d has been completely rewritten. The goals of this blog post are to show:

  • What is k3d/k3s

  • What’s new with k3d v3

  • Create a simple kubernetes cluster on your local machine

  • Create a multi-server (masters) and multi-agent (workers) kubernetes cluster on your local machine

  • Create a cluster with a specific Kubernetes version

  • How to replace the default CNI plugin of k3s

  • How to replace the default ingress controller of k3s

  • How to use a dedicated registry to download images with k3s

  • What are the other Alternatives


k3s/k3d ?


k3s is a very efficient and lightweight fully compliant Kubernetes distribution. k3d is a utility designed to easily run k3s in Docker, it provides a simple CLI to create, run, delete a fully compliance Kubernetes cluster with 1 to n nodes.

K3s includes:

  • Flannel: a very simple L2 overlay network that satisfies the Kubernetes requirements. This is a CNI plugin (Container Network Interface), such as Calico, Romana, Weave-net Flannel doesn’t support Kubernetes Network Policy, but it can be replaced by Calico (see next sections).

  • CoreDNS: a flexible, extensible DNS server that can serve as the Kubernetes cluster DNS

  • Traefik is a modern HTTP reverse proxy and load balancer. In a next section, I will also show how to replace it either by Traefik v2 or Nginx

  • Klipper Load Balancer : Service load balancer that uses available host ports.

  • SQLite3: The storage backend used by default (also support MySQL, Postgres, and etcd3)

  • Containerd is a runtime container like Docker without the image build part

The choices of these components were made to have the most lightweight distribution. But as we will see later in this blog, k3s is a modular distribution where components can easily be replaced.


Recently k3s has joined the Cloud Native Computing Foundation (CNCF) at the sandbox level as first Kubernetes Distribution (raising a lot of debates whether or not k3s should be a kubernetes sub-project instead).



Installation


Installation is very easy and available through many installers: wget, curl, Homebrew, Aur, … and supports all well known OSes (linux, darwin, windows) and processor architectures (386, amd64) !

Note that you only need to install the k3d client, which will create a k3s cluster using the right Docker image.


Once installed, configure the completion with your preferred shell (bash, zsh, powershell), for instance with zsh:

k3d completion zsh > ~/.zsh/completions/_k3d
source .zshrc



What’s new with k3d v3


In one year, the k3d team did a great job and completely rewrote k3d v3. It is therefore not a simple major version, they have implemented new concepts and structures to make it an evolving tool with very practical and interesting features.

  • New terminology of k3d and k3s: To be as inclusive to the community as possible, "Server" and "Agent" words are now used to design "master" and "worker" node.

  • Every cluster you create will now spawn at least 2 containers: 1 load balancer and 1 “server” node. The load balancer will be the access point to the Kubernetes API, so even for multi-server clusters, you only need to expose a single api port. The load balancer will then take care of proxying your requests to the correct server node. (can be disabled with the --no-lb flag)

  • Adoption of the “NOUN VERB” synt