Take2 — Setting up k8s Cluster with k3s on Raspberry Pi 3 & Pi 4

Back to a teach-myself-something type of project, with setting up K8s Cluster on Raspberry Pi devices. I had a couple of older Raspberry Pi 3s and a recently purchased Pi 4 available for this. My old Pi3 devices had Docker Swarm (yes that one) from a couple of years back. This time around I want to get k8s on the 3 Pi devices.

The steps are fairly simple but it can be a bit of work. You should find blogs out there with decent enough directions. So here goes me adding to that library of blogs to make it easier for the next person trying to set this up.

This is an upgrade from my previous similar blog installing Docker Swarm on a two node Pi-3 cluster https://54.210.73.122/running-docker-swarm-raspberry-pi-3/

Disclaimer: In the real world software developers will not have to setup their own k8s cluster (except for maybe a local 1 node cluster for development purposes). But I find installing your own cluster and deploying applications to it a great way to learn (and re-learn) the k8s primitives. I highlight ‘re-learn’, since honestly I forget some of this if I come back in a month.

What do I have with me (my setup is from a Mac)

  1. Two Raspberry Pi 3’s
  2. One Raspberry Pi 4
  3. Compatible power supplies
  4. (Optional) A case or rack mounts
  5. A network switch so you can connect the devices directly to ethernet by wire. Strongly recommend this to make setup easier.
    • Small ethernet cables to run from switch to devices
    • One longer ethernet cable to run from your router to the network switch.
  6. SD Cards for the 3 Pi’s
  7. Raspberry Pi Imager
  8. Install Kubectl on your Mac (so you can talk to k8s on the Pi devices)
  9. Highly recommend a power strip to connect all these devices for power.

My setup looks like this

SD Card Image Setup

For each SD Card repeat the following to setup the OS image

  1. Insert the SD card into your Mac
  2. Spin up the Raspberry Pi Imager
  3. Install the OS Lite with no desktop environment. I choose this since I plan to run the Pi’s in headless mode. Meaning no keyboard or monitor attached directly. Also not having the desktop will save me some space — don’t need all those bytes wasting capacity & compute on the Pi.
  4. Once the SD card is ready go to the boot folder (mostly the only folder accessible). Drop in an empty file named “ssh” in there. No extension , we just need a file with that name. What this will do is to allow you to SSH into the headless Pi. Without this the Pi is dead to you — and you will need a keyboard & monitor.

Boot Up & SSH

Repeat one by one for each device…

  1. Insert the SD card into Pi.
  2. Power it up and go get some coffee. It could take a min or two (or more).
  3. Once enabled go to your router settings and look for the hostname or IP addresses of each device OR install a lan sniffing tool such as LanScan (on Mac) and look for that info
    • The new device shows up on my network as raspberrypi.local. Read further down to change hostname.
  4. Once fully setup you can SSH into each device using ssh pi@hostname  (where hostname is the IP address or hostname of your device). You will be asked for the password. Default password is ‘raspberry’.
  5. Feel free to upgrade software
    • sudo apt update
    • sudo apt upgrade
  6. If you prefer not to keep entering the password then copy over public key from a keypair into the Pi using command . Assumes you have a keypair created. If not create one using keygen and then copy over the public key to the Pi device
  7. Now let’s rename the hostname for the Pi from the default name to something you like. In my case I renamed them Pi1 , Pi2 and Pi3 . Not very imaginative, but works for me.
    • ssh into the Pi using the previous instructions
    • Run command – sudo raspi-config
    • (optional – I did not do this) Go to Network Options -> Wireless Lan, to setup Wifi connectivity
    • Go to Network Options -> Hostname to change hostname to desired name
    • Now reboot using command (after this you no longer need the ethernet cable)
      • sudo reboot

Post the above you should be running 3 Pi devices in headless mode and able to SSH into them successfully. Do not proceed further if you cannot. You will just be wasting your time.

Assign Static IP Address (Optional  – I skipped this)

Edit the file /etc/dhcpcd.conf. Add the following to the bottom of the file. For the ip_address choose the IP address you want to assign to the device. You will need to find the routers and DNS server for your network router. Check router settings locally to get to them. Routers and DNS may be different, as was the case for me.

Once saved, do a sudo reboot and your Pi will now use the static IP address

K3s

So now to install Kubernetes. The easiest approach I found was to use k3sup to install a lightweight version of k8s called k3s. K3s is a lightweight version of K8s that is CNCF compliant and easy to install. K3sup makes installing k2s even easier.

Excerpt from k3sup github page – “k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

K3s comes with what is needed to get up and running fast. It includes Traefik as the default Ingress Controller.

Some basics

  1. The smallest deployment unit in k8s is a pod.
  2. A pod can contain one or more containers (docker or other cncf compatible containers). typically a pod may only contain one app service container/
  3. A Deployment is used to describe the desired state you would like for your application , such as
    1. How many pods would you like to deploy (replicaset)?
    2. Which container image would you like to deploy?
  4. A Service defines an abstract way to target all your pods (as defined by the Deployment). If you deployment specifies 3 pods then the Service will expose an internal IP address that will load balance between the 3 pods.
  5. Finally the Ingress Controller with Traefik. While the Service object provides a load balanced endpoint, it is only accessible from within the Cluster. If you want your application to be accessible from outside the cluster then you need an Ingress Controller to target external traffic to an internal Service target.

The diagram below tries to show the relationship at an abstract level (it is not a network diagram).

Install k3sup on Host Machine

Install k3sup on your laptop (not the Raspberry Pi where you plan to install k3s). k3sup GitHub page describes the tool as…(and boy were they right)

k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

Installing Kubernetes on Raspberry Pi device ONE

You will need the IP addresses that were dynamically assigned by your router to the Pi devices. For ease of use set them in env variables.

I plan to use PI1_IP device as the master in the k8s cluster.

Installing on device#2 & device#3

Verify Cluster Setup

To verify at either step, run ‘kubectl get node -o wide‘ to view info about the cluster. If all was good you should see something like…

Install the Kubernetes Dashboard

You have two options here. One is to use Arkade to install the k8s dashboard and the other is to follow the instructions at https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/

I used Arkade which is a CLI to install charts & apps to your cluster.

Follow the instructions that arkade spits out to bring up the k8s dashboard

Now when I did this, sure it installed k8s dashboard, but my dashboard was not able to access the cluster details due to lack of permissions. So go to https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/ and run through the steps required to set up the RBAC configuration. Now when you follow the instructions to get the token and use that in the specified dashboard URL — life is good and you see live data from your cluster.

Run a Sample App

For this let’s use a simple docker image I have uploaded to gcr.io/contactszoid/hellozoid. It simply prints a message.

To run this

Give it a few seconds or use following commands to track deployment

Once all the objects are deployed hit any one of your device IP addresses with the curl command

To delete the application (Deployment, Service & Ingress Controller) run

(end)