Take2 — Setting up k8s Cluster with k3s on Raspberry Pi 3 & Pi 4

Back to a teach-myself-something type of project, with setting up K8s Cluster on Raspberry Pi devices. I had a couple of older Raspberry Pi 3s and a recently purchased Pi 4 available for this. My old Pi3 devices had Docker Swarm (yes that one) from a couple of years back. This time around I want to get k8s on the 3 Pi devices.

The steps are fairly simple but it can be a bit of work. You should find blogs out there with decent enough directions. So here goes me adding to that library of blogs to make it easier for the next person trying to set this up.

This is an upgrade from my previous similar blog installing Docker Swarm on a two node Pi-3 cluster https://54.210.73.122/running-docker-swarm-raspberry-pi-3/

Disclaimer: In the real world software developers will not have to setup their own k8s cluster (except for maybe a local 1 node cluster for development purposes). But I find installing your own cluster and deploying applications to it a great way to learn (and re-learn) the k8s primitives. I highlight ‘re-learn’, since honestly I forget some of this if I come back in a month.

What do I have with me (my setup is from a Mac)

Two Raspberry Pi 3’s
One Raspberry Pi 4
Compatible power supplies
(Optional) A case or rack mounts
A network switch so you can connect the devices directly to ethernet by wire. Strongly recommend this to make setup easier.
- Small ethernet cables to run from switch to devices
- One longer ethernet cable to run from your router to the network switch.
SD Cards for the 3 Pi’s
Raspberry Pi Imager
Install Kubectl on your Mac (so you can talk to k8s on the Pi devices)
Highly recommend a power strip to connect all these devices for power.

My setup looks like this

SD Card Image Setup

For each SD Card repeat the following to setup the OS image

Insert the SD card into your Mac
Spin up the Raspberry Pi Imager
Install the OS Lite with no desktop environment. I choose this since I plan to run the Pi’s in headless mode. Meaning no keyboard or monitor attached directly. Also not having the desktop will save me some space — don’t need all those bytes wasting capacity & compute on the Pi.
Once the SD card is ready go to the boot folder (mostly the only folder accessible). Drop in an empty file named “ssh” in there. No extension , we just need a file with that name. What this will do is to allow you to SSH into the headless Pi. Without this the Pi is dead to you — and you will need a keyboard & monitor.

Boot Up & SSH

Repeat one by one for each device…

Insert the SD card into Pi.
Power it up and go get some coffee. It could take a min or two (or more).
Once enabled go to your router settings and look for the hostname or IP addresses of each device OR install a lan sniffing tool such as LanScan (on Mac) and look for that info
- The new device shows up on my network as raspberrypi.local. Read further down to change hostname.
Once fully setup you can SSH into each device using ssh pi@hostname (where hostname is the IP address or hostname of your device). You will be asked for the password. Default password is ‘raspberry’.
Feel free to upgrade software
- sudo apt update
- sudo apt upgrade
If you prefer not to keep entering the password then copy over public key from a keypair into the Pi using command . Assumes you have a keypair created. If not create one using keygen and then copy over the public key to the Pi device
- sh-copy-id -f -i ~/.ssh/id_rsa.pub pi@$<IP_ADDRESS_OF_NEW_PI_DEVICE>
  
  1
  
  sh-copy-id -f -i ~/.ssh/id_rsa.pub pi@$<IP_ADDRESS_OF_NEW_PI_DEVICE>
Now let’s rename the hostname for the Pi from the default name to something you like. In my case I renamed them Pi1 , Pi2 and Pi3 . Not very imaginative, but works for me.
- ssh into the Pi using the previous instructions
- Run command – sudo raspi-config
- (optional – I did not do this) Go to Network Options -> Wireless Lan, to setup Wifi connectivity
- Go to Network Options -> Hostname to change hostname to desired name
- Now reboot using command (after this you no longer need the ethernet cable)
  - sudo reboot

Post the above you should be running 3 Pi devices in headless mode and able to SSH into them successfully. Do not proceed further if you cannot. You will just be wasting your time.

Assign Static IP Address (Optional – I skipped this)

Edit the file /etc/dhcpcd.conf. Add the following to the bottom of the file. For the ip_address choose the IP address you want to assign to the device. You will need to find the routers and DNS server for your network router. Check router settings locally to get to them. Routers and DNS may be different, as was the case for me.

interface wlan0 (or eth0 for wired connection)
static ip_address=192.168.1.92/24  (REPLACE WITH IP YOU WANT TO ASSIGN TO THIS DEVICE)
static routers=192.168.x.x (REPLACE WITH YOUR VALUE)
static domain_name_servers=192.168.x.x (REPLACE WITH YOUR VALUE)

interface wlan0 (or eth0 for wired connection)

static ip_address=192.168.1.92/24 (REPLACE WITH IP YOU WANT TO ASSIGN TO THIS DEVICE)

static routers=192.168.x.x (REPLACE WITH YOUR VALUE)

static domain_name_servers=192.168.x.x (REPLACE WITH YOUR VALUE)

Once saved, do a sudo reboot and your Pi will now use the static IP address

K3s

So now to install Kubernetes. The easiest approach I found was to use k3sup to install a lightweight version of k8s called k3s. K3s is a lightweight version of K8s that is CNCF compliant and easy to install. K3sup makes installing k2s even easier.

Excerpt from k3sup github page – “k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.“

K3s comes with what is needed to get up and running fast. It includes Traefik as the default Ingress Controller.

Some basics

The smallest deployment unit in k8s is a pod.
A pod can contain one or more containers (docker or other cncf compatible containers). typically a pod may only contain one app service container/
A Deployment is used to describe the desired state you would like for your application , such as
1. How many pods would you like to deploy (replicaset)?
2. Which container image would you like to deploy?
A Service defines an abstract way to target all your pods (as defined by the Deployment). If you deployment specifies 3 pods then the Service will expose an internal IP address that will load balance between the 3 pods.
Finally the Ingress Controller with Traefik. While the Service object provides a load balanced endpoint, it is only accessible from within the Cluster. If you want your application to be accessible from outside the cluster then you need an Ingress Controller to target external traffic to an internal Service target.

The diagram below tries to show the relationship at an abstract level (it is not a network diagram).

Install k3sup on Host Machine

Install k3sup on your laptop (not the Raspberry Pi where you plan to install k3s). k3sup GitHub page describes the tool as…(and boy were they right)

k3sup is a light-weight utility to get from zero to KUBECONFIG with k3s on any local or remote VM. All you need is ssh access and the k3sup binary to get kubectl access immediately.

install curl -sLS https://get.k3sup.dev | sh uninstall /usr/local/bin/k3s-uninstall.sh

1
2
3
4
5

install
curl -sLS https://get.k3sup.dev | sh

uninstall
/usr/local/bin/k3s-uninstall.sh

Installing Kubernetes on Raspberry Pi device ONE

You will need the IP addresses that were dynamically assigned by your router to the Pi devices. For ease of use set them in env variables.

export PI1_IP="192.168.1.207"
export PI2_IP="192.168.1.165"
export PI3_IP="192.168.1.166"

export PI1_IP="192.168.1.207"

export PI2_IP="192.168.1.165"

export PI3_IP="192.168.1.166"

I plan to use PI1_IP device as the master in the k8s cluster.

k3sup install --ip $PI1_IP --user pi --ssh-key ~/.ssh/id_rsa.pem
export KUBECONFIG=/Users/mathew/temp/k3sup/kubeconfig
kubectl get node -o wide

k3sup install --ip $PI1_IP --user pi --ssh-key ~/.ssh/id_rsa.pem

export KUBECONFIG=/Users/mathew/temp/k3sup/kubeconfig

kubectl get node -o wide

Installing on device#2 & device#3

k3sup join --ip $PI2_IP --server-ip $PI1_IP --user pi --ssh-key ~/.ssh/id_rsa.pem
k3sup join --ip $PI3_IP --server-ip $PI1_IP --user pi --ssh-key ~/.ssh/id_rsa.pem

1 2	k3sup join --ip $PI2_IP --server-ip $PI1_IP --user pi --ssh-key ~/.ssh/id_rsa.pem k3sup join --ip $PI3_IP --server-ip $PI1_IP --user pi --ssh-key ~/.ssh/id_rsa.pem

Verify Cluster Setup

To verify at either step, run ‘kubectl get node -o wide‘ to view info about the cluster. If all was good you should see something like…

Install the Kubernetes Dashboard

You have two options here. One is to use Arkade to install the k8s dashboard and the other is to follow the instructions at https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/

I used Arkade which is a CLI to install charts & apps to your cluster.

curl -sLS https://dl.get-arkade.dev | sudo sh
arkade install kubernetes-dashboard

1 2	curl -sLS https://dl.get-arkade.dev \| sudo sh arkade install kubernetes-dashboard

Follow the instructions that arkade spits out to bring up the k8s dashboard

Now when I did this, sure it installed k8s dashboard, but my dashboard was not able to access the cluster details due to lack of permissions. So go to https://rancher.com/docs/k3s/latest/en/installation/kube-dashboard/ and run through the steps required to set up the RBAC configuration. Now when you follow the instructions to get the token and use that in the specified dashboard URL — life is good and you see live data from your cluster.

Run a Sample App

For this let’s use a simple docker image I have uploaded to gcr.io/contactszoid/hellozoid. It simply prints a message.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
  labels:
    app: myapp
spec:
  selector:
    matchLabels:
      app: myapp
  replicas: 2
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
        - name: hellozoid
          image: gcr.io/contactszoid/hellozoid:latest
          ports:
            - containerPort: 8080
              protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: myapp-svc
spec:
  selector:
    app: myapp
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: myapp-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: myapp-svc
          servicePort: 8080

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: myapp

spec:

selector:

matchLabels:

app: myapp

replicas: 2

template:

metadata:

labels:

app: myapp

spec:

containers:

- name: hellozoid

image: gcr.io/contactszoid/hellozoid:latest

ports:

- containerPort: 8080

protocol: TCP

---

apiVersion: v1

kind: Service

metadata:

spec:

selector:

app: myapp

ports:

- port: 8080

targetPort: 8080

protocol: TCP

---

apiVersion: networking.k8s.io/v1beta1

kind: Ingress

metadata:

annotations:

kubernetes.io/ingress.class: traefik

spec:

rules:

- http:

paths:

- path: /

backend:

serviceName: myapp-svc

servicePort: 8080

To run this

# to apply the configuration
kubectl apply -f hello-application.yaml

# to delete it 
# kubectl delete -f hello-application.yaml

# to apply the configuration

kubectl apply -f hello-application.yaml

# to delete it

# kubectl delete -f hello-application.yaml

Give it a few seconds or use following commands to track deployment

Once all the objects are deployed hit any one of your device IP addresses with the curl command

curl http://192.168.1.166/

# response should be
Hello Blue World!

curl http://192.168.1.166/

# response should be

Hello Blue World!

To delete the application (Deployment, Service & Ingress Controller) run

kubectl delete -f hello-application.yaml

1	kubectl delete -f hello-application.yaml

(end)

{"Mat's Random Thoughts"}

Mathew's Tech Notes..

Take2 — Setting up k8s Cluster with k3s on Raspberry Pi 3 & Pi 4

What do I have with me (my setup is from a Mac)

SD Card Image Setup

Boot Up & SSH

Assign Static IP Address (Optional – I skipped this)

K3s

Install k3sup on Host Machine

Installing Kubernetes on Raspberry Pi device ONE

Installing on device#2 & device#3

Verify Cluster Setup

Install the Kubernetes Dashboard

Run a Sample App