Will it cluster? k3s on your Raspberry Pi

In this post we'll test-drive k3s which is a stripped-down Kubernetes distribution from Rancher Labs. With a single binary and a one-line bootstrap process it's even easier than before to create a light-weight cluster. So grab your Raspberry Pi and get ready to deploy the smallest Kubernetes distribution ever.

You may have seen my previous work with Kubernetes and Docker on Raspberry Pi such as Build your own bare-metal ARM cluster. I'm hoping that this post will be a lot simpler to follow, with fewer workarounds and even more resources left over for your projects to consume.

Featured: Raspberry Pi x5 Compute Module (COM) holder with Gigabit ethernet, from mininodes.com

Why k3s?

Darren Shepherd, Chief Architect at Rancher Labs is known for building simple solutions and accessible user-experiences for distributed systems. k3s is one of his latest experiements to reduce the footprint and bootstrap-process of Kubernetes to a single binary.

The k3s binary available on GitHub comes in at around 40mb and bundles all the low-level components required such as containerd, runc and even kubectl. k3s can take the place of kubeadm which started as part of a response from the Kubernetes community to up their game for user-experience of bootstrapping clusters.

kubeadm is now able to create production-ready multi-master clusters, but is not well-suited for the Raspberry Pi. This is because it assumes hosts have high CPU/memory and low-latency. When I ran through the installation for k3s the first time it was several times quicker to boot up than kubeadm, but the important part was that it worked first-time, every time without any manual hacks or troubleshooting.

Note: k3s just like Kubernetes, also works on armhf (Raspberry Pi), ARM64 (Packet/AWS/Scaleway) and x86_64 (regular PCs/VMs).


I'll list the pre-requisites and add some affiliate links to Amazon US.

Clustering parts

If you're running with more than one RPi then buying multiple cases or multiple power adapters can be a false economy.

Prepare the RPi

Let's start the tutorial.

Flash the OS to the SD card

Let's not make things complicated by messing about with bespoke operating systems. The Raspberry Pi team have done a great job with Raspbian and for a headless system Raspbian Lite is easy to use and quick to flash.

On MacOS you can usually type in: sudo touch /Volumes/boot/ssh for this step.

Etcher.io in action

Power-up the device & customise it

Now power-up your device. It will be accessible on your network over ssh using the following command:

$ ssh pi@raspberrypi.local

Log in with the password raspberry and then type in sudo raspi-config.

Update the following:

  • Set the GPU memory split to 16mb
  • Set the hostname to whatever you want (write it down?)
  • Change the password for the pi user

I also highly recommend setting a static IP for each Raspberry Pi in your cluster.

Copy over your ssh key

Do you have an ssh key?

$ ls -l ~/.ssh/id_rsa.pub

If that says file not found, then let's generate a key-pair for use with SSH. This means you can set a complicated password, or disable password login completely and rely on your public key to log into each RPi without typing a password in.

Hit enter to everything:

$ ssh-keygen

Finally run: ssh-copy-id pi@raspberrypi.local

Enable container features

We need to enable container features in the kernel, edit /boot/cmdline.txt and add the following to the end of the line:

 cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory

Now reboot the device.

Create the k3s cluster

Note: during installation kubectl will be aliased to the command k3s kubectl so that we can use the pre-packaged version of kubectl.

If you type in docker after the installation, you won't find the command installed. This is because k3s uses a low-level component called containerd directly.

Bootstrap the k3s server

We can install k3s using a utility script which gets the latest stable version from the releases page and then installs a systemd service to start k3s automatically.

On one of the nodes log-in and run the following:

$ curl -sfL https://get.k3s.io | sh -

Check that the systemd service started correctly:

$ sudo systemctl status k3s

Wait for k3s to start and to download the required images from the Kubernetes registry. This may take a few minutes.

Grab the join key from this node with:

$ sudo cat /var/lib/rancher/k3s/server/node-token


Introducing k3sup (update)

You can now automate the installation and bootstrap of k3s onto any cloud, VM or Raspberry Pi with k3sup.

k3sup gives you access to kubectl in under a minute:

k3sup install --ip $SERVER --user pi
k3sup join --ip $AGENT --server-ip $SERVER --user pi

Try it out to fetch your KUBECONFIG for use from your laptop.

Once you have a KUBECONFIG from your k3s cluster, or any Kubernetes cluster at all you can use k3sup app install to add things like OpenFaaS, inlets-operator, metrics-sever, nginx, and more. Just check which of the apps is available for arm.

Here's an example of how easy it becomes to install OpenFaaS for instance:

k3sup app install openfaas

If you liked k3sup, then please add your Star on GitHub

Join a worker

Now log into another node and download the binary as before, moving it to /usr/local/bin/.

Now join any number of your worker nodes to the server with the following:

$ export K3S_URL=""

$ export K3S_TOKEN="K1089729d4ab5e51a44b1871768c7c04ad80bc6319d7bef5d94c7caaf9b0bd29efc::node:1fcdc14840494f3ebdcad635c7b7a9b7"

$ curl -sfL https://get.k3s.io | sh -

If you installed k3s manually using a binary, then you can join your node to the server in this way:

$ sudo k3s agent --server ${K3S_URL} --token ${K3S_TOKEN}

List your nodes

$ kubectl get node -o wide

cm3    Ready    <none>   9m45s   v1.13.4-k3s.1   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.79-v7+      containerd://1.2.4+unknown
cm4    Ready    <none>   13m     v1.13.4-k3s.1   <none>        Raspbian GNU/Linux 9 (stretch)   4.14.79-v7+      containerd://1.2.4+unknown

We can see our nodes and that they are using containerd rather than full Docker. This is part of how Darren was able to reduce the footprint.

Deploy a microservice

We can now log into the k3s server and deploy a microservice. We'll deploy figlet which will take a body over HTTP on port 8080 and return an ASCII-formatted string.

  • Create a service (with a NodePort):

Save: openfaas-figlet-svc.yaml.

apiVersion: v1
kind: Service
  name: openfaas-figlet
    app: openfaas-figlet
  type: NodePort
    - port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 31111
    app: openfaas-figlet
  • Now create a deployment

The deployment will be used to schedule a Pod using a Docker image published in the OpenFaaS Function Store.

Save: openfaas-figlet-dep.yaml.

apiVersion: apps/v1
kind: Deployment
  name: openfaas-figlet
   app: openfaas-figlet
  replicas: 1
      app: openfaas-figlet
        app: openfaas-figlet
      - name: openfaas-figlet
        image: functions/figlet:latest-armhf
        imagePullPolicy: Always
        - containerPort: 8080
          protocol: TCP
  • Now apply the configuration:
$ kubectl apply -f openfaas-figlet-dep.yaml,openfaas-figlet-svc.yaml

deployment.apps/openfaas-figlet created
service/openfaas-figlet created

Wait for the figlet microservice to come up:

$ kubectl rollout status deploy/openfaas-figlet

deployment "openfaas-figlet" successfully rolled out

Now invoke the function:

echo -n "I like $(uname -m)" | curl --data-binary @-
 ___   _ _ _                                    _____ _
|_ _| | (_) | _____    __ _ _ __ _ __ _____   _|___  | |
 | |  | | | |/ / _ \  / _` | '__| '_ ` _ \ \ / /  / /| |
 | |  | | |   <  __/ | (_| | |  | | | | | \ V /  / / | |
|___| |_|_|_|\_\___|  \__,_|_|  |_| |_| |_|\_/  /_/  |_|

Share your microservice with your friends

You can use an Open Source tool like inlets.dev to create a tunnel to the public Internet for your Raspberry Pi k3s cluster. All you need to do is to create a cheap VPS or EC2 node to get a public IP address that connects back to your cluster.

The advantages of using inlets are that it's Open Source, has no limitations and can use an exit node wherever you like in the world with a custom DNS entry and TLS for free.

Option A) Setup inlets with TLS

Follow the guide for inlets:

HTTPS for your local endpoints with inlets and Caddy

Option B) Use ngrok's free tier

If you only need your tunnel for 7 hours or are happy to pay for a SaaS service, then then ngrok is also very easy to use and well-known amongst developers.

$ wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-arm.zip
$ unzip ngrok-stable-linux-arm.zip
$ sudo mv ngrok /usr/local/bin/
  • Start a HTTP tunnel to the microservice
$ ngrok http

You'll get a web address appear such as:

Forwarding                    https://16f7c980.ngrok.io ->

Now you can share that URL with your friends:

$ curl -SLs --data $(whoami) https://16f7c980.ngrok.io
  __ _| | _____  __
 / _` | |/ _ \ \/ /
| (_| | |  __/>  < 

You can check the logs to see if they tried it:

$ kubectl logs deploy/openfaas-figlet -f

You can even scale up the microservice:

$ kubectl scale deploy/openfaas-figlet --replicas=4

Then find out which nodes the Pods were created on:

$ kubectl get pods -l app=openfaas-figlet -o wide
NAME                               READY   STATUS    RESTARTS   AGE   IP          NODE
openfaas-figlet-8486c9f585-4ks2f   1/1     Running   0          26s   cm4 
openfaas-figlet-8486c9f585-d7kpk   1/1     Running   0          26s   cm3 
openfaas-figlet-8486c9f585-l7x89   1/1     Running   0          10m   cm3 
openfaas-figlet-8486c9f585-nhqj6   1/1     Running   0          25s   cm3 

We're only scratching the surface here. You can see Darren demo k3s and OpenFaaS in a CNCF Webinar below:

Try Serverless with OpenFaaS

One way of looking at your cluster is like a big computer. The cluster is your computer. With that in mind we can deploy OpenFaaS - Serverless Functions Made Simple.

It allows you to define a function or endpoint on Kubernetes in a very short period of time with minimal investment and a low learning curve.

You can deploy OpenFaaS to Kubernetes or k3s on ARM using my tutorial or the documentation:

sudo apt update && sudo apt install -qy git
git clone https://github.com/openfaas/faas-netes
cd faas-netes
kubectl apply -f namespaces.yml,yaml_armhf

The YAML files applied are generated with helm. You can customize the files to turn on authentication.

Once up and running you will be able to access the gateway UI on port 31112 on any of the nodes using their IP address, found with ifconfig.

A complete OpenFaaS online training workshop is available here, designed for PCs / cloud.

See k3s & OpenFaaS auto-scaling on Raspberry Pi 4

In this live walkthrough with my brand new Raspberry Pi 4s, I show you how to install Kubernetes (k3s) to create a cluster and then how to deploy OpenFaaS and see it auto-scale based upon metrics.

Tear down (optional)

Either run:

$ sudo /usr/local/bin/k3s-uninstall.sh

Or stop k3s on each node and remove the data directory.

$ sudo systemctl stop k3s
$ sudo systemctl disable k3s
$ sudo rm -rf /var/lib/rancher

For more information on k3s, see the page on GitHub.


It is very early for k3s on ARM, but at this stage it's certainly more usable than the alternatives. If you're considering building a cluster for tinkering and for learning more about Kubernetes then you can't go wrong with trying k3s.

Next time you setup k3s try out my new tool k3sup. It uses your SSH key to bring up k3s on any cloud, VM, or Raspberry Pi and then downloads a KUBECONFIG file so that you can use kubectl from your own computer.

Did you like the blog post? Follow me on Twitter @alexellisuk for more.

Share your cluster

It would be great to add your build my Readers' Clusters.

Here's how to take part:

  • Write a short blog post on your experiences, learnings or feedback.
  • Or Tweet a photo

Then send a Pull Request to the README.md file of my k8s-on-raspbian repo.

Become an insider

You can become an insider and get insider updates on all of my work with OSS, blogs, links, tweets, videos, events and new features for OpenFaaS. You can become an insider by sponsoring me on GitHub from 5 USD / mo.

Don't miss the next update, sponsor me on GitHub today.

Continue learning more about Kubernetes:

k3s is a compliant Kubernetes distribution which means if you learn k3s, you're learning Kubernetes and as I tweeted earlier last week - it's never too late to start learning Kubernetes and nobody ever got fired for that.

You may also like

For even more - Follow me on Twitter @alexellisuk.

Alex Ellis

Read more posts by this author.

Subscribe to alex ellis' blog

Get the latest posts delivered right to your inbox.

or subscribe via RSS with Feedly!