Building k3s on a Pi

I have had a Pi laying around used for a simple task for a while now. A few days ago, I was browsing the web, learning more about privacy, when I stumbled upon AdGuard Home.

I have been using it as my internal DNS on top of the security and privacy layers I add to my machine. Its benefits can be argued but it is a DNS after all and I wanted to see what else it can do for me. Anyway, I digress. I searched to see if I could find a container for AdGuard Home and I did.

At this point, I started thinking about what I could do to make the Pi more useful.

That’s when k3s came into the picture.

Pre-requisites

As this is not a Pi tutorial, I am going to be assuming that you have a Raspberry Pi with Raspberry Pi OS Buster installed on it. The assumption does not mean you cannot install any other OS on the Pi and run this setup. It only means that I have tested this on Buster and that your milage will vary.

Prepare the Pi

Now that you have Buster already installed, let’s go ahead and fix a small default configuration issue with it.

K3s uses iptables to route things around correctly. Buster uses nftables by default, let’s switch it to iptables.

 $ sudo iptables -F
 $ sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
 $ sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
 $ sudo reboot

At this point, your Pi should reboot. Your OS is configured for the next step.

Pre-install Configuration

After testing k3s a few times, I found out that by default it will deploy a few extra services like Traefik.

Unfortunately, just like anything the default configuration is just that. It’s plain and not very useful from the start. You will need to tweak it.

This step could be done either post or pre deploy. Figuring out the pre-deploy is a bit more involving but a bit more fun as well.

The first thing you need to know is that the normal behavior of k3s is to deploy anything found in /var/lib/rancher/k3s/server/manifests/. So a good first step is, of course, to proceed with creating that.

 $ mkdir -p /var/lib/rancher/k3s/server/manifests/

The other thing to know is that k3s can deploy Helm Charts. It will create the manifests it will deploy by default, before beginning the setup, in the manifest path I mentioned. If you would like to see what it deployed and how, visit that path after k3s runs. I did, and I took their configuration of Traefik which I was unhappy with its defaults.

My next step was securing the defaults as much as possible and I found out that Traefik can do basic authentication. As a starting point, that’s great. Let’s create the credentials.

 $ htpasswd -c ./auth myUser

That was easy so far. Let’s turn up the notch and create the manifest for k3s.

Create traefik.yaml in /var/lib/rancher/k3s/server/manifests/ with the following content.

---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: traefik
  namespace: kube-system
spec:
  chart: https://%{KUBERNETES_API}%/static/charts/traefik-1.81.0.tgz
  valuesContent: |-
    rbac:
      enabled: true
    ssl:
      enabled: true
    dashboard:
      enabled: true
      domain: traefik-ui.example.com
      auth:
        basic:
          myUser: $ars3$4A5tdstr$trSDDa4467Tsa54sTs.
    metrics:
      prometheus:
        enabled: false
    kubernetes:
      ingressEndpoint:
        useDefaultPublishedService: true
    image: "rancher/library-traefik"
    tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - key: "node-role.kubernetes.io/master"
        operator: "Exists"
        effect: "NoSchedule"    

It’s a Pi, I don’t need prometheus so I disabled it. I also enabled the dashboard and added the credentials we created in the previous step.

Now, the Helm Chart will deploy an ingress and expose the dashboard for you on the value of domain.

K3s

If everything is in place, you are ready to proceed. You can install k3s, now, but before I get to that step, I will say a few things about k3s.

K3s has a smaller feature set than k8s, hence the smaller footprint. Read the documentation to see if you need any of the missing features. The second thing to mention is that k3s is a one binary deploy that uses containerd. That’s why we will use the script installation method as it adds the necessary systemd configuration for us. It is a nice gesture.

Let’s do that, shall we ?

 $ curl -sfL https://get.k3s.io | sh -s - --no-deploy traefik

Point traefik.example.com to your Pi IP in /etc/hosts on your machine.

 traefik.example.com  192.168.0.5

When the installation command is done, you should be able to visit http://traefik.example.com/

You can get the kubeconfig from the Raspberry Pi, you can find it in /etc/rancher/k3s/k3s.yaml. You will need to change the server IP.

Conclusion

If you’ve made it so far, you should have a k3s cluster running on a single Raspberry Pi. The next steps you might want to look into is disable the metrics server and use the resources for other things.