Install kata-containers on K3S on aarch64

To install kata-containers on a raspberry-pi and integrating it into kubernetes a few steps are currently required:

• Install kata-containers
• Setup integration into containerd
• Setup integration into kubernetes

Install kata-containers

Currently the easiest (and apparently only) officially supported way to install kata-containers on an aarch64 system is to use snaps.

If you are using the Ubuntu arm version this is already included in the installation and you can install kata-containers with a single command¹:

1sudo snap install kata-containers --classic

Integration into containerd

When running k3s there is no containerd binary as it is embedded into k3s, so configuration must be performed in the k3s configuration².

The whole process includes the following steps:

• A runtime must be configured for kata-containers in containerd.
• shims must be created for containerd to use kata-containers.
• The node must be configured to be able to run kata-container workloads.
• The kubernetes cluster must be aware that kata is a valid runtime.

To configure a runtime, the configuration file for k3s is found under: /var/lib/rancher/k3s/agent/etc/containerd/config.toml, however it can not be edited as it will be regenerated every time k3s is restarted. Instead it should be copied to /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl and that file can be modified.

1cp /var/lib/rancher/k3s/agent/etc/containerd/config.toml{,.tmpl}

The content should be:

1[plugins.cri.containerd.runtimes.kata]
2  runtime_type = "io.containerd.kata.v2"
3
4[kata.options]
5  ConfigPath = "/etc/kata-containers/configuration.toml"

The configuration files for /etc/kata-containers/*.toml must be copied from the snap, because the default configuration will not be able to start on Raspberry Pis with < 4GB of memory, as qemu requests too much memory by default:

1mkdir -p /etc/kata-containers/
2cp /snap/kata-containers/current/usr/share/defaults/kata-containers/*.toml /etc/kata-containers/

Inside these configuration files (depending on the Pi) make sure to adjust the value of default_memory to something the hardware can handle. You should at least adjust the default configuration.toml and configuration-qemu.toml.

In addition shim files must be created in /usr/local/bin to point to the shim binaries of kata-containers for containerd to pick it up.

The following script will create all the shims (including some that might not actually be supported) - it was copied out of the kata-deploy project.

 1#!/bin/bash
 2shims=(
 3    "fc"
 4    "qemu"
 5    "qemu-virtiofs"
 6    "clh"
 7)
 8
 9for shim in "${shims[@]}"; do
10    shim_binary="containerd-shim-kata-${shim}-v2"
11    shim_file="/usr/local/bin/${shim_binary}"
12    shim_backup="/usr/local/bin/${shim_binary}.bak"
13
14    if [ -f "${shim_file}" ]; then
15        echo "warning: ${shim_binary} already exists" >&2
16        if [ ! -f "${shim_backup}" ]; then
17            mv "${shim_file}" "${shim_backup}"
18        else
19            rm "${shim_file}"
20        fi
21    fi
22    cat << EOT | tee "$shim_file"
23#!/bin/bash
24KATA_CONF_FILE=/etc/kata-containers/configuration.toml /snap/kata-containers/current/usr/bin/containerd-shim-kata-v2 \$@
25EOT
26    chmod +x "$shim_file"
27done
28
29# On the PI a default shim was also needed
30cp /usr/local/bin/containerd-shim-kata-qemu-v2 /usr/local/bin/containerd-shim-kata-v2

Now the k3s-agent must be restarted to reload the containerd configuration and we can test that kata-container runtime is working by running the following commands:

1systemctl restart k3s-agent
2ctr image pull docker.io/library/busybox:latest
3ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh

If the tests succeed the node can now be labeled as supporting kata-containers:

1kubectl label node "$NODE_NAME" --overwrite katacontainers.io/kata-runtime=true

Now as a last step a new RuntimeClass must be created for kata that can be used to force pods to be run using it:

1apiVersion: node.k8s.io/v1beta1
2kind: RuntimeClass
3metadata:
4  name: kata
5handler: kata

Now pods can use this runtime by specifying runtimeClassName inside the spec:

1apiVersion: v1
2kind: Pod
3metadata:
4  name: nginx-untrusted
5spec:
6  runtimeClassName: kata
7  containers:
8  - name: nginx
9    image: nginx