Turbolift-Compatible K8s Pi Cluster Setup

This checklist includes the setup instructions for a turbolift-compatible pi cluster, and includes additional configuration and tooling (marked with [optional]) that I find useful. It is available as a manual installation checklist, with explanations for each command, as well as in script format.

In short, this setup is a microk8s cluster running on ubuntu with the default microk8s ingress. Turbolift does not rely on any microk8s-specific APIs, and this is not the only turbolift-compatible cluster configuration. This setup requires a 64-bit pi and was built/tested using a pi4 with 4GB of RAM. If you are building a cluster using these instructions, I would recommend going through this installation once and storing the image after completion to be reused without having to re-install for each pi.

Installation Instructions

  1. On your dev device, install the most recent ubuntu 64-bit LTS server OS using raspberry pi imager v1.4 to an SD card. I used 20.04.
  2. Access you pi with an external monitor and keyboard. Connect it to ethernet and power the pi on. The default login is username ubuntu password ubuntu; you will need to change it while logging in. The remaining steps will be run on the pi. Note that, on first boot, there may be a delay of a few minutes before the ubuntu user is generated and ready to be accessed. Also note that some unattended upgrades will be triggered the first time that the system connects to the internet, which may temporarily block access to a lockfile necessary to run install commands through apt/apt-get.
  3. Start by updating the registry: sudo apt-get update
  4. [optional] enable wifi access:
    1. sudo apt-get install network-manager -y (I prefer using nmcli, iwlist etc. to iw directly, though if you don't have any wired connection, you can activate the wifi card using sudo ip link set wlan0 up and then use wpa_supplicant to connect to your network using this example).
    2. Use nmcli d wifi to see available networks.
    3. Connect to your network with nmcli d wifi connect <YOUR SSID HERE> password <YOUR PASSWORD HERE>
    4. If the connection succeeds, make the pi automatically reconnect using nmcli connection modify <YOUR SSID HERE> connection.autoconnect yes
  5. [optional] install k9s for monitoring. This currently needs to be done by building from source. Look for the current go version from the k9s repo and then install it and build k9s. Here are the instructions for building the latest k9s as of writing (hash e5759218e6cf70767399c465891dba5161f8ffea), which uses go1.17:
    sudo apt-get install bison build-essential golang-go -y
    bash <(curl -s -S -L 'https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer' )
    bash -c "source ~/.gvm/scripts/gvm
    gvm install go1.17
    gvm use go1.17 --default
    cd ~ && git clone 'https://github.com/derailed/k9s'
    cd ~/k9s && git checkout e5759218e6cf70767399c465891dba5161f8ffea
    go mod tidy
    make build
    echo 'PATH=$PATH:~/k9s/execs' >> ~/.zshrc
    "; ls ~/k9s/execs/k9s || exit 1
  6. Install rust. I like to set the default toolchain to the same nightly build used in turbolift's current CI, and then add add rust to ~/.zshrc. With the CI's current rust version as of writing, those commands are:
    curl --proto '=https' --tlsv1.2 -sSf 'https://sh.rustup.rs' | sh -s -- -y --default-toolchain nightly-2020-09-28
    echo 'source ~/.cargo/env' >> ~/.zshrc
  7. Setup docker (docker install instructions). Here is a copy of the current instructions, along with the config to setup a local registry:
    sudo apt-get update
    sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
    echo "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
    sudo apt-get update
    sudo apt-get install docker-ce docker-ce-cli containerd.io -y
    echo '{"insecure-registries": ["localhost:32000"]}' | sudo tee /etc/docker/daemon.json
  8. Grant user access to docker with sudo usermod -aG docker $USER
  9. [optional] set docker to clean itself up at least once a day:
    echo 'docker system prune -f' | sudo tee /etc/cron.daily/docker-system-prune
  10. Enable cgroups (needed by microk8s) and restart: cat /boot/firmware/cmdline.txt | tr '\n' ' '| xargs -i% echo "% cgroup_enable=memory cgroup_memory=1" | sudo tee /boot/firmware/cmdline.txt
    sudo reboot
  11. Set up microk8s:
    sudo apt-get install iptables-persistent -y
    sudo snap install microk8s --classic
    sudo usermod -aG microk8s $USER
    /usr/bin/newgrp microk8s <<EOT
    microk8s start & microk8s status --wait-ready
    sudo snap install kubectl --classic
    microk8s enable ingress
    cd ~/.kube && microk8s config > config
  12. [optional] allow user to change their shell without a password and switch to zsh
    sudo apt install zsh -y
    chsh -s /bin/zsh and reboot.
  13. [optional] set the zsh history to retain the last million commands, and to share history across sessions:
    cat <<EOT>> ~/.zshrc
    setopt SHARE_HISTORY
  14. [optional] install powerlevel10k using
    git clone --depth=1 'https://github.com/romkatv/powerlevel10k.git' ~/powerlevel10k
    echo 'source ~/powerlevel10k/powerlevel10k.zsh-theme' >> ~/.zshrc

    The configuration wizard for p10k will automatically start on reboot.
  15. [optional] enable terminal autocomplete with fzf (will take effect on reboot)
    sudo apt install fzf -y
    cat <<EOT>> ~/.zshrc
    source /usr/share/doc/fzf/examples/key-bindings.zsh
    source /usr/share/doc/fzf/examples/completion.zsh
  16. [optional] make a ~10GiB swap file, enable it, and set it to be enabled automatically on boot, but only to be used if no remaining memory exists (note: this takes around 8 minutes to run):
    sudo dd if=/dev/zero of=/swap0 bs=1K count=10M
    sudo chmod 0600 /swap0
    sudo mkswap /swap0
    sudo swapon /swap0
    echo '/swap0 swap swap defaults 0 0' | sudo tee -a /etc/fstab
    sudo sysctl vm.swappiness=0
  17. [optional] make sure that every console is attached to a session at startup. This allows for tiling, scrolling and a bunch of other nice features:
    cat <<EOT>> ~/.zshrc
  18. [optional] increase the maximum scrollback per session, and disable the startup message. Scrollback (Ctrl-A, then escape, then PgUp/PgDown or arrow keys) is useful for commands with verbose output: cat <<EOT>> ~/.screenrc
    defscrollback 20000
    startup_message off
  19. If you haven't rebooted the pi yet since installing powerlevel10k, docker, and microk8s, do so now with sudo reboot
  20. At this point, you have completed the installation for one pi. Repeat all prior steps for each pi you want to add to your cluster. Alternatively, if your SD cards are all the same size, you can clone the image from the SD card you just set up to your other SD cards.
  21. Connect all pis to the same network.
  22. Run microk8s join and follow the instructions to start clustering the pis.

Script Installation

The script completes the above instructions, excluding attempting to connect to a wifi network, enabling cgroups in the firmware, changing the user's shell, or running the microk8s join command. The script triggers a reboot on completion.

To run the script, prepare your freshly imaged pi by enabling cgroups (used by microk8s) and rebooting, using cat /boot/firmware/cmdline.txt | tr '\n' ' '| xargs -i% echo "% cgroup_enable=memory cgroup_memory=1" | sudo tee /boot/firmware/cmdline.txt && sudo reboot

After logging back in, you're ready to run the script. It's available on a designated page and can be applied by running bash <(curl --proto '=https' --tlsv1.2 -sSf 'https://dominic.computer/code?page=/blog/2021/turbolift_pi_k8s_install&lang=Bash' ). Then, run chsh -s /bin/zsh, exit and on login you should be done with installation!