Turbolift-Compatible K8s Pi Cluster Setup

This checklist includes the setup instructions for a turbolift-compatible pi cluster, and includes additional configuration and tooling (marked with [optional]) that I find useful. It is available as a manual installation checklist, with explanations for each command, as well as in script format.

In short, this setup is a microk8s cluster running on ubuntu with the default microk8s ingress. Turbolift does not rely on any microk8s-specific APIs, and this is not the only turbolift-compatible cluster configuration. This setup requires a 64-bit pi and was built/tested using a pi4 with 4GB of RAM. If you are building a cluster using these instructions, I would recommend going through this installation once and storing the image after completion to be reused without having to re-install for each pi.

Installation Instructions

1. On your dev device, install the most recent ubuntu 64-bit LTS server OS using raspberry pi imager v1.4 to an SD card. I used 20.04.
2. Access you pi with an external monitor and keyboard. Connect it to ethernet and power the pi on. The default login is username ubuntu password ubuntu; you will need to change it while logging in. The remaining steps will be run on the pi. Note that, on first boot, there may be a delay of a few minutes before the ubuntu user is generated and ready to be accessed. Also note that some unattended upgrades will be triggered the first time that the system connects to the internet, which may temporarily block access to a lockfile necessary to run install commands through apt/apt-get.
3. Start by updating the registry: sudo apt-get update
4. [optional] enable wifi access:
   1. sudo apt-get install network-manager -y (I prefer using nmcli, iwlist etc. to iw directly, though if you don't have any wired connection, you can activate the wifi card using sudo ip link set wlan0 up and then use wpa_supplicant to connect to your network using this example).
   2. Use nmcli d wifi to see available networks.
   3. Connect to your network with nmcli d wifi connect <YOUR SSID HERE> password <YOUR PASSWORD HERE>
   4. If the connection succeeds, make the pi automatically reconnect using nmcli connection modify <YOUR SSID HERE> connection.autoconnect yes
5. [optional] install k9s for monitoring. This currently needs to be done by building from source. Look for the current go version from the k9s repo and then install it and build k9s. Here are the instructions for building the latest k9s as of writing (hash e5759218e6cf70767399c465891dba5161f8ffea), which uses go1.17:
   sudo apt-get install bison build-essential golang-go -y
bash <(curl -s -S -L 'https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer' )
bash -c "source ~/.gvm/scripts/gvm
gvm install go1.17
gvm use go1.17 --default
cd ~ && git clone 'https://github.com/derailed/k9s'
cd ~/k9s && git checkout e5759218e6cf70767399c465891dba5161f8ffea
go mod tidy
make build
echo 'PATH=$PATH:~/k9s/execs' >> ~/.zshrc
"; ls ~/k9s/execs/k9s || exit 1
6. Install rust. I like to set the default toolchain to the same nightly build used in turbolift's current CI, and then add add rust to ~/.zshrc. With the CI's current rust version as of writing, those commands are:
   curl --proto '=https' --tlsv1.2 -sSf 'https://sh.rustup.rs' | sh -s -- -y --default-toolchain nightly-2020-09-28
echo 'source ~/.cargo/env' >> ~/.zshrc
7. Setup docker (docker install instructions). Here is a copy of the current instructions, along with the config to setup a local registry:
   sudo apt-get update
sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io -y
echo '{"insecure-registries": ["localhost:32000"]}' | sudo tee /etc/docker/daemon.json

8. Grant user access to docker with sudo usermod -aG docker $USER
9. [optional] set docker to clean itself up at least once a day:
   echo 'docker system prune -f' | sudo tee /etc/cron.daily/docker-system-prune
10. Enable cgroups (needed by microk8s) and restart: cat /boot/firmware/cmdline.txt | tr '\n' ' '| xargs -i% echo "% cgroup_enable=memory cgroup_memory=1" | sudo tee /boot/firmware/cmdline.txt
sudo reboot
11. Set up microk8s:
    sudo apt-get install iptables-persistent -y
sudo snap install microk8s --classic
sudo usermod -aG microk8s $USER
/usr/bin/newgrp microk8s <<EOT
microk8s start & microk8s status --wait-ready
sudo snap install kubectl --classic
microk8s enable ingress
cd ~/.kube && microk8s config > config
EOT

12. [optional] allow user to change their shell without a password and switch to zsh
    sudo apt install zsh -y
chsh -s /bin/zsh and reboot.
13. [optional] set the zsh history to retain the last million commands, and to share history across sessions:
    cat <<EOT>> ~/.zshrc
HISTFILE="$HOME/.zsh_history"
HISTSIZE=1000000
SAVEHIST=1000000
setopt SHARE_HISTORY
EOT
14. [optional] install powerlevel10k using
    git clone --depth=1 'https://github.com/romkatv/powerlevel10k.git' ~/powerlevel10k
echo 'source ~/powerlevel10k/powerlevel10k.zsh-theme' >> ~/.zshrc
    The configuration wizard for p10k will automatically start on reboot.
    
15. [optional] enable terminal autocomplete with fzf (will take effect on reboot)
    sudo apt install fzf -y
cat <<EOT>> ~/.zshrc
source /usr/share/doc/fzf/examples/key-bindings.zsh
source /usr/share/doc/fzf/examples/completion.zsh
EOT
16. [optional] make a ~10GiB swap file, enable it, and set it to be enabled automatically on boot, but only to be used if no remaining memory exists (note: this takes around 8 minutes to run):
    sudo dd if=/dev/zero of=/swap0 bs=1K count=10M
sudo chmod 0600 /swap0
sudo mkswap /swap0
sudo swapon /swap0
echo '/swap0 swap swap defaults 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=0
17. [optional] make sure that every console is attached to a session at startup. This allows for tiling, scrolling and a bunch of other nice features:
    cat <<EOT>> ~/.zshrc
screen
EOT
18. [optional] increase the maximum scrollback per session, and disable the startup message. Scrollback (Ctrl-A, then escape, then PgUp/PgDown or arrow keys) is useful for commands with verbose output: cat <<EOT>> ~/.screenrc
defscrollback 20000
startup_message off
EOT
19. If you haven't rebooted the pi yet since installing powerlevel10k, docker, and microk8s, do so now with sudo reboot
20. At this point, you have completed the installation for one pi. Repeat all prior steps for each pi you want to add to your cluster. Alternatively, if your SD cards are all the same size, you can clone the image from the SD card you just set up to your other SD cards.
21. Connect all pis to the same network.
22. Run microk8s join and follow the instructions to start clustering the pis.

Script Installation

The script completes the above instructions, excluding attempting to connect to a wifi network, enabling cgroups in the firmware, changing the user's shell, or running the microk8s join command. The script triggers a reboot on completion.

To run the script, prepare your freshly imaged pi by enabling cgroups (used by microk8s) and rebooting, using cat /boot/firmware/cmdline.txt | tr '\n' ' '| xargs -i% echo "% cgroup_enable=memory cgroup_memory=1" | sudo tee /boot/firmware/cmdline.txt && sudo reboot

After logging back in, you're ready to run the script. It's available on a designated page and can be applied by running bash <(curl --proto '=https' --tlsv1.2 -sSf 'https://dominic.computer/code?page=/blog/2021/turbolift_pi_k8s_install&lang=Bash' ). Then, run chsh -s /bin/zsh, exit and on login you should be done with installation!