This checklist includes the setup instructions for a [turbolift](https://dominic.computer/turbolift)-compatible pi cluster, and includes additional configuration and tooling (marked with [optional]) that I find useful. It is available as a [manual installation checklist](#manual), with explanations for each command, as well as in [script format](#script).

In short, this setup is a microk8s cluster running on ubuntu with the default microk8s ingress. Turbolift does not rely on any microk8s-specific APIs, and this is not the only turbolift-compatible cluster configuration. This setup requires a 64-bit pi and was built/tested using a pi4 with 4GB of RAM. If you are building a cluster using these instructions, I would recommend going through this installation once and storing the image after completion to be reused without having to re-install for each pi.

## Installation Instructions {#manual}

1. On your dev device, install the most recent ubuntu 64-bit LTS server OS using [raspberry pi imager v1.4](https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/) to an SD card. I used 20.04.

2. Access you pi with an external monitor and keyboard. Connect it to ethernet and power the pi on. The default login is username ubuntu password ubuntu; you will need to change it while logging in. The remaining steps will be run on the pi. Note that, on first boot, there may be a delay of a few minutes before the ubuntu user is generated and ready to be accessed. Also note that some unattended upgrades will be triggered the first time that the system connects to the internet, which may temporarily block access to a lockfile necessary to run install commands through apt/apt-get.

3. Start by updating the registry: `sudo apt-get update`

4. [optional] enable wifi access:
   1. `sudo apt-get install network-manager -y` (I prefer using nmcli, iwlist etc. to iw directly, though if you don't have any wired connection, you can activate the wifi card using `sudo ip link set wlan0 up` and then use wpa_supplicant to connect to your network using [this example](https://askubuntu.com/a/138476)).
   2. Use `nmcli d wifi` to see available networks.
   3. Connect to your network with `nmcli d wifi connect <YOUR SSID HERE> password <YOUR PASSWORD HERE>`
   4. If the connection succeeds, make the pi automatically reconnect using `nmcli connection modify <YOUR SSID HERE> connection.autoconnect yes`

5. [optional] install k9s for monitoring. This currently needs to be done by building from source. Look for the current go version from the [k9s repo](https://github.com/derailed/k9s) and then install it and build k9s. Here are the instructions for building the latest k9s as of writing (hash e5759218e6cf70767399c465891dba5161f8ffea), which uses go1.17:
   ```bash
   sudo apt-get install bison build-essential golang-go -y
   bash <(curl -s -S -L 'https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer')
   bash -c "source ~/.gvm/scripts/gvm
   gvm install go1.17
   gvm use go1.17 --default
   cd ~ && git clone 'https://github.com/derailed/k9s'
   cd ~/k9s && git checkout e5759218e6cf70767399c465891dba5161f8ffea
   go mod tidy
   make build
   echo 'PATH=$PATH:~/k9s/execs' >> ~/.zshrc
   "; ls ~/k9s/execs/k9s || exit 1
   ```

6. Install rust. I like to set the default toolchain to the same nightly build used in turbolift's [current CI](https://github.com/DominicBurkart/turbolift/blob/main/.github/workflows/examples.yml), and then add add rust to ~/.zshrc. With the CI's current rust version as of writing, those commands are:
   ```bash
   curl --proto '=https' --tlsv1.2 -sSf 'https://sh.rustup.rs' | sh -s -- -y --default-toolchain nightly-2020-09-28
   echo 'source ~/.cargo/env' >> ~/.zshrc
   ```

7. Setup docker ([docker install instructions](https://docs.docker.com/engine/install/ubuntu/)). Here is a copy of the current instructions, along with the config to setup a local registry:
   ```bash
   sudo apt-get update
   sudo apt-get install apt-transport-https ca-certificates curl gnupg lsb-release -y
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
   echo "deb [arch=arm64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
   sudo apt-get update
   sudo apt-get install docker-ce docker-ce-cli containerd.io -y
   echo '{"insecure-registries": ["localhost:32000"]}' | sudo tee /etc/docker/daemon.json
   ```

8. Grant user access to docker with `sudo usermod -aG docker $USER`

9. [optional] set docker to clean itself up at least once a day:
   ```bash
   echo 'docker system prune -f' | sudo tee /etc/cron.daily/docker-system-prune
   ```

10. Enable cgroups (needed by microk8s) and restart:
    ```bash
    cat /boot/firmware/cmdline.txt | tr '\n' ' '| xargs -i% echo "% cgroup_enable=memory cgroup_memory=1" | sudo tee /boot/firmware/cmdline.txt
    sudo reboot
    ```

11. Set up microk8s:
    ```bash
    sudo apt-get install iptables-persistent -y
    sudo snap install microk8s --classic
    sudo usermod -aG microk8s $USER
    /usr/bin/newgrp microk8s <<EOT
    microk8s start & microk8s status --wait-ready
    sudo snap install kubectl --classic
    microk8s enable ingress
    cd ~/.kube && microk8s config > config
    EOT
    ```

12. [optional] allow user to change their shell without a password and switch to zsh:
    ```bash
    sudo apt install zsh -y
    chsh -s /bin/zsh
    ```
    and reboot.

13. [optional] set the zsh history to retain the last million commands, and to share history across sessions:
    ```bash
    cat <<EOT>> ~/.zshrc
    HISTFILE="$HOME/.zsh_history"
    HISTSIZE=1000000
    SAVEHIST=1000000
    setopt SHARE_HISTORY
    EOT
    ```

14. [optional] install powerlevel10k using:
    ```bash
    git clone --depth=1 'https://github.com/romkatv/powerlevel10k.git' ~/powerlevel10k
    echo 'source ~/powerlevel10k/powerlevel10k.zsh-theme' >> ~/.zshrc
    ```
    The configuration wizard for p10k will automatically start on reboot.

15. [optional] enable terminal autocomplete with fzf (will take effect on reboot):
    ```bash
    sudo apt install fzf -y
    cat <<EOT>> ~/.zshrc
    source /usr/share/doc/fzf/examples/key-bindings.zsh
    source /usr/share/doc/fzf/examples/completion.zsh
    EOT
    ```

16. [optional] make a ~10GiB swap file, enable it, and set it to be enabled automatically on boot, but only to be used if no remaining memory exists (note: this takes around 8 minutes to run):
    ```bash
    sudo dd if=/dev/zero of=/swap0 bs=1K count=10M
    sudo chmod 0600 /swap0
    sudo mkswap /swap0
    sudo swapon /swap0
    echo '/swap0 swap swap defaults 0 0' | sudo tee -a /etc/fstab
    sudo sysctl vm.swappiness=0
    ```

17. [optional] make sure that every console is attached to a session at startup. This allows for tiling, scrolling and a bunch of other nice features:
    ```bash
    cat <<EOT>> ~/.zshrc
    screen
    EOT
    ```

18. [optional] increase the maximum scrollback per session, and disable the startup message. Scrollback (Ctrl-A, then escape, then PgUp/PgDown or arrow keys) is useful for commands with verbose output:
    ```bash
    cat <<EOT>> ~/.screenrc
    defscrollback 20000
    startup_message off
    EOT
    ```

19. If you haven't rebooted the pi yet since installing powerlevel10k, docker, and microk8s, do so now with `sudo reboot`

20. At this point, you have completed the installation for one pi. Repeat all prior steps for each pi you want to add to your cluster. Alternatively, if your SD cards are all the same size, you can clone the image from the SD card you just set up to your other SD cards.

21. Connect all pis to the same network.

22. Run `microk8s join` and follow the instructions to start clustering the pis.

## Script Installation {#script}

The script completes the above instructions, excluding attempting to connect to a wifi network, enabling cgroups in the firmware, changing the user's shell, or running the microk8s join command. The script triggers a reboot on completion.

To run the script, prepare your freshly imaged pi by enabling cgroups (used by microk8s) and rebooting, using:
```bash
cat /boot/firmware/cmdline.txt | tr '\n' ' '| xargs -i% echo "% cgroup_enable=memory cgroup_memory=1" | sudo tee /boot/firmware/cmdline.txt && sudo reboot
```

After logging back in, you're ready to run the script. It's available on a [designated page](https://dominic.computer/code?page=/blog/2021/turbolift_pi_k8s_install&lang=Bash) and can be applied by running:
```bash
bash <(curl --proto '=https' --tlsv1.2 -sSf 'https://dominic.computer/code?page=/blog/2021/turbolift_pi_k8s_install&lang=Bash')
```

Then, run `chsh -s /bin/zsh`, `exit` and on login you should be done with installation!