I will go straight to the point ..., please be patient as this is a very long article because I want to include all dependencies and necessary setup to ensure a working environment.

This is part 1 of the 3 parts article, links to part 2 and 3 below:

  1. Part 1 - Basic OS setup and master node.
  2. Part 2 - Worker node and creating a new service.
  3. Part 3 - Make your service stateful via persistent volume.

What you will need:

  1. At least 2 Raspberry Pi 3/4 machines with LAN connectivity to form a Cluster. (wired connectivity preferred, WIFI will work but it will mean you have to tingle with it)
  2. Ubuntu image for Raspberry Pi 3/4, I am using Ubuntu 20.04.2
  3. Both machines time synchronised, all updates installed, accessible to Internet. By default, Ubuntu in Raspberry Pi will get DHCP IP address from your router, hostname will be ubuntu, username and password are also ubuntu. So it is recommended that you change the hostnames when you set them up.
  4. Please note that this setup is sensitive to network changes. Hence please plan your network topology carefully, including all the machines' IP addresses, before starting.

For all machines:

Ubuntu configurations - memory cgroup

Enable "memory cgroup" in Linux kernel by modifying the kernel command line file /boot/firmware/cmdline.txt and add the following to the end of the line:

cgroup_enable=cpuset swapaccount=1 cgroup_enable=memory cgroup_memory=1

The whole line would look like the following:

net.ifnames=0 dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc cgroup_enable=cpuset swapaccount=1 cgroup_enable=memory cgroup_memory=1

Then change some of the system configurations using below:

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sudo sysctl --system

The above will enable bridging in Linux which is required for Kubernetes to work.

Ubuntu packages and repositories setup

Install a few pieces from standard Ubuntu packages

sudo apt install apt-transport-https ca-certificates curl gnupg-agent software-properties-common

Then, add the Docker and Kubernetes official repositories into apt and update the repositories.

# For Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
echo "deb [arch=arm64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list

# For Kubernetes
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
sudo apt update

Reboot the machines.

Installation of Docker

The simpliest way of making Kubernetes Cluster work is to install Docker, which at the moment is still the default container runtime. It may change in the future.

Kubernetes is only verified to work with a older version of Docker, hence we need to use the following to install Docker properly and avoid the warnings from Kubernetes setup. At the time being, we need to install docker 5.19 rather than 5.20.

You can find out the versions of Docker available by:

sudo apt list -a docker-ce docker-ce-cli containerd.io

Output will look like this:

Listing... Done
containerd.io/focal 1.4.3-1 arm64
containerd.io/focal 1.3.9-1 arm64
containerd.io/focal 1.3.7-1 arm64
containerd.io/focal 1.2.13-2 arm64

docker-ce-cli/focal 5:20.10.3~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:20.10.2~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:20.10.1~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:20.10.0~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.15~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.14~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.13~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.12~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.11~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.10~3-0~ubuntu-focal arm64
docker-ce-cli/focal 5:19.03.9~3-0~ubuntu-focal arm64

docker-ce/focal 5:20.10.3~3-0~ubuntu-focal arm64
docker-ce/focal 5:20.10.2~3-0~ubuntu-focal arm64
docker-ce/focal 5:20.10.1~3-0~ubuntu-focal arm64
docker-ce/focal 5:20.10.0~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.15~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.14~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.13~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.12~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.11~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.10~3-0~ubuntu-focal arm64
docker-ce/focal 5:19.03.9~3-0~ubuntu-focal arm64

Install docker-ce, docker-ce-cli and containerd.io version 5.19, 5.19 and 1.39 and lock them to this version respectively by:

sudo apt install docker-ce=5:19.03.15~3-0~ubuntu-focal docker-ce-cli=5:19.03.15~3-0~ubuntu-focal containerd.io=1.3.9-1
sudo apt-mark hold docker-ce docker-ce-cli containerd.io

After installation, a new unix group called docker will be added. Add yourself to this group and apply the changes by:

sudo usermod -aG docker <username>
newgrp docker

Then, change the docker configuration slightly by modifying the configuration file /etc/docker/daemon.json.

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}

Then restart docker accordingly.

Install Kubernete core softwares

sudo apt-get install kubeadm kubelet kubectl

Then mark them as hold to avoid accidental upgrades that may break things.

sudo apt-mark hold kubeadm kubelet kubectl

By the time, all core software packages are installed.

Setup for Master Node

Node setup

You need to have a least 1 master node. I am still exploring how to do master node HA so that will come as a separate guide later. It is very easy to initiate a master note by the following:

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

I have modified the above on Feb 6, 2021 to specify the network configuration within the cluster. During setup, I started to explore a much more simplified network backend called Flannel which requires this

A lot of logs will throw up, just be patient and wait for the successful messages below:

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join <some IP>:6443 --token <some token> \
    --discovery-token-ca-cert-hash sha256:<some long sha256 hash> 

Just to as the log say. Keep the environments, and the join command handy because whenever you need to control the environment or add a node, you will need them.

Network setup

There are a number of network backend, called CNI network provider, in Kubernetes. I am using Flannel, which is much simplier than the rest of the CNI provider (comparisons here).

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Conclusion

By this point of time, your Kubernetes cluster is setup with 1 master node. However, it cannot run any applications in it yet because master node cannot do so. In the next session, we will deploy a few applications to see how it works.