Setting up a K8S cluster with Cilium

Hachem

— 6 min read

Hello. This is a tutorial on how to set up a K8S cluster that uses Cilium as its CNI. We use fedora for the host VMs.

Preferred Prerequisites

  • 3 different servers accessible through ssh.
  • 1 IPv4 IP per server, where each server can reach another with them over a (preferably) private network.
  • Enough resources and the correct set of open ports. Refer to the kubernetes documentation as well as Cilium's.

We do not aim for a minimum viable product here, but a comfortable possible way to setup a cluster with Cilium.

If your existing resources differ, adapt the tutorial accordingly. This includes the existence of a private network to your cloud setup, the number of nodes you have, the operating system you use, the container runtime environment you use, ...

Installation specifications

These are the same specifications per server:

  • OS: Fedora 41
  • Kernel: 6.13.6
  • Shell: bash
  • CPU: 2 cores, amd64
  • Memory: 4 GB
  • Diskspace: 10 GB

The versions of the main software used are:

  • Installed kubernetes version: v1.32.0
  • Containerd version: 1.7.26
  • Cilium helmchart version: 1.17.2

The master node requires in general less cpu and memory than this. We will stick to these specifications for simplicity.

Each server has a unique hostname. Here we set them to master1, worker1 and worker2.

You can modify the hostnames of each machine to match its label:

  • hostnamectl hostname master1 on master1 to-be node.
  • hostnamectl hostname worker1 on worker1 to-be node.
  • hostnamectl hostname worker2 on worker2 to-be node.

(Optional) Preconfiguration

To simplify testing and reference to servers later on, it might be useful to modify each of the servers' /etc/hosts file and define one another with their IPs.

...

# For kubernetes administration
10.0.0.2 master1
10.0.0.3 worker1
10.0.0.4 worker2

Here I have a unique private IP per server under the IP range 10.0.0.0/8. Your set of IPs might differ.

It is also a good security (and/or management) practice to have a separate user from root.
We therefore create a user user and run most commands through it.

If you already have a separate user from root, this step is redundant.
You can also optionally set your ssh access to be through this newly created user (not covered in tutorial).

Run the following as root:

useradd user # Create user "user".
echo user:user | chpasswd # Set user's password to "user". Modify to liking.
usermod -aG wheel user # Add the user to wheel group, in order to allow sudo access.

Adding user to wheel should be sufficient to give the user sudo priviledges by default.
Refer to sudo's documentation and/or visudo in case this is not the case.

Installation

All the bash commands from this point on will be run as the user user with sudo priviledges.

Installing a CRI: containerd

On each server: we first update the repositories.

sudo dnf update -y

Then we install containerd

sudo dnf install -y containerd

Now we ought to modify the containerd configuration file, as the default one for Fedora 41 is not the one kubernetes might expect.

# Backup original config and update it
sudo mv /etc/containerd/config.toml /etc/containerd/config.toml.bkup
sudo containerd config default > /etc/containerd/config.toml

If you use cgroupv2 with systemd, you need to change the configuration accordingly

# Set SystemdCgroup to true if we are using cgroupv2
sudo sh -c "if [ `stat -fc %T /sys/fs/cgroup/` = cgroup2fs ]; then sed -i 's/SystemdCgroup = .*$/SystemdCgroup = true/' /etc/containerd/config.toml; fi"

Finally, we can enable the service for containerd

sudo systemctl enable --now containerd

Installing kubelet, kubeadm and kubectl

As per the kubernetes documentation, we proceed to install the three programs, namely kubelet, kubeadm and kubectl on every server.

The kubernetes official documentation indicates that we ought to disable SELinux. Until SELinux support is improved, certain containers might require access to the host filesystem or some host resources (e.g. the CNI's). Maybe we'll attempt a proper SELinux compatible install in the future. But for now:

sudo setenforce 0
sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
echo "SELinux: `getenforce`"

The latter should show "SELinux: Permissive".

Note: If SELinux isn't enabled kernel-wise or is not supported by host's distribution, you can skip the commands shown above.

Next we add the kubernetes YUM repository to each server.

sudo cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

And finally we install the three tools - kubelet, kubeadm and kubectl on the servers.

# Note: --disableexcludes=kubernetes here does not work, unlike what the k8s documentation suggests.
sudo yum install -y kubelet kubeadm kubectl

Note: It is not necessary to install kubectl on servers where you will not use it. Here we only use it in master1.

Now we may enable the appropriate service for kubelet.

sudo systemctl enable --now kubelet

Initializing the first master node

To initialize the first node in the control plane, we use kubeadm.

We decided to use the CIDR 172.16.0.0/12 as the IP private range for pods.

The apiserver-advertise-address should be the private IP of master1. The following commands are to be executed on the server we denoted as 'master1'.

sudo kubeadm init --apiserver-advertise-address=10.0.0.2 --pod-network-cidr "172.16.0.0/12"

Now, for convenience, we can copy the generated /etc/kubernetes/admin.conf as /home/user/.kube/config.
This will allow us to run kubectl on this master node without sudo privileges having to provide the --kubeconfig flag every call.

mkdir -p /home/user/.kube
sudo cp -f /etc/kubernetes/admin.conf /home/user/.kube/config
sudo chown user:user /home/user/.kube/config

Now you can verify that the basic kube-system pods are present.

kubectl get pods -n kube-system

NAME                               READY   STATUS    RESTARTS     AGE
coredns-668d6bf9bc-rw2lk           1/1     Pending   1 (1h ago)   2h
coredns-668d6bf9bc-tkjpj           1/1     Pending   1 (1h ago)   2h
etcd-master                        1/1     Running   6 (1h ago)   2h
kube-apiserver-master              1/1     Running   6 (1h ago)   2h
kube-controller-manager-master     1/1     Running   1 (1h ago)   2h
kube-proxy-vrg9w                   1/1     Running   1 (1h ago)   2h
kube-scheduler-master              1/1     Running   8 (1h ago)   2h

Install Cilium

In this tutorial we isntall cilium before joining the other nodes.

First we need to install helm.

sudo dnf install -y helm

Then we may add the cilium repository.

helm repo add cilium https://helm.cilium.io/

Note: helm relies on the same configuration file as kubectl.

Finally we install cilium v1.17.2, and we make sure to match the pod CIDR we defined above.
Here I decided to use kubernetes IPAM mode.

helm install cilium cilium/cilium --version 1.17.2 --namespace kube-system --set ipam.operator.clusterPoolIPv4PodCIDRList=172.16.0.0/12 --set ipam.mode=kubernetes

Now Cilium should be installed.

Join the other nodes

You can create a join token on the master1 server using kubeadm.

kubeadm token create --print-join-command

Copy the given given command and execute it on the servers worker1 and worker2. The command shown below is just for refernece to show what it looks like.

#kubeadm join 10.0.0.2:6443 --token s7iji8.uikjekosa12uum3k --discovery-token-ca-cert-hash sha256:980f1256703de29b9c34fe1ba9e5a93683f4c1d1866025881cd1a44cecce947d

(Optional) Install the cilium CLI

We can install the cilium CLI to test cilium's connectivity. Following cilium's official documentation, we run the following on the master node:

CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) # Change if fixed version
CLI_ARCH=amd64 # Change if not AMD64 architecture
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-${CLI_ARCH}.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz{,.sha256sum}

Note: the cilium CLI relies on the same configuration file as kubectl.

You can now use cilium status --wait to check the status of the cilium installaton.

    /¯¯\
 /¯¯\__/¯¯\    Cilium:             OK
 \__/¯¯\__/    Operator:           OK
 /¯¯\__/¯¯\    Envoy DaemonSet:    OK
 \__/¯¯\__/    Hubble Relay:       disabled
    \__/       ClusterMesh:        disabled

DaemonSet              cilium                   Desired: 3, Ready: 3/3, Available: 3/3
DaemonSet              cilium-envoy             Desired: 3, Ready: 3/3, Available: 3/3
Deployment             cilium-operator          Desired: 2, Ready: 2/2, Available: 2/2
Containers:            cilium                   Running: 3
                       cilium-envoy             Running: 3
                       cilium-operator          Running: 2
                       clustermesh-apiserver    
                       hubble-relay             
Cluster Pods:          10/10 managed by Cilium
Helm chart version:    1.17.2

Note: You might get an error before joining at least one worker node. This is normal (unless you have a one-node cluster).

Run cilium connectivity test to test your cilium installation. It will take quite some time, but if everything is working properly, you will see this:

[user@master1 ~]$ cilium connectivity test                                                                                                          
ℹ  Monitor aggregation detected, will skip some flow validation steps                                                                               
✨ [kubernetes] Creating namespace cilium-test-1 for connectivity check...                                                                           
✨ [kubernetes] Deploying echo-same-node service...                                                                                                  
✨ [kubernetes] Deploying DNS test server configmap...
...
[=] [cilium-test-1] Test [check-log-errors] [112/112]
✅ [cilium-test-1] All 70 tests (640 actions) successful, 42 tests skipped, 1 scenarios skipped.

I hope you enjoyed this tutorial!

Hachem.

Read more