Minikube + Cilium on Ubuntu 18.04

December 10th at 6:59am

We're investigating Kubernetes network overlays at work and I am spinning up sample environments to try things out. One that stands out so far is Cilium due to the fine-grained access controls that can be enforced. They have instructions for how to deploy on Minikube, but it took some finangling for me to be successful with my deployment configuration (Ubuntu 18.04 Server running Minikube 'local' without vagrant).

To cut to the chase, skip to the end to see a deploy script that deploys everything in order.

References

Pre-requirements to run Minikube 'Locally' on a VM

I use VMWare workstation and didn't want to deal with Virtualbox and Vagrant for setting up a local test instance for kubernetes network overlays. As a result I went to great lengths to figure out how to run minikube on a stand-alone VM without the vagrant orchestration. Ubuntu Server 18.04 was selected as over time I've gradually drifted away from RHEL/CentOS to Ubuntu as my 'go to' deployment platform. Other OS's will likely work fine with Minikube.

Going the 'local' route requires you to pre-stage a couple of things:

  • docker (Versions 17.03 -> 18.06. 18.09 will cause minikube to fail)
  • kubectl (ideally version 1.10 as that is what ships with minikube 0.30 [Minikube runs a bit behind on updating k8s versions])

To get a version of docker that works, pass in a version argument to apt: sudo apt install -y docker-ce=18.06.1~ce~3-0~ubuntu (This requires configuring the docker repository)

With Kubectl I took the default version using the repo (1.13 at the time of this writing) and did not encounter any difficulties.

Installing Minikube

The latest release as of Dec 10, 2018, is v0.30.0 which is based on Kubernetes 1.10. I followed the Linux installation steps listed on the release page: curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.30.0/minikube-linux-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube

Deploying a Local Minikube Environment to Suppot Cilium

With Docker, Kubectl and Minikube all installed I can deploy a self-contained kubernetes environment to test Cilium in. To do this properly I had to set a few CLI flags when starting Minikube:

  • --vm-driver=none # To run Minikube without vagrant
  • --apiserver-ips 127.0.0.1 # To run Minikube locally
  • --apiserver-name localhost # To run Minikube locally
  • --network-plugin=cni # Required for Cilium
  • --extra-config=kubelet.network-plugin=cni # Required for Cilium
  • --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf # Required to fix a CrashLoopBackup with CoreDNS

Here's the resulting CLI command: sudo minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf

Note: While Minikube will successfully deploy, few of the expected pods will start until the next step (below) is completed to install Cilium. Minikube is configured to use a CNI plugin and we haven't yet installed a network overlay (cilium).

Installing Cilium

Following the minikube deploy guide for cilium I was able to get the Cilium CNI plugin installed and operational. The page describes how to install on both Docker and CRI-O and supports deployments to various versions of Kubernetes (1.8, 1.9, 1.10, 1.11 and 1.12) at the time of writing. I selected the K8s 1.10 deployment YAML.

Note: Cilium requires direct access to an EtcD cluster. For security it should not be the same EtcD cluster that your Kubernetes deployment uses. The installation steps for Minikube will walk you through setting up a dedicated cluster for EtcD.

These are the commands I ran to get Cilium installed in my isolated test environment (production environments should not kubectl create -f from uncontrolled endpoints):

  • kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
  • kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/1.10/cilium.yaml

Once Cilium was installed Kubernetes started to function and pods moved from 'Pending' to 'ContainerCreating' and then 'Running'

Deployment Script

While debugging my installation I had to re-install my minikube environment multiple times. Along the way I came up with this quick script to get a test environment working:

#!/bin/bash
#
# Quick, hacky script to install Docker, Kubectl, Minikube and Cilium
# Tested on Minikube 0.30.0 (Kubernetes 1.10)
#
#   2018-12-09

# Install Docker 18.06
#   IMPORTANT NOTE: This does _NOT_ mark docker-ce as a package to be ignored during update!
#                   To ensure that your minikube version keeps working after applying updates it is
#                   strongly suggested you 'hold back' the docker-ce package
sudo apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88

sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
sudo apt update

sudo apt install -y docker-ce=18.06.1~ce~3-0~ubuntu

#
# Kubectl stuff
#   At the time of writing this pulled Kubectl version 1.13
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubectl

#
# Minikube stuff
#   At the time of writing this pulled Minikube version 0.30.0
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.30.0/minikube-linux-amd64 && sudo chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube

sudo minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf && sudo chown -R rion:rion .

#
# Cilium stuff
#   NOTE: This is pulling the Kubernetes 1.10 yaml. You may need to update this based on your chosen Kubernetes version
kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/1.10/cilium.yaml