We're investigating Kubernetes network overlays at work and I am spinning up sample environments to try things out. One that stands out so far is Cilium due to the fine-grained access controls that can be enforced. They have instructions for how to deploy on Minikube, but it took some finangling for me to be successful with my deployment configuration (Ubuntu 18.04 Server running Minikube 'local' without vagrant).
To cut to the chase, skip to the end to see a deploy script that deploys everything in order.
- Getting Started: Minikube [cilium.readthedocs.io]
- Getting Started Using Minikube [docs.cilium.io]
- Running Minikube with
- Install docker-ce using the repository (ubuntu) [docs.docker.com]
- Install kubectl binary using native package management [kubernetes.io]
- Minikube Releases [github.com/kubernetes/minikube]
- Plugin [loop] not work with systemd-resolved running [github.com/coredns]
Pre-requirements to run Minikube 'Locally' on a VM
I use VMWare workstation and didn't want to deal with Virtualbox and Vagrant for setting up a local test instance for kubernetes network overlays. As a result I went to great lengths to figure out how to run minikube on a stand-alone VM without the vagrant orchestration. Ubuntu Server 18.04 was selected as over time I've gradually drifted away from RHEL/CentOS to Ubuntu as my 'go to' deployment platform. Other OS's will likely work fine with Minikube.
Going the 'local' route requires you to pre-stage a couple of things:
- docker (Versions 17.03 -> 18.06. 18.09 will cause minikube to fail)
- kubectl (ideally version 1.10 as that is what ships with minikube 0.30 [Minikube runs a bit behind on updating k8s versions])
To get a version of docker that works, pass in a version argument to apt:
sudo apt install -y docker-ce=18.06.1~ce~3-0~ubuntu
(This requires configuring the docker repository)
With Kubectl I took the default version using the repo (1.13 at the time of this writing) and did not encounter any difficulties.
The latest release as of Dec 10, 2018, is
v0.30.0 which is based on Kubernetes 1.10. I followed the Linux installation steps listed on the release page:
curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.30.0/minikube-linux-amd64 && chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube
Deploying a Local Minikube Environment to Suppot Cilium
With Docker, Kubectl and Minikube all installed I can deploy a self-contained kubernetes environment to test Cilium in. To do this properly I had to set a few CLI flags when starting Minikube:
- --vm-driver=none # To run Minikube without vagrant
- --apiserver-ips 127.0.0.1 # To run Minikube locally
- --apiserver-name localhost # To run Minikube locally
- --network-plugin=cni # Required for Cilium
- --extra-config=kubelet.network-plugin=cni # Required for Cilium
- --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf # Required to fix a CrashLoopBackup with CoreDNS
Here's the resulting CLI command:
sudo minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf
Note: While Minikube will successfully deploy, few of the expected pods will start until the next step (below) is completed to install Cilium. Minikube is configured to use a CNI plugin and we haven't yet installed a network overlay (cilium).
Following the minikube deploy guide for cilium I was able to get the Cilium CNI plugin installed and operational. The page describes how to install on both Docker and CRI-O and supports deployments to various versions of Kubernetes (1.8, 1.9, 1.10, 1.11 and 1.12) at the time of writing. I selected the K8s 1.10 deployment YAML.
Note: Cilium requires direct access to an EtcD cluster. For security it should not be the same EtcD cluster that your Kubernetes deployment uses. The installation steps for Minikube will walk you through setting up a dedicated cluster for EtcD.
These are the commands I ran to get Cilium installed in my isolated test environment (production environments should not
kubectl create -f from uncontrolled endpoints):
- kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml
- kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/1.10/cilium.yaml
Once Cilium was installed Kubernetes started to function and pods moved from 'Pending' to 'ContainerCreating' and then 'Running'
While debugging my installation I had to re-install my minikube environment multiple times. Along the way I came up with this quick script to get a test environment working:
#!/bin/bash # # Quick, hacky script to install Docker, Kubectl, Minikube and Cilium # Tested on Minikube 0.30.0 (Kubernetes 1.10) # # 2018-12-09 # Install Docker 18.06 # IMPORTANT NOTE: This does _NOT_ mark docker-ce as a package to be ignored during update! # To ensure that your minikube version keeps working after applying updates it is # strongly suggested you 'hold back' the docker-ce package sudo apt-get install -y \ apt-transport-https \ ca-certificates \ curl \ software-properties-common curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo apt-key fingerprint 0EBFCD88 sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable" sudo apt update sudo apt install -y docker-ce=18.06.1~ce~3-0~ubuntu # # Kubectl stuff # At the time of writing this pulled Kubectl version 1.13 curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubectl # # Minikube stuff # At the time of writing this pulled Minikube version 0.30.0 curl -Lo minikube https://storage.googleapis.com/minikube/releases/v0.30.0/minikube-linux-amd64 && sudo chmod +x minikube && sudo cp minikube /usr/local/bin/ && rm minikube sudo minikube start --vm-driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost --network-plugin=cni --extra-config=kubelet.network-plugin=cni --extra-config=kubelet.resolv-conf=/var/run/systemd/resolve/resolv.conf && sudo chown -R rion:rion . # # Cilium stuff # NOTE: This is pulling the Kubernetes 1.10 yaml. You may need to update this based on your chosen Kubernetes version kubectl create -n kube-system -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/addons/etcd/standalone-etcd.yaml kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.3.0/examples/kubernetes/1.10/cilium.yaml