M1 K8s The Hard Way

What

Lima provides a ‘sort of unofficial “macOS subsystem for Linux”’ which is designed to provide CLI access to (Contai)nerdctl on a Mac. It does this by leveraging QEMU with HVF accelerator enabling running Arm on Intel and Intel on Arm. The tool has a shorthand that references a ‘default’ vm where:

lima nerdctl run -d --name nginx -p 127.0.0.1:8080:80 nginx:alpine

translates into something like

limactl shell default nerdctl run -d --name nginx -p 127.0.0.1:8080:80 nginx:alpine

Having spent a little time evaluating Lima as a replacement for my own container usage on Mac I starting looking into running multiple VMs and customising the network configurations. It turns out that with autoconf and automake installed on a vanilla M1 Monterey Mac, vde_vmnet can be installed providing “vmnet.framework support for rootless QEMU”. This translates into (a slightly manual) ability to bring VMs up with specific network interfaces and specific DHCP issued static IP addresses.

The instructions at the vde_vmnet repository describe how to build the two required components, vde_vmnet and vde_switch.

Install vde_switch:

git clone https://github.com/virtualsquare/vde-2.git
cd vde-2
autoreconf -fis
./configure --prefix=/opt/vde
make
sudo make install

Install vde_vmnet:

git clone https://github.com/lima-vm/vde_vmnet
cd vde_vmnet
make PREFIX=/opt/vde
sudo make PREFIX=/opt/vde install

The configuration for the networks can be found in e.g. ~/.lima/_config/networks.yaml

Support for managing network daemons automatically is backed in to Lima as of v0.7.0. To do this a networks configuration is added to the VM lima.yaml configuration file:

networks:
  # Lima can manage daemons for networks defined in $LIMA_HOME/_config/networks.yaml
  # automatically. Both vde_switch and vde_vmnet binaries must be installed into
  # secure locations only alterable by the "root" user.
  - lima: shared

Static IP address can be configured by specifying a MAC address for each VM and configuring /etc/bootptab with Host / MAC / IP mappings, for example:

# bootptab
%%
# hostname      hwtype  hwaddr              ipaddr          bootfile
tmp-vm01        1       de:ad:be:ef:00:01   192.168.105.100

And in a VMs configuration:

networks:
  - lima: shared
    macAddress: "de:ad:be:ef:00:01"

With a change to bootptab the DHCP deamon needs to be reloaded:

sudo /bin/launchctl unload -w /System/Library/LaunchDaemons/bootps.plist
sudo /bin/launchctl load -w /System/Library/LaunchDaemons/bootps.plist

For Lima to be able to control the networking daemons it (currently) requires sudo access. This can be added via:

limactl sudoers | sudo tee /etc/sudoers.d/lima

Why

With an ability to spin up small VMs with custom network configuration running almost seamlessly on an M1 macbook, I reached for a problem to match the solution. Having not run through Kubernetes The Hard Way for quite some time it seemed I had found the problem.

Can I run an ARM based multi-VM hand-crafted Kubernetes cluster on an M1 powered macbook - easily ?

There are multiple ways to get Kubernetes up and running on an M1 Macbook with substantially less effort required than the approach taken in this blog post. Lima itself provides a config file, Minikube is still winning and some new entries to the scene look interesting like Rancher Desktop, which uses Lima under the hood.

How

To set up the VMs for an Arm based Kubernetes cluster on an M1 Mac I used the following network configuration:

paths:
  vdeSwitch: /opt/vde/bin/vde_switch
  vdeVMNet: /opt/vde/bin/vde_vmnet
  varRun: /private/var/run/lima
  sudoers: /private/etc/sudoers.d/lima

group: staff

networks:
  shared:
    mode: shared
    gateway: 10.1.0.1
    dhcpEnd: 10.1.254.254
    netmask: 255.255.0.0

Lima configuration allows specifying a remote iso (which it can cache locally), for example: https://github.com/lima-vm/alpine-lima/releases/download/v0.2.2/alpine-lima-std-3.14.3-aarch64.iso

With the ability to orchestrate VMs and network configuration locally with Lima, we can run through installing a setting up a cluster using resources such as Kubernetes The Hard Way.