Bridging Network on Mac OS X for Kubernetes Cluster

[timesheets]

Bridge Network on Mac OS X

I am trying to build a kubernetes cluster on M1 and Intel Mac mini nodes. I am trying to build x86_64 nodes because I am mixing M1 and Intel. This means it's both ARM and x86 architecture. I will use QEMU VM's on the M1 Mac Mini's to emulate x86 architecture. However, the trouble with this is that the VMs want to run on a NAT private network inside my private network. This makes it difficult for nodes to communicate with the master. The best solution (I think) is to use a bridge on the Mac OS side and allow the VMs to participate in the bridge.

Spoiler: it worked.

Install Tunnelblick

Part of the requirement of setting up a bridge bridge0 is also setting up a tap device tap0. Mac OS X does not do this out-of-the-box. It comes with tunnelblick.

Download and install tunnelblick here.

Once tunnelblick is installed, you will need to give permissions to activate a kernel extension. After you can load kernel extensions safely, you can now do that.

sudo kextload /Library/Extensions/tunnelblick-tap.kext

Check that it loaded properly.

> kextstat | grep -i tap
Executing: /usr/bin/kmutil showloaded
No variant specified, falling back to release
  242    0 0xfffffe0006f30000 0x913      0x913      net.tunnelblick.tap (5.0) 2F65AD5B-B8BE-371D-84E9-8194AAB4E815 <7 5 4 1>

You will also now see tap devices in /dev

> ls -1 /dev/tap*
/dev/tap0
/dev/tap1
/dev/tap10
/dev/tap11
/dev/tap12
/dev/tap13
/dev/tap14
/dev/tap15
/dev/tap2
/dev/tap3
/dev/tap4
/dev/tap5
/dev/tap6
/dev/tap7
/dev/tap8
/dev/tap9

Create the Bridge in Terminal

There is already a bridge0 device in Mac OS. It is typically the thunderbolt bridge. Let's get rid of it.

sudo ifconfig bridge0 delete
sudo ifconfig bridge0 down

Get rid of the existing ethernet interface (need to start from scratch).

sudo ifconfig en0 down
sudo ifconfig en0 delete

Now, I create a new bridge which will attach to a fresh interface.

sudo ifconfig bridge0 create
sudo ifconfig bridge0 addm en0 addm tap0

You can verify this worked with

> ifconfig -a
...
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether 00:00:00:00:00:00
        inet6 fe80::1c3f:400:b951:e868%bridge0 prefixlen 64 secured scopeid 0xd
        inet6 2600:8800:7180:2cd:c65:21de:6786:1cfc prefixlen 64 autoconf secured
        inet6 2600:8800:7180:2cd:a9ae:4139:ecd2:5908 prefixlen 64 autoconf temporary
        inet6 2600:8800:7180:2cd::1116 prefixlen 64 dynamic
        inet 192.168.222.29 netmask 0xffffff00 broadcast 192.168.222.255
        Configuration:
                id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
                maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
                root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
                ipfilter disabled flags 0x0
        member: en0 flags=3<LEARNING,DISCOVER>
                ifmaxaddr 0 port 6 priority 0 path cost 0
        member: tap0 flags=3<LEARNING,DISCOVER>
                ifmaxaddr 0 port 20 priority 0 path cost 0
        nd6 options=201<PERFORMNUD,DAD>
        media: autoselect
        status: active
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        ether 00:00:00:00:00:00
        media: autoselect
        status: active
        open (pid 21168)

I masked my hardware addresses, but this is the output you will see.

Up Next

• Configuring the bridge with QEMU to run K3OS in a VM.
• My experience manually and automatically updated K3OS (Manual is deprecated).