VyOS as Kubernetes Pod
I spotted that Google Doc in my Twitter feed back in december, and this picked up my interest, i.e. what if we were to use VyOS as a Pod, i.e. to basically run some network functionality (like IPSec tunnel, etc) ?
Turned out it works quite well, especially if you pair it with Multus, the framework to add supplemental interfaces via CNI plugins.
Multus
First, we deploy multus on our k8s homelab cluster:
kubectl apply -f https://github.com/k8snetworkplumbingwg/multus-cni/blob/master/deployments/multus-daemonset-thick-plugin.yml
We create a Multus NetworkAttachmentDefinition CRD:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: vyos
namespace: default
spec:
config: '{
"cniVersion": "0.3.0",
"plugins": [{
"type": "macvlan",
"master": "eth0",
"mode": "bridge"
}]
}'
This would basically enable the pod to have a second interface bridged on the main baremetal node eth0 main interface, once the proper annotation is provided.
VyOS configuration file
We can store the configuration of VyOS under a configmap object:
apiVersion: v1
kind: ConfigMap
metadata:
name: vyos
namespace: default
data:
config.boot: |
interfaces {
ethernet eth1 {
address dhcp
}
}
service {
ssh {
disable-host-validation
}
}
system {
config-management {
commit-revisions 100
}
host-name vyosk8s
login {
user vyos {
authentication {
encrypted-password $6$QxPS.uk6mfo$9QBSo8u1FkH16gMyAVhus6fU3LOzvLR9Z9.82m3tiHFAxTtIkhaZSWssSgzt4v4dGAL8rhVQxTg0oAG9/q11h/
plaintext-password ""
}
level admin
}
}
name-server 10.96.0.10
time-zone UTC
}
/* Warning: Do not remove the following line. */
/* === vyatta-config-version: "wanloadbalance@3:ntp@1:webgui@1:dhcp-server@5:pptp@1:webproxy@2:quagga@6:qos@1:firewall@5:ssh@1:dhcp-relay@2:config-management@1:l2tp@1:dns-forwarding@1:ipsec@5:cluster@1:system@9:conntrack-sync@1:conntrack@1:snmp@1:mdns@1:nat@4:webproxy@1:broadcast-relay@1:vrrp@2:zone-policy@1" === */
/* Release version: 1.3.0 */
.vyatta_config: |
Note that the password stored is the default vyos. The rest is matching the notes of the documentation provided in the Google document, with the addition of a DHCP client in the eth1 interface, which will be our Multus one.
VyOS pod
Now let’s try to write a minimal deployment CRD. I came up with this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vyos
namespace: default
labels:
app: vyos
spec:
replicas: 1
selector:
matchLabels:
app: vyos
template:
metadata:
name: vyos
labels:
app: vyos
annotations:
k8s.v1.cni.cncf.io/networks: '[{"name":"vyos","interface":"eth1"}]'
spec:
initContainers:
- name: fix-all-the-things
image: busybox
command:
- /bin/sh
- -c
- |
mount -o remount, rw /sys
securityContext:
privileged: true
containers:
- name: vyos
image: vyos/image:1.3
ports:
- containerPort: 22
command:
- /sbin/init
volumeMounts:
- name: config-boot
mountPath: /config/config.boot
subPath: config.boot
readOnly: false
- name: empty-file
mountPath: /config/.vyatta_config
subPath: .vyatta_config
readOnly: false
- name: host-modules
mountPath: /lib/modules
readOnly: true
securityContext:
privileged: true
volumes:
- name: host-modules
hostPath:
path: /lib/modules
- name: config-boot
configMap:
name: vyos
items:
- key: config.boot
path: config.boot
- name: empty-file
configMap:
name: vyos
items:
- key: .vyatta_config
path: .vyatta_config
Several interesting things to note here:
- Multus interfaces are named net1, net2… we override this because VyOS is expecting eth* interfaces, and we have our multus interface running as eth1.
- Host network mode could be interesting (to run VyOS on public nodes, maybe?) but would require to have the kernel booted with legacy interface names. This might be hard to do without control on the booting process.
- I was hitting a /sys read-only mount even with privileged mode. This might be related to namespace tag, so remounting might be actually necessary in that particular case.
Nevertheless, after a kubectl apply of those two YAMLs we have a nice VyOS container up and running:
❯ kubectl get pods | grep vyos
vyos-58579d958-nr8lt 1/1 Running 0 3m10s
We can now login using kubectl stdin access, and then pivot to the vyos user:
❯ kubectl exec -it vyos-58579d958-nr8lt -- /bin/bash
Defaulted container "vyos" out of: vyos, fix-all-the-things (init)
root@vyosk8s:/# su vyos
vyos@vyosk8s:/$ show version
Version: VyOS 1.3.0-epa3
Release train: equuleus
Built by: Sentrium S.L.
Built on: Sun 31 Oct 2021 17:38 UTC
Build UUID: 383e45ad-b32a-4359-8183-9baacc8e69d9
Build commit ID: bb511522cc3bb2-dirty
Architecture: x86_64
Boot via: installed image
System type: bare metal
Neat, we can run configure on the VyOS instance and use commit, but the save command will not work due to the ConfigMap mounted in the pod - a PVC could do the trick, but this is totally fine for my lab CI/CD usage.
Let’s create an LLDP daemon for our Multus interface as an example:
vyos@vyosk8s:/$ configure
[edit]
vyos@vyosk8s# set service lldp interface eth1
[edit]
vyos@vyosk8s# commit
[edit]
vyos@vyosk8s# exit
Warning: configuration changes have not been saved.
We can check that the interface is sending stuff:
vyos@vyosk8s:~$ sudo tcpdump -i eth1 not port 22 | grep -i lldp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:34:44.685761 LLDP, length 227: vyosk8s
Which is also reported by my UniFi network controller at home.
We can probably also access via ssh:
❯ ssh IP -l vyos
Welcome to VyOS
vyos@IP's password:
Linux vyosk8s 5.11.0-43-generic #47~20.04.2-Ubuntu SMP Mon Dec 13 11:06:56 UTC 2021 x86_64
Welcome to VyOS!
Check out project news at https://blog.vyos.io
and feel free to report bugs at https://phabricator.vyos.net
Visit https://support.vyos.io to create a support ticket.
You can change this banner using "set system login banner post-login" command.
VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Use of this pre-built image is governed by the EULA you can find at
/usr/share/vyos/EULA
Last login: Sun Jan 9 14:34:48 2022 from IP
vyos@vyosk8s:~$
Great! We can build more complex things, like testing VyOS configuration, annonce BGP in a declarative way, interconnect k8s services with some VPN based peers etc.
Public cloud and future ideas
Could this leverage this in a public cloud environment? Well, some early thoughts:
- you would probably need a cloud that enable multiple network interface, like AWS ENIs.
- if there is only one interface, it would mean running it in host networking mode, and the host should have ethX as interface name.
- host networking mode means probably a high risk of messing up with the node if the configuration is broken.
Nevertheless, I find the whole exercice interesting: VyOS could be used in a operator pattern, with the following approach:
- user defines some CRDs format, like “BGPpeers”, “VPNEndpoint” high level utilities.
- a couple of network template with VyOS config covering firewall, routing and service entries matching those CRDs.
- operator generate minimalist VyOS config from the template and CRDs and inject the deployment in the cluster, and track the deployment lifecycle
- eventually test/analyse the configuration with Batfish
- some interfacing at BGP level with the underlying CNI driver (like Calico) could lead to interesting automations with the rest of the k8s cluster.
so yes, Kubernetes is still exciting (reference to that tweet), at least for network engineers.
Lots of things to build.