Bellbird routing
New Internet Routing
The network 172.16.0.0/16 routes to the Internet through Bellbird. Bellbird is set up to use both the 10.146.12.1 router and the 192.168.1.1, dividing the traffic between the two using what is called “Equal Cost Multi-Path” (ECMP) routing. The sharing algorithm is such that each pair of endpoints (local machine and remote site) will always use the same router. Note that a reboot of Bellbird will reshuffle which pair uses which router.
Network address translation (NAT) is used. This does mean that users’ connections will go through NAT twice.
The NAT is set up in /etc/ufw/before.rules . The routing is set up in the script /usr/lib/networkd-dispatcher/routable.d/ecmp-routing. If we want to give a preference for one router over another, we can change the “weight” values in this file. Note that no default route is set in /etc/netplan/99-init.yaml any more.
Also in the ecmp-routing file are commands to set up a special default route for ssh traffic (either to or from port 22). This traffic is routed only via the 192 net. This is necessary to allow ssh’ing in from outside; otherwise a connection might go in one net and out the other, which would fail.
To make this work, it was necessary to change two lines in /etc/sysctl.conf:
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0
Currently, the only access point on the 172 net is Admin-test. This is expected to change in the future.
Contents of /usr/lib/networkd-dispatcher/routable.d/ecmp-routing.
#!/bin/sh
# This sets up equal cost multipath routing to the internet
ip route add default proto static scope global nexthop via 192.168.1.1 weight 1 nexthop via 10.146.12.1 weight 1
# This sets up a special route for packets from our ssh server
# Note that the use of the number 22 for the table and for the mark
# is just for mnemonic convenience, since 22 is the ssh port.
ip route add default via 192.168.1.1 dev enp1s0 table 22
ip rule add from all fwmark 22 table 22
# This flags packets from the ssh server to use the special rule above
iptables -t mangle -A OUTPUT -p tcp -o enp1s0 -s 192.168.1.234 --sport 22 -j MARK --set-mark 22
iptables -t mangle -A OUTPUT -p tcp -o enp2s0 -s 192.168.1.234 --sport 22 -j MARK --set-mark 22
iptables -t mangle -A OUTPUT -p tcp -o enp3s0 -s 192.168.1.234 --sport 22 -j MARK --set-mark 22
iptables -t mangle -A OUTPUT -p tcp -o enp1s0 -s 192.168.1.234 --dport 22 -j MARK --set-mark 22
iptables -t mangle -A OUTPUT -p tcp -o enp2s0 -s 192.168.1.234 --dport 22 -j MARK --set-mark 22
iptables -t mangle -A OUTPUT -p tcp -o enp3s0 -s 192.168.1.234 --dport 22 -j MARK --set-mark 22
Helpful commands to bash:
sudo ufw status verbose
ip route show
ip link list
man netplan
sudo ls -la /etc/netplan
# NO NO NO sudo ls -la /etc/network
sudo cat /etc/network/interfaces # see, I told you NO
sudo ls -la /etc/networkd-dispatcher/
cat /etc/default/ufw | more
sudo cat /etc/ufw/before.rules | more
cat /etc/sysctl.conf | more
cat /etc/netplan/99-init.yaml | more
cat /etc/default/networkd-dispatcher | more
cat /usr/lib/networkd-dispatcher/routable.d/ecmp-routing | more
cat /etc/dhcp/dhcpd.conf | more
sudo lshw -class network
ip a
sudo ethtool enp3s0
cat /etc/iproute2/rt_tables.d/README
cat /etc/iproute2/rt_tables
sudo iptables -L
man iptables
