I wanted to write a post on scaling Wireguard based on Ansible and
OpenBSD. There has been some work on this subject recently and previously I have
also given my two cents on using Algo for VPN. However, the Algo Ansible scripts
doesn't support OpenBSD, and they are a bit complicated - thus an increased
attack surface.
From me everything that is 1) secure, 2) understood by the users and 3) easily
setup and managed, gets a good grade.
Find the accompanying GitHub repo
here. I will be referring to it
and some of its files throughout this post.
Other posts I collected that makes a sensible starting point:
- Ted Unangst blogpost on
Wireguard
is a good starting point. However, it lacks deployment details for OpenBSD clients
(using wg-quick). It also needed some modifications to work with multiple clients.
- This OpenBSD-Wireguard repo on
Gitlab published recently.
This was also a magnificent possibility to migrate some of my virtual servers to
Exoscale. Reading about different cloud providers, I amusingly came over Theo de
Raadts 2007-statement on virtualisation-technology.
Before we get started, it is good to know the difference between kmodtools and
wireguard-go. While wireguard-go is a user-land implementation of a
client/server in Go, said to be pretty fast, kmodtools comes with a utility
named wg used to
configure the interfaces of the tunnels.
All of the following can be found in the Github repo of this post. The
following is merely a comment/walk-through of the steps found in the Ansible
playbooks.
Deployment
This was a golden opportunity to get away from Digital Ocean, which
I've learned doesn't expand the instance limits easily. Exoscale I found to be
promising and to deliver OpenBSD as a template with no fuzz.
To setup a virtual machine, make sure to grab a pair of API-keys from
Exoscale. Then
give the variables in create-instances-playbook.yml a look. Install
Python through install-python.yml afterwards. If you haven't used Ansible
previously, you run these playbook files with the
ansible-playbook-command (installed by pip3 install -r requirements
--user). After running these you will find the server we are going to work on
in the inventory file.
Alternatively, have a look at setup.sh which gives the chronological order
of the above playbooks.
Building and Installing
Building and installing Wireguard is quick. The basic starting point here is
installing the following packages:
Other than that, I pulled a trick from the official install script to get the
current snapshot of kmodtools and wireguard-go from the distro
list.
At this point I also noticed that it is hard to run wireguard-go as anything
else than root due to the privileges required to configure the interfaces. I did
one temporary thing here, and that was change the uid to the Wireguard user in
main.go. The patch probably have limited value in terms of what admin
context a threat actor can glean from the context, but it is a start.
To install Wireguard on the newly created server, run the
install-wireguard.yml playbook. This configures unbound to use DNS over TLS
and enables that service for the tunnel. It also compiles and installs the
Wireguard binaries.
Worth noting, I designed this for a future chroot to /var/wgd with the
_wireguard user. This process can be found in the wireguard role.
Wireguard Configuration
Next I found that the configuration was confusing, since the public and
private keys are mixed between client and server configuration files - and the
section names in both are the same, but the content is different. Well, if that
confused you - you know how I felt. At this point I ended up writing a script to
generate a number of client configurations. Running this, like follows:
python3 generate_clients.py --server=<remote_ip> --start_ip=10.99.0.10 --port=8443 --clients=5
Gives a directory structure like follows (relative from git repo root):
config/
\- generate_clients.py
\- conf
\- server.conf
\- clients
\- 10.99.0.10.conf
\- 10.99.0.11.conf
\- 10.99.0.12.conf
\- 10.99.0.13.conf
\- 10.99.0.14.conf
Wireguard doesn't have the best configuration control I have seen, but it is a trade-off
for simplicity. Thus, there are no comment section available in the scripts. The
IP-addresses is the one thing that is static here (it is also used for internal
routing), so make sure to keep a good inventory of those if you track the devices
and who uses them. Actually, considering that Wireguard is designed with low
count of lines for minimalising the attack surface, they should probably leave
it like this.
After this point server.conf can be placed in /var/wgd/etc. Running the
following will reload the configuration and enable those five clients/certificates: rcctl restart wgd.
DNS and Routing Configuration
Note that this section is required for getting Wireguard up and running
To get the above configuration working for multiple clients, I modified Ted
Unangst's approach a bit. The scripts does create the interface with a subnet and
enables IP forwarding:
cd /dev && sh MAKEDEV tun6
ifconfig tun6 up 10.99.0.0/24 netmask 255.255.255.0
sysctl net.inet.ip.forwarding=1
In addition I ended up with adding the following line to the packet filter config:
pass out on egress inet from (tun6:network) nat-to (egress:0)
You also need unbound running locally per default. Unbound comes with OpenBSD,
so what I added here was more or less:
access-control: 0.0.0.0/0 refuse
access-control: 127.0.0.1 allow
access-control: 10.99.0.0/24 allow
I also added TLS over DNS while I was at it:
forward-zone:
name: "."
forward-addr: 1.1.1.1@853
forward-addr: 1.0.0.1@853
Using on Ios and Android
Both Ios and Android has Wireguard applications. Both worked pretty well. The
beauty of the setup above is that client configs are never sent to the server
(other than the public key in server.conf).
Using the barcode scanner in the application, you can generate a config QR code
by:
qrencode -t ansiutf8 config/conf/clients/10.99.0.10.conf
Using on OpenBSD
Compared to the mobile clients of Wireguard, setting up the OpenBSD client is a
different story. To get this working I needed to use the openbsd.bash script
from WireGuard/src/tools/wg_quick. This
does the same as on the server locally and works out of the box. It also creates
routes to the tunnel interface. To enforce this I drew on some experience from
the openbsd-wg repo. To keep the interface up, I added the following to
ifstated.conf. In addition ifstated must be enabled (rcctl enable
ifstated).
In the case below I copied the client configuration file to
/etc/wireguard/tun6.conf which is the path the init script expects.
init-state start
state start {
if tun6.link.up {
set-state online
}
if tun6.link.down {
set-state offline
}
}
state online {
if tun6.link.down {
set-state offline
}
}
state offline {
init {
run "sh /etc/netstart tun6"
}
if tun6.link.up {
set-state online
}
As for /etc/hostname.tun6, I added the following content:
!/usr/local/bin/wg_start.sh
#!route add <exception ip> <local gateway>
Enforcing the VPN policy is vital. For this pf, packet filter, is an option. An
example set of strict local rules could be:
int_if = "{" tun0 tun6 "}"
ext_if = "{" iwm0 "}"
# Generally allow non-encrypted and encrypted web ports on TCP
pass out on $int_if proto tcp to port 80
pass out on $int_if proto tcp to port 8080
pass out on $int_if proto tcp to port 443
pass out on $int_if proto tcp to port 8443
[...]
pass out on $ext_if proto tcp to port 22
pass out on $ext_if proto udp to port 8443
I also became familiar with local routes, as these were needed if anything were
to be routed outside the tunnel.
That is pretty much it. Thanks for reading.