Introduction

Lets say you want to test a fresh batch of pot images on a local cluster of virtualbox hosts, because FreeBSD 13.2 is out and you just upgraded.

Only it turns out that virtualbox is no longer working after the upgrade to 13.2.

Running dmesg shows the output:

KLD vboxdrv.ko: depends on kernel - not available or version mismatch
linker_load_file: /boot/modules/vboxdrv.ko - unsupported file type

Previously this could be solved by removing the packages and re-installing them, but this time it makes no difference.

Switching to latest package stream results in the same error as above.

However, there is a solution, and it involves compiling virtualbox-ose from ports yourself, and then setting up new virtualbox machines with optimal settings.

“Why not use Bhyve?” - because other projects use vagrant, and vagrant uses virtualbox

• How To Set Up a Minio Cluster From Potluck, Complete With Nextcloud, Monitoring And Alerting
• How To Easily Set Up OpenLDAP and Matrix Synapse with Potluck Images.

It’s not possible to run Bhyve and VirtualBox concurrently.

Below is a series of steps to install virtualbox from ports, and then setup FreeBSD-13.2 virtual machines with optimal settings.

Phase One: Install VirtualBox from Ports

Delete VirtualBox

First, in case you already have virtualbox installed, delete the installed packages:

pkg delete -f virtualbox-ose virtualbox-ose-kmod

Download FreeBSD src Tree

Then download the FreeBSD src tree, as we need it to compile virtualbox:

fetch -o /tmp ftp://ftp.freebsd.org/pub/`uname -s`/releases/`uname -m`/`uname -r | cut -d'-' -f1,2`/src.txz
tar -C / -xvf /tmp/src.txz

Download Ports src Tree

Next download the Ports src tree:

portsnap fetch update
portsnap extract

Compile VirtualBox

Change to the virtualbox-ose directory to setup the config and compile virtualbox.

The QT5 and X11 components are not needed for my development host, so they’ve been deselected in the config below.

To setup config and build from ports tree, do the following:

cd /usr/ports/emulators/virtualbox-ose

make config

deselect all QT and X11 options
[ ] Native Language support
[ ] QT5
[ ] X11

make install

This process will take 1-2 hours to complete, and will download and compile various packages in the process.

Do not proceed until compiling is complete.

Configure VirtualBox

Now that virtualbox is compiled and installed, we’ll check if aio is used, and if so, add items to sysctl.conf.

https://man.freebsd.org/cgi/man.cgi?query=aio

The aio facility provides system calls for asynchronous I/O. Asynchronous I/O operations are not completed synchronously by the calling thread. Instead, the calling thread invokes one system call to request an asynchronous I/O operation. The status of a completed request is retrieved later via a separate system call.

Run the following to check if AIO is in use:

kldstat -v | grep aio

If there’s any result, edit /etc/sysctl.conf and add:

# VIRTUALBOX aio(4) SETTINGS
vfs.aio.max_aio_queue_per_proc=65536
vfs.aio.max_aio_per_proc=8192
vfs.aio.max_aio_queue=65536
vfs.aio.max_buf_aio=8192

Next we need to enable the vboxnet service as follows:

service vboxnet enable

Add your username, and any other username which needs to run virtualbox, to the vboxusers group:

(sudo) pw groupmod vboxusers -m <username>

Setup a very broad networking file as follows:

mkdir -p /usr/local/etc/vbox
vi /usr/local/etc/vbox/networks.conf

And add:

* 0.0.0.0/0

This step is optional, as it might already be fixed in the package, setup a symlink as follows:

mkdir -p /etc/vbox
ln -s /usr/local/etc/vbox/networks.conf /etc/vbox/networks.conf

Make sure the host computer has gateway enabled by editing /etc/rc.conf and adding:

gateway_enable="YES"

Enable the vboxdrv driver by editing /boot/loader.conf and adding:

vboxdrv_load="YES"

Finally shutdown to restart your box:

shutdown -r now

Phase Two: Optimised VirtualBox Hosts

For our purposes we want 5 virtual machines running FreeBSD-13.2, which can be combined into a single cluster using a tool such as the Beginner’s Guide to Building a Virtual Datacenter on FreeBSD with Ansible, Pot.

We’ll setup each virtual server individually, because a manual step is required to remove the attached ISO as CDROM.

It might also be possible to clone the first machine and simplify the process, however for this guide we are opting for manual repetition.

Prerequisites

Make directories and download install media. These directories could be ZFS datasets, and also located elsewhere. For our purposes they’re in the root.

mkdir /vms
mkdir /iso
cd /iso
fetch https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-amd64-disc1.iso

Make sure you have 5 free IP addresses in your local range, to allocate one to each virtual machine, or configure reserved addresses in a DHCP solution.

Setup and Install server1

Protip: the --cpus flag should only be set to 1 CPU for the best results when running multiple virtualbox instances on a single FreeBSD host.

Configuring for multiple CPUs doesn’t result in the same outcome as with other hypervisors. Instead it slows things down and causes network delays between the individual virtual machines.

For our first server we want to configure the basics, attach a disk and ISO image, set a VNC password, and then start the VM:

VBoxManage createvm --ostype FreeBSD_64 --register --basefolder /vms --name server1

VBoxManage modifyvm server1 --memory 2048 --ioapic on --cpus 1 --chipset ich9 --nic1 bridged --nictype1 virtio --bridgeadapter1 re0

VBoxManage createhd --size 50000 --filename "/vms/server1/server1.vdi"

VBoxManage storagectl server1 --name "SATA Controller" --add sata --portcount 4 --bootable on

VBoxManage storageattach server1 --storagectl "SATA Controller" --port 1 --type hdd --medium "/vms/server1/server1.vdi"

VBoxManage storageattach server1 --storagectl "SATA Controller" --port 2 --type dvddrive --medium "/iso/FreeBSD-13.2-RELEASE-amd64-disc1.iso"

VBoxManage modifyvm server1 --vrdeproperty VNCPassword=password

You can now start the virtual machine, listening for VNC on port 5001:

VBoxHeadless --startvm server1 --vrde on --vrdeproperty TCP/Ports=5001

From another host, using a VNC client, connect to your box on port 5001 with the password “password”. You should get a regular FreeBSD boot process and install screen.

You can now complete the regular FreeBSD install process.

Make sure to do the following during the installation:

• ZFS on root, automatic
• Set root password
• When adding a user, make sure to add them to the wheel group when asked to invite to other groups

When installation is complete and you get an opportunity to exit to a shell for some last minute changes, you can do the following:

vi /etc/ssh/sshd_config

And set the following parameters:

PermitRootLogin prohibit-password
PasswordAuthentication no

Then you can also add your SSH key to the root and user .ssh/authorized_keys provided a copy of the file is in a web-accessible location. For example:

cd /root
mkdir .ssh
cd .ssh
fetch http://your.host/keys.txt -o authorized_keys
chmod 600 authorized_keys
cd ..
chmod 700 .ssh
cd /home/yourusername
mkdir .ssh
cd .ssh
fetch http://your.host/keys.txt -o authorized_keys
chmod 600 authorized_keys
cd ..
chmod 700 .ssh
chown -R yourusername:yourusername .ssh

This will allow you to login to the system using your SSH keys on first boot.

You can now exit the installer and reboot, but before you do, you need to be prepared to unmount the CDROM drive.

In a new terminal window to your box, run the following during the step where the VNC window boots into the BIOS

VBoxManage storageattach server1 --storagectl "SATA Controller" --port 2 --medium emptydrive

If you’re too late, just hit 4 during the installer boot screen to reboot and it should boot from disk.

The host will reboot and you can test a login within the VNC window, then type

shutdown -h now

This will shutdown the host and the VNC window will prompt to press any key to reboot. Don’t do that!

Press “ctrl+c” back in your original terminal window where the virtual machine was started. This will end the process and the VNC window should die too.

Start up the virtual machine again in a headless fashion with

VBoxManage startvm server1 --type headless

You will need to connect via VNC to port 5001 and “press any key to reboot”

Your server1 host will now be running in the background at the IP you configured during setup. You can SSH to it and perform some initial package installs.

ssh yourusername@server1
sudo su -
pkg install sudo nano rsync jq python39 bash

Make sure to run visudo and uncomment the following line:

%wheel ALL=(ALL:ALL) NOPASSWD: ALL

The host is now ready for further setup via automation tools, we can move on to the next server.

Setup and Install server2

For the second server we’ll do everything the same as the first, except use different file names, IP address, VNC port.

VBoxManage createvm --ostype FreeBSD_64 --register --basefolder /vms --name server2

VBoxManage modifyvm server2 --memory 2048 --ioapic on --cpus 1 --chipset ich9 --nic1 bridged --nictype1 virtio --bridgeadapter1 re0

VBoxManage createhd --size 50000 --filename "/vms/server2/server2.vdi"

VBoxManage storagectl server2 --name "SATA Controller" --add sata --portcount 4 --bootable on

VBoxManage storageattach server2 --storagectl "SATA Controller" --port 1 --type hdd --medium "/vms/server2/server2.vdi"

VBoxManage storageattach server2 --storagectl "SATA Controller" --port 2 --type dvddrive --medium "/iso/FreeBSD-13.2-RELEASE-amd64-disc1.iso"

VBoxManage modifyvm server2 --vrdeproperty VNCPassword=password

You can now start the virtual machine, listening for VNC on port 5002:

VBoxHeadless --startvm server2 --vrde on --vrdeproperty TCP/Ports=5002

Complete the FreeBSD installation, making sure to invite your user to the wheel group, and completing other steps for server1.

Then during reboot remove the CDROM in a new terminal window with:

VBoxManage storageattach server2 --storagectl "SATA Controller" --port 2 --medium emptydrive

Then boot up, login, shutdown and exit the process used to start server2

Start up server2 in a headless fashion:

VBoxManage startvm server2 --type headless

You will need to connect via VNC to port 5002 and “press any key to reboot”

Your server2 host will now be running in the background at the IP you configured during setup. You can SSH to it and perform some initial package installs as above.

Setup and Install server3

For the this server we’ll do everything the same as before, except use different file names, IP address, VNC port.

VBoxManage createvm --ostype FreeBSD_64 --register --basefolder /vms --name server3

VBoxManage modifyvm server3 --memory 2048 --ioapic on --cpus 1 --chipset ich9 --nic1 bridged --nictype1 virtio --bridgeadapter1 re0

VBoxManage createhd --size 50000 --filename "/vms/server3/server3.vdi"

VBoxManage storagectl server3 --name "SATA Controller" --add sata --portcount 4 --bootable on

VBoxManage storageattach server3 --storagectl "SATA Controller" --port 1 --type hdd --medium "/vms/server3/server3.vdi"

VBoxManage storageattach server3 --storagectl "SATA Controller" --port 2 --type dvddrive --medium "/iso/FreeBSD-13.2-RELEASE-amd64-disc1.iso"

VBoxManage modifyvm server3 --vrdeproperty VNCPassword=password

You can now start the virtual machine, listening for VNC on port 5003:

VBoxHeadless --startvm server3 --vrde on --vrdeproperty TCP/Ports=5003

Complete the FreeBSD installation, making sure to invite your user to the wheel group, and completing other steps for server1.

Then during reboot remove the CDROM in a new terminal window with:

VBoxManage storageattach server3 --storagectl "SATA Controller" --port 2 --medium emptydrive

Then boot up, login, shutdown and exit the process used to start server3

Start up server3 in a headless fashion:

VBoxManage startvm server3 --type headless

You will need to connect via VNC to port 5003 and “press any key to reboot”

Your server3 host will now be running in the background at the IP you configured during setup. You can SSH to it and perform some initial package installs as above.

Setup and Install server4

For the this server we’ll do everything the same as before, except use different file names, IP address, VNC port.

VBoxManage createvm --ostype FreeBSD_64 --register --basefolder /vms --name server4

VBoxManage modifyvm server4 --memory 2048 --ioapic on --cpus 1 --chipset ich9 --nic1 bridged --nictype1 virtio --bridgeadapter1 re0

VBoxManage createhd --size 50000 --filename "/vms/server4/server4.vdi"

VBoxManage storagectl server4 --name "SATA Controller" --add sata --portcount 4 --bootable on

VBoxManage storageattach server4 --storagectl "SATA Controller" --port 1 --type hdd --medium "/vms/server4/server4.vdi"

VBoxManage storageattach server4 --storagectl "SATA Controller" --port 2 --type dvddrive --medium "/iso/FreeBSD-13.2-RELEASE-amd64-disc1.iso"

VBoxManage modifyvm server4 --vrdeproperty VNCPassword=password

You can now start the virtual machine, listening for VNC on port 5004:

VBoxHeadless --startvm server4 --vrde on --vrdeproperty TCP/Ports=5004

Complete the FreeBSD installation, making sure to invite your user to the wheel group, and completing other steps for server1.

Then during reboot remove the CDROM in a new terminal window with:

VBoxManage storageattach server4 --storagectl "SATA Controller" --port 2 --medium emptydrive

Then boot up, login, shutdown and exit the process used to start server4

Start up server4 in a headless fashion:

VBoxManage startvm server4 --type headless

You will need to connect via VNC to port 5004 and “press any key to reboot”

Your server4 host will now be running in the background at the IP you configured during setup. You can SSH to it and perform some initial package installs as above.

Setup and Install server5

For the this server we’ll do everything the same as before, except use different file names, IP address, VNC port.

VBoxManage createvm --ostype FreeBSD_64 --register --basefolder /vms --name server5

VBoxManage modifyvm server5 --memory 2048 --ioapic on --cpus 1 --chipset ich9 --nic1 bridged --nictype1 virtio --bridgeadapter1 re0

VBoxManage createhd --size 50000 --filename "/vms/server5/server5.vdi"

VBoxManage storagectl server5 --name "SATA Controller" --add sata --portcount 4 --bootable on

VBoxManage storageattach server5 --storagectl "SATA Controller" --port 1 --type hdd --medium "/vms/server5/server5.vdi"

VBoxManage storageattach server5 --storagectl "SATA Controller" --port 2 --type dvddrive --medium "/iso/FreeBSD-13.2-RELEASE-amd64-disc1.iso"

VBoxManage modifyvm server5 --vrdeproperty VNCPassword=password

You can now start the virtual machine, listening for VNC on port 5005:

VBoxHeadless --startvm server5 --vrde on --vrdeproperty TCP/Ports=5005

Complete the FreeBSD installation, making sure to invite your user to the wheel group, and completing other steps for server1.

Then during reboot remove the CDROM in a new terminal window with:

VBoxManage storageattach server5 --storagectl "SATA Controller" --port 2 --medium emptydrive

Then boot up, login, shutdown and exit the process used to start server5

Start up server5 in a headless fashion:

VBoxManage startvm server5 --type headless

You will need to connect via VNC to port 5005 and “press any key to reboot”

Your server5 host will now be running in the background at the IP you configured during setup. You can SSH to it and perform some initial package installs as above.

Complete

You now have 5 manually installed virtualbox hosts running FreeBSD-13.2. They are optimised for 1 CPU yet run with the full power of the host.

If you’re going to use ansible to configure them for something, make sure sudo and python39 are installed. And make sure visudo allows users of the wheel group to perform commands.

We can recommend trying out the Beginner’s Guide to Building a Virtual Datacenter on FreeBSD with Ansible, Pot.