A Beginner's Guide to Building a Virtual Datacenter on FreeBSD with Ansible, Pot and More


Back in 2020, a three-part blog series was published on building your own Virtual Datacenter (vDC).

While the detailed configuration instructions are outdated meanwhile (the images offer a lot more options today than back then), you can read part 1, part 2 and part 3 as a refresher.

This guide introduces an ansible script to automate the provisioning of a vDC for your applications.

In addition to building a vDC with consul and nomad, it offers enhancements over the original blog series, such as the inclusion of monitoring and alerting, wireguard mesh networking, along with openldap and mariadb instances, and nginx nomad jobs.

The target audience for this post is comfortable installing FreeBSD on servers or virtual machines and comfortable enough to rename the default network interface, or to find out how to do so.

You don’t need to know much ansible, you can pick it up from the scripts.

Part 1: Understanding the Basics

1.1 FreeBSD And Containers

There are several container or jail management solutions for FreeBSD these days. We like pot.

1.2 Pot For Container Management

pot is a jail manager, a collection of shell scripts connecting standard features in FreeBSD, such as ZFS and jails, into portable images that can run on other hosts.

1.3 Potluck Repository

The Potluck repository is a DockerHub-like repository of pot application images that can be downloaded and used without having to build your own.

Most of the potluck images have consul integration. You’ll need a consul image running to make use of them.

Part 2: The Ansible Pot Foundation Framework

2.1 What Is Installed?

The script will provision 1, 3 or 5 servers. Where 3 or 5 servers are used, a wireguard mesh network will connect them.

Two VLANs will be created off the default network interface, which must be named untrusted before you begin.

One VLAN will be for pot images, and the other will be for nomad jobs.

In cluster environments with the wireguard mesh, all VLANs can see all other server VLANs over wireguard links. This makes it possible to build consul or nomad clusters over wireguard.

A pf firewall with basic ruleset will be applied. One host will be a frontend host and accept HTTP requests.

For service discovery, a consul server or cluster will be setup, and for job orchestration, a nomad server or cluster is included. A nomad client will be installed on the servers directly.

A traefik-consul image is included, and also haproxy for dealing with the frontend public website, along with acme.sh for certificate registration.

Two nomad jobs for nginx pot jails are included, a public website and an intranet site linking to various services in the vDC.

An openldap instance is included, which has a useful web GUI automatically setup. This is the easiest, fully-functional openldap service available.

A mariadb instance is also included, but has no databases added at this stage.

For monitoring and alerting, the Beast-of-Argh pot image is used, which is a single pot collection of prometheus, alertmanager, promtail, loki, grafana and custom dashboards and alerts.

(These applications are also available individually on the potluck website, albeit for more complex security environments)

2.2 What Are The Applications Used For?

In the simplest vDC, consul is used for service discovery, while nomad is used for job orchestration. The traefik-consul image helps connect the two.

The addition of a monitoring and alerting component uses node_exporter for metrics, and publishes via consul, which is pulled into the The Beast-of-Argh image running prometheus. Additionally syslog-ng sends logs to a loki syslog server.

The Beast-of-Argh image is in daily usage on live systems and is instumental for keeping an eye on things. Custom dashboards are included.

wireguard is used to link servers in a cluster, or clients to the network for administration purposes.

pf is used to handle firewalling.

openldap is used for accounts management, and has more functionality with other pot images for mail or chat services.

mariadb is used as a general purpose database for pot wordpress images, or application databases. You might find postgresql is better suited for your needs and that can be catered for with updates.

We hope that this example setup with the basic components is sufficient to accelerate development of your own custom services hosted in the environment.

Part 3: Setting Up The Virtual Datacenter

3.1 Ensure The System Meets The Requirements

You network interface must be named untrusted. The ansible script is expecting this name. The reasons stem from practises in live pot environments, where it’s deemed a good idea.

You can configure this in /etc/rc.conf. Make sure to get the interface name correct, then reboot or restart networking.

As an example with a non-public IP, edit /etc/rc.conf for default interface to:

ifconfig_untrusted="inet netmask"

You must have ssh key access to the server as a user. This user must be a member of the wheel group and have sudo permissions without a password.

pkg install sudo


You must have decided on the quarterly or latest package stream beforehand. This script will install from the already configured stream. pot images make use of quarterly packages.

3.2 Clone The git Repository

Clone the git repo:

pkg install git
git clone https://codeberg.org/Honeyguide/ansible-pot-foundation.git
cd ansible-pot-foundation

3.3 Copy host.sample to hosts File

Inside the correct inventory directory, copy the hosts.sample file to hosts and edit to your needs.

For example, a 5 server cluster will use the file in inventory.five:

cd inventory.five
cp hosts.sample hosts
vi hosts

3.4 Configure Variables in hosts File

Replace the following values in your inventory hosts file with your data:

* REPLACE-WITH-USERNAME              (username for your server)
* REPLACE-WITH-IP-HOSTNAME           (IP or hostname of your server)
* REPLACE-WITH-EMAIL-ADDRESS         (your email address)
* REPLACE-WITH-DOMAIN-NAME           (your domain name)
* REPLACE-FRONTEND-IP                (the IP address of your frontend host)
* REPLACE-GRAFANA-USER               (username for grafana, e.g. admin)
* REPLACE-GRAFANA-PASSWORD           (password for grafana user)
* REPLACE-WITH-LDAP-PASSWORD         (ldap master password)
* REPLACE-DEFAULT-LDAP-USER          (default openldap user)
* REPLACE-DEFAULT-LDAP-USER-PASSWORD (openldap user password)
* REPLACE-WITH-MYSQL-ROOT-PASSWORD   (mariadb root password)
* REPLACE-WITH-PROM-SCRAPE-PASSWORD  (password for prometheus stat scraping)

Some of these appear multiple times, for example REPLACE-WITH-EMAIL-ADDRESS, and may differ in value for different parts of the system. You might want a one email address associated with acme.sh registration and another to get system notices from alertmanager.

3.5 Run The site.yml Play

Install a full system by running:

ansible-playbook -i inventory.five/hosts site.yml

Make sure to specify the correct inventory. Single-server setups will use inventory.single, while cluster setups will use inventory.three or inventory.five.

The whole setup can take an hour to complete for new systems, and some errors are expected (and ignored) because they trigger tasks, such a missing keys which need to be created.

3.6 If Problems Occur, Clean Environment And Try Again

You can remove the clone images, persistent data, and wireguard setup with:

ansible-playbook -i inventory.five/hosts clean.yml

Part 4: Exploring The Demonstration vDC

4.1 This Is Great, Now What?

Command Line Activities

You can test some of the functionality in the base system. ssh to a server and try running the following commands as root:

# consul members

root@server1:~/bin # consul members
Node                          Address          Status  Type    Build   Protocol  DC   Partition  Segment
server1_consul_server   alive   server  1.12.4  2         dc1  default    <all>
server2_consul_server   alive   server  1.12.4  2         dc1  default    <all>
server3_consul_server   alive   server  1.12.4  2         dc1  default    <all>
server4_consul_server   alive   server  1.12.4  2         dc1  default    <all>
server5_consul_server   alive   server  1.12.4  2         dc1  default    <all>
server1_beast_server  alive   client  1.12.4  2         dc1  default    <default>
server1_consul_client    alive   client  1.12.4  2         dc1  default    <default>
server1_mariadb_server   alive   client  1.12.4  2         dc1  default    <default>
server1_nomad_server   alive   client  1.12.4  2         dc1  default    <default>
server1_openldap1_server   alive   client  1.12.4  2         dc1  default    <default>
server1_traefikconsul_server   alive   client  1.12.4  2         dc1  default    <default>
server2_consul_client    alive   client  1.12.4  2         dc1  default    <default>
server2_nomad_server   alive   client  1.12.4  2         dc1  default    <default>
server3_consul_client    alive   client  1.12.4  2         dc1  default    <default>
server3_nomad_server   alive   client  1.12.4  2         dc1  default    <default>
server4_consul_client    alive   client  1.12.4  2         dc1  default    <default>
server4_nomad_server   alive   client  1.12.4  2         dc1  default    <default>
server5_consul_client    alive   client  1.12.4  2         dc1  default    <default>
server5_nomad_server   alive   client  1.12.4  2         dc1  default    <default>

Or check the nomad tools in the /root/bin directory:

# bin/nomad-raft-status.sh

root@server1:~ # bin/nomad-raft-status.sh
Node                               ID                                    Address         State     Voter  RaftProtocol
nomad-server-clone.server4.global  bf7c92a3-966b-e245-5ca2-2f544365b008  leader    true   3
nomad-server-clone.server2.global  13b42085-af1c-3e82-6210-2361556fe3cb  follower  true   3
nomad-server-clone.server5.global  4b336424-fe5b-ba84-6bc9-eb67dae5d2c6  follower  true   3
nomad-server-clone.server3.global  ca64ecf5-a7b0-1910-8769-681835ad6dc1  follower  true   3
nomad-server-clone.server1.global  2f13c1c4-43a0-1464-1b78-3cccdb35e616  follower  true   3

And check on nomad jobs:

# bin/nomad-job-status.sh

root@server1:~ # bin/nomad-job-status.sh
ID             Type     Priority  Status   Submit Date
publicwebsite  service  50        running  2023-05-04T22:53:16+02:00
stdwebsite     service  50        running  2023-05-04T22:53:16+02:00

Graphical Interfaces

Wireguard Access

To access the intranet site, you need to connect to the wireguard network as a roadwarrior client.

Two wireguard roadwarrior clients are configured on the primary server. Using the keys created you can add a wireguard link to the server.

root@server1:/usr/local/etc/wireguard # ls -al
total 26
drwx------   2 root  wheel    11 May  4 22:48 .
drwxr-xr-x  24 root  wheel    42 May  1 21:27 ..
-rw-------   1 root  wheel    45 May  4 22:48 admin_one_preshared.key
-rw-------   1 root  wheel    45 May  4 22:48 admin_one_private.key
-rw-------   1 root  wheel    45 May  4 22:48 admin_one_public.key
-rw-------   1 root  wheel    45 May  4 22:48 admin_two_preshared.key
-rw-------   1 root  wheel    45 May  4 22:48 admin_two_private.key
-rw-------   1 root  wheel    45 May  4 22:48 admin_two_public.key
-rw-------   1 root  wheel    45 May  4 22:48 server_private.key
-rw-------   1 root  wheel    45 May  4 22:48 server_public.key
-rw-------   1 root  wheel  1283 May  4 22:48 wg0.conf

Get the contents of admin_one_preshared.key, admin_one_private.key, admin_one_public.key and server_public.key and configure a wireguard connection on your client host:

PrivateKey = { contents admin_one_private.key } 
Address =
ListenPort = 51820
MTU = 1360

PublicKey = { contents server_public.key }
PresharedKey = { contents admin_one_preshared.key }
Endpoint = { server1 public IP }:51820
AllowedIPs =
PersistentKeepalive = 25

Then connect wireguard to join the environment.

Dashboards And More

You should now be able to access the intranet page at, with links to the following services on different IPs within the environment.

  • Alertmanager
  • Grafana
  • Prometheus
  • Consul UI
  • Nomad UI
  • Traefik-Consul Dashboard

4.2 Build Your Own Applications!

You can now build our own applications, to be run via nomad job, or as additional pot images. Monitoring and alerting is catered for, along with authentication via opeldap, or database in mariadb.

Perhaps you’d like to add a mail server pot image, or a run one or many wordpress sites via nomad jobs?

The basic foundation is in place to get experimenting, or adapted to your needs.

4.3 Potential Applications For The vDC

Multiple live environments make use of a similar setup to this environment, using pot images on FreeBSD.

The infrastructure which hosts this site, or the Potluck image repository, all works in the same way, just with more pot image applications and nomad jobs in the mix.

Part 5: Expanding Knowledge And Resources

5.2 Advanced Topics

You can check out the other pot images on the Potluck site and integrate them manually, or with updates to the ansible playbooks.

You can also try your hand at building pot images within your environment. Check out the sources on github for ideas.

5.3 Feedback

We’d love to get some feedback, or even better, pull requests on github. Let us know how the setup performs in your environment.


The ansible pot foundation vDC is an easy to deploy, functional environment to host your applications. It can help reduce the time taken to setup an environment evaluating FreeBSD with pot containers.