Aug 2, 2020

FreeBSD Virtual Data Centre with Potluck: DevOps & Infrastructure as Code - Part II

Content

This series is split in three parts:

Part I - Overview and Basic Setup
Part II - Setting Up Consul, Nomad & Traefik
Part III - Testing the Environment with Nginx & Git

Step 3 - Nomad, Consul & Traefik Servers

Note 1: If you do not want to use the prebuilt images as shown below for whatever reason, you can easily recreate the jails from the flavour configuration files yourself by following the instructions on the page of each of the images at Potluck.

Installing the consul, nomad and traefik servers on host 10.10.10.10 is as simple as cloning the three Potluck images with 3*5 commands.

This creates three jails with consul listening on 10.10.10.12, nomad listening on 10.10.10.13 and traefik listening on 10.10.10.14. Please replace em0 below with the network interface of your host.

Note 2: Potluck uses Let’s Encrypt certificates, so if you get certificate errors when running pot import, install the root certificate package first with pkg install ca_root_nss.

# consul
$ pot import -p consul-amd64-12_1 -t 1.0 -U https://potluck.honeyguide.net/consul 
...
$ pot clone -P consul-amd64-12_1_1_0 -p consul-clone -N alias -i "em0|10.10.10.12" 
$ pot set-env -p consul-clone -E DATACENTER=my-vdc -E NODENAME=consulserver -E IP=10.10.10.12
$ pot set-attr -p consul-clone -A start-at-boot -V True
$ pot start consul-clone

# nomad 
$ pot import -p nomad-server-amd64-12_1 -t 1.0 -U https://potluck.honeyguide.net/nomad-server
...
$ pot clone -P nomad-server-amd64-12_1_1_0 -p nomad-server-clone -N alias -i "em0|10.10.10.13" 
$ pot set-env -p nomad-server-clone -E DATACENTER=my-vdc -E IP=10.10.10.13 -E CONSULSERVER=10.10.10.12
$ pot set-attr -p nomad-server-clone -A start-at-boot -V True 
$ pot start nomad-server-clone

# traefik
$ pot import -p traefik-consul-amd64-12_1 -t 1.1 -U https://potluck.honeyguide.net/traefik-consul 
...
$ pot clone -P traefik-consul-amd64-12_1_1_1 -p traefik-consul-clone -N alias -i "em0|10.10.10.14"
$ pot set-env -p traefik-consul-clone -E CONSULSERVER=10.10.10.12 
$ pot set-attr -p traefik-consul-clone -A start-at-boot -V True
$ pot start traefik-consul-clone

The nomad and traefik instances are linked to the consul instance through the CONSULSERVER=10.10.10.12 configuration parameter.

In the example above, we have cloned the images coming from Potluck to assign static host addresses to them.

Aside from nomad which needs a routable address not being NATted, you could run consul and traefik also with the public bridge default configuration of the image and port forwarding.
For details, see the traefik and consul Potluck pages and note that the consul GIT example below will not be as simple since port forwarding for UDP ports (necessary for DNS) is a little bit more complicated with the current pot version.

Step 4 - Nomad Compute Nodes (Clients)

Unfortunately, for the compute nodes it is not possible to provide an image or flavour as the setup needs to be done on the (physical or virtual) host and not inside a jail - the compute node needs to be able to use pot to start containers after all.

The setup is not complicated though and can easily be automated via e.g. ansible or salt.

Since only consul clients may speak with the consul server we set up above, each compute node needs to have consul installed as well:

$ pkg install consul nomad nomad-pot-driver
...
$ sysrc nomad_enable="YES"
$ sysrc nomad_user="root"
$ sysrc nomad_env="PATH=/usr/local/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/sbin:/bin"
$ sysrc nomad_args="-config=/usr/local/etc/nomad/client.hcl"
$ sysrc consul_enable="YES"

Also, we need a work around for a bug in the /usr/local/etc/rc.d/consul script:

$ sysrc consul_group="wheel"

Create the directory /usr/local/etc/consul.d:

$ mkdir -p /usr/local/etc/consul.d && chmod 750 /usr/local/etc/consul.d

…and there this agent.json file:

{

    "bind_addr": "10.10.10.11",
    "server": false,
    "datacenter": "my-vdc",
    "log_level": "INFO",
    "enable_syslog": true,
    "leave_on_terminate": true,
    "start_join": [
        "10.10.10.12"
    ]
}

Also create a log directory and file:

$ mkdir -p /var/log/consul && touch /var/log/consul/consul.log

Our /usr/local/etc/nomad/client.hcl looks like this:

bind_addr = "10.10.10.11"

plugin_dir = "/usr/local/libexec/nomad/plugins"
datacenter = "my-vdc"

client {
  enabled = true
  options {
    "driver.raw_exec.enable" = "1"
  }
  servers = ["10.10.10.13"]
}

consul {
  # The address to the Consul agent.
  address = "127.0.0.1:8500"

  # The service name to register the server and client with Consul.
  client_service_name = "test-compute-node"

  # Enables automatically registering the services.
  auto_advertise = true

  # Enabling the server and client to bootstrap using Consul.
  client_auto_join = true
}

enable_syslog=true
log_level="INFO"
syslog_facility="LOCAL1"

Again create a log file:

$ mkdir -p /var/log/nomad && touch /var/log/nomad/nomad.log

…and start both services:

$ /usr/local/etc/rc.d/consul start
...
$ /usr/local/etc/rc.d/nomad start

Access Consul, Nomad & Traefik Dashboards

Consul

Each of the servers has web dashboards.

The consul dashboard lists all the services that you have registered, here it already shows the nginx service you will start in step 5:

Consul dashboard

Nomad

The nomad dashboard is running at http://10.10.10.13:4646:

Nomad Dashboard

You can place the nomad jobs described later simply via the dashboard at http://10.10.10.13:4646/ui/jobs/run.

Nomad Job Placement

Traefik

The traefik dashboard is running at http://10.10.10.14:9002:

Traefik dashboard