Mastodon and Related Infrastructure with Podman Containers #
Introduction #
We previously documented running Mastodon using Pot containers in this blog post.
In that post we used ansible to provision a host using a series of tasks in a playbook. You would clone the git repo of the playbook, configure some settings in an inventory file, and run the ansible playbook to provision your server.
Since then podman has become available for FreeBSD, along with podman compose, making dockerisation of jails possible.
There is still a way to go with developing podman on FreeBSD before it becomes as user-friendly as docker or podman containers on other systems, but it’s definitely usable now.
This post covers some of the adjustments required, and decisions made, to run Mastodon and related services with podman containers on FreeBSD.
Background: Available FreeBSD Container Images #
First some background on the different FreeBSD container images, aka OCI-format images.
The FreeBSD Foundation releases formal images for every release, and these are also available on Docker Hub and Github Container Registry.
As written by Doug Rabson in his FreeBSD Foundation blog post:
OCI container engines such as containerd or podman need images. A container image is a read-only directory tree which typically contains an application with supporting files and libraries. Running this image on a container engine makes a writable clone of the image and executes the application in some kind of isolation environment such as a jail.
Images are distributed via registries which store the image data and provide a simple REST API to access images and their metadata. The registry APIs, image formats and metadata are standardised by the Open Container Initiative which largely replaces earlier docker formats.
For example, with FreeBSD 15.0, the following images are available:
- FreeBSD-15.0-RELEASE-amd64-container-image-static.txz
- FreeBSD-15.0-RELEASE-amd64-container-image-dynamic.txz
- FreeBSD-15.0-RELEASE-amd64-container-image-runtime.txz
- FreeBSD-15.0-RELEASE-amd64-container-image-notoolchain.txz
- FreeBSD-15.0-RELEASE-amd64-container-image-toolchain.txz
A detailed explanation for the types of FreeBSD OCI images is paraphrased below, sourced from this pull request for the FreeBSD handbook.
freebsd-static #
The static image is intended as a base image, for a workload which is entirely statically linked. It contains no libraries, nor binaries, just the supporting files that most applications of this nature require. There is no UNIX shell, no command-line tools, no dynamic libraries, nor package manager.
freebsd-dynamic #
The dynamic image uses the static image as a parent layer, and supports using shared libraries, including libc. It doesn’t have a shell, rc system, nor a package manager.
freebsd-runtime #
Again, runtime builds on the preceding dynamic layer, and finally adds the minimum that a user would expect - a UNIX shell, rc system, and the package manager pkg.
freebsd-notoolchain #
This base image contains almost all tools one would expect on a typical FreeBSD system, excluding those that are directly hardware-related, and thus not generally useful within a container, and the compiler and related toolchain, as it is quite large.
freebsd-toolchain #
The Toolchain base image is the sum of all preceding images, including a full compiler and toolchain.
Real world usage notes #
Contrary to the proposed handbook documentation, we’ve found that the most useful OCI images are the freebsd-notoolchain and freebsd-toolchain ones.
While initial development and experimentation with the freebsd-runtime container image proved useful, there are coverage gaps better suited by the freebsd-notoolchain and freebsd-toolchain images.
For example, to build a varnish container you will need the freebsd-toolchain base container, because varnish compiles its VCL at runtime. This needs a compiler.
It’s a labourious exercise to install compiler tools on a leaner container base, and far simpler to use the applicable OCI image which has a compiler already installed.
Although a larger base image to download, the build process for custom containers completes much faster, which means a quicker live service.
Background: Podman Container Hooks #
Not all FreeBSD services will run automatically with podman containers. For example, if you try to build a postgresql container, then regardless of which OCI image you use, when you run it you will get complaints about shared memory, and postgresql will refuse to start.
The solution is to edit jail parameters to configure sysvshm, sysvsem and sysvmsg along with some others. Not an obvious solution, but there is a way to automate it.
Podman has the ability to run scripts via a container hooks subsystem, which can modify containers at various stages of the container lifecycle.
This is first discussed in this github issue from March 2025, along with a Mastodon thread from David Chisnall in May 2025.
The podman container hooks concept is also explored in a series of blog posts by Dave Cottlehuber:
- Using Podman hooks to mount persistent ZFS datasets into ephemeral Containers
- Using Podman hooks to attach Nebula mesh networking to containers
- Extending OCI container networks with Hooks
We cover our implementation of automatic container hooks, specifically for postgresql and opensearch containers, and annotations in podman-compose.yml, in the setup of micropod-mastodon-suite below.
What is required to run Mastodon? #
At the most basic level, all you need to run Mastodon is a database, redis, and a webserver.
You might also want S3 storage for media, as well as elasticsearch or opensearch integration for search.
The setup we cover below is a bit more involved, inclusive of database backups, haproxy and varnish cache, separate containers for the different sidekiq queues, and mastodon-streaming and mastodon-web processes. Serving media from S3 is a separate nginx container from the one which provides a reverse proxy to the mastodon-web process.
On a ~100 user system you will need a host or virtual machine with at least 16 vCPU and 24GB memory, and at least 80GB disk space when media storage is in separate S3, or 300GB or more if not.
Micropod-mastodon-suite #
Micropod-mastodon-suite is a podman compose recipe for running a suite of services to support a Mastodon instance.
It assumes you have external S3 storage for media files, along with a separate firewall/proxy frontend to handle SSL certificates and haproxy rules to the internal host.
It uses included shell scripts for starting and stopping the containers in a staggered order, rather than the conventional podman compose up -d or podman compose down.
System Preparation #
You will need to prepare your FreeBSD 15.0 (or higher) host to run podman containers, along with additional preparation for podman container hooks to be able to run the postgresql and opensearch containers.
All the following commands are run by the root user. Podman on FreeBSD doesn’t support rootless containers.
Create ZFS datasets #
Podman will by default store containers in /var/db/containers. Make sure there is a ZFS dataset as follows
zfs create -o mountpoint=/var/db/containers zroot/containers
It is also recommended to create a ZFS dataset to clone the repo to, and retain stored data.
zfs create -o mountpoint=/mnt/data zroot/data
/mnt/data could also be an NFS mount to a host server if running on a vm-bhyve virtual machine.
Create a container logs directory #
Make sure there is a directory /var/log/containers for container logs.
mkdir -p /var/log/containers
Most containers in Micropod-mastodon-suite will write log files here, with the exception of
postgresqlwhich writes internally and is limited to 10 MB in size. Failure to limitpostgresqlcontainer logs will quickly use up all available disk space on a Mastodon instance.
Set package stream to “latest” #
Make sure you’re on the latest package stream as follows:
mkdir -p /usr/local/etc/pkg/repos
nano /usr/local/etc/pkg/repos/FreeBSD.conf
and add contents
FreeBSD-ports: {
url: "pkg+https://pkg.FreeBSD.org/${ABI}/latest",
mirror_type: "srv",
signature_type: "fingerprints",
fingerprints: "/usr/share/keys/pkg",
enabled: yes
}
Then run
pkg update -f
pkg upgrade -yf
Install podman, podman-compose, git, jq #
Install podman-suite, py311-podman-compose, git and jq as follows
pkg install podman-suite py311-podman-compose git jq
Configure PF firewall #
The following pf ruleset differs from the documented one for Podman on FreeBSD as it supports IPv6.
Create /etc/pf.conf with the following contents, making sure to set vtnet0 to the interface name of your host or virtual machine if it differs:
v4egress_if = "vtnet0"
v6egress_if = "vtnet0"
cni_if0 = "cni-podman0"
cni_if1 = "cni-podman1"
table <cni-nat> persist { 10.88.0.0/16, 10.89.0.0/24, fdca:1db3:f328:2d7c::/64 }
nat on $v4egress_if inet from <cni-nat> to any -> ($v4egress_if)
nat on $v6egress_if inet6 from <cni-nat> to !ff00::/8 -> ($v6egress_if)
rdr-anchor "cni-rdr/*"
nat-anchor "cni-rdr/*"
nat-anchor "cni-nat/*"
anchor "cni-nat/*"
# Pass rules so packets aren’t blocked
pass quick on $cni_if0 inet from ($cni_if0:network) to any keep state
pass quick on $cni_if1 inet from ($cni_if1:network) to any keep state
pass quick on $cni_if0 inet6 from ($cni_if0:network) to any keep state
pass quick on $cni_if1 inet6 from ($cni_if1:network) to any keep state
pass out quick on $v4egress_if inet from ($cni_if0:network) to any keep state
pass out quick on $v4egress_if inet from ($cni_if1:network) to any keep state
pass out quick on $v6egress_if inet6 from ($cni_if0:network) to any keep state
pass out quick on $v6egress_if inet6 from ($cni_if1:network) to any keep state
# ICMP/ICMPv6 (ping + PMTU)
pass inet proto icmp from ($cni_if0:network) to any keep state
pass inet proto icmp from ($cni_if1:network) to any keep state
pass inet6 proto icmp6 from ($cni_if0:network) to any keep state
pass inet6 proto icmp6 from ($cni_if1:network) to any keep state
Configure the following sysctl parameter:
sysctl net.pf.filter_local=1
echo "net.pf.filter_local=1" >> /etc/sysctl.conf
Enable pf and pflog services, and start pf:
service pf enable
service pflog enable
service pf start
Enable gateway services for correct routing #
Configure the following gateway services for the host to ensure correct routing:
sysrc gateway_enable="YES"
sysrc ipv6_gateway_enable="YES"
service routing restart
Configure fdescfs mount #
Make sure fdescfs is mounted, and will mount on reboot. This is required for Podman.
mount -t fdescfs fdesc /dev/fd
echo "fdesc /dev/fd fdescfs rw 0 0" >> /etc/fstab
Enable and start the podman_service and podman services #
Enable and start the podman_service service as follows:
service podman_service enable
service podman_service start
Enable and start the podman service as follows:
service podman enable
service podman start
Both services must be enabled. The podman service will ensure containers restart automatically on reboot.
Setup source containers with official FreeBSD OCI images #
Use podman to load the relevant source FreeBSD OCI images:
podman pull docker.io/freebsd/freebsd-notoolchain:15.0
podman pull docker.io/freebsd/freebsd-toolchain:15.0
You only need to do this once, it doesn’t need to be repeated for every container build.
Create custom podman hooks #
The postgresql container needs the following jail parameters enabled
sysvmsg=1 sysvsem=1 sysvshm=1
The opensearch container needs the following jail parameters enabled
allow.mount=1 allow.mount.fdescfs=1 allow.mount.procfs=1 enforce_statfs=1 allow.mlock=1
We will automate this by creating custom hooks for podman.
Create the file /usr/local/etc/containers/hooks.d/customattributes.json with contents:
{
"version": "1.0.0",
"hook": {
"path": "/usr/local/etc/containers/hooks.d/customattributes.sh"
},
"when": {
"annotations": {
"^com\\.micropod\\.jail\\.profile$": "^(postgresql|opensearch)$"
}
},
"stages": [
"createRuntime"
]
}
Then create the file /usr/local/etc/containers/hooks.d/customattributes.sh with contents:
#!/bin/sh
set -eu
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
STATE="$(cat)"
CONTAINER_ID="$(printf '%s' "$STATE" | jq -r '.id')"
PROFILE="$(printf '%s' "$STATE" | jq -r '.annotations["com.micropod.jail.profile"] // empty')"
JAIL_STATE="/var/run/ocijail/${CONTAINER_ID}/state.json"
JAIL_ID="$(jq -r '.jid' < "$JAIL_STATE")"
case "$PROFILE" in
postgresql)
jail -m jid="$JAIL_ID" \
sysvmsg=1 \
sysvsem=1 \
sysvshm=1
logger -t oci "custom jail profile=postgresql container=${CONTAINER_ID} jid=${JAIL_ID}"
;;
opensearch)
jail -m jid="$JAIL_ID" \
allow.mount=1 \
allow.mount.fdescfs=1 \
allow.mount.procfs=1 \
enforce_statfs=1 \
allow.mlock=1
logger -t oci "custom jail profile=opensearch container=${CONTAINER_ID} jid=${JAIL_ID}"
;;
*)
logger -t oci "unknown or missing jail profile='${PROFILE}' container=${CONTAINER_ID}"
exit 0
;;
esac
Make it executable:
chmod +x /usr/local/etc/containers/hooks.d/customattributes.sh
Then make sure the file /usr/local/etc/containers/containers.conf has the hooks section enabled (uncommented) and a custom directory location set as follows:
[engine]
hooks_dir = [
"/usr/local/etc/containers/hooks.d/",
]
Finally, restart the podman_service service:
service podman_service restart
In the podman-compose.yml file in the repo below, the annotations are set for the postgresql image as
postgresql:
...
container_name: postgresql
annotations:
com.micropod.jail.profile: "postgresql"
and for the opensearch image as
opensearch:
...
container_name: opensearch
annotations:
com.micropod.jail.profile: "opensearch"
These annotations will trigger the custom podman container hooks and set the required jail parameters as specified in /usr/local/etc/containers/hooks.d/customattributes.sh.
System preparation is now complete. You can move onto setting up Micropod-mastodon-suite.
Cloning the git repo and configuring .env file #
Clone the Micropod-mastodon-suite repo #
Clone the Micropod-mastodon-suite repo, in the ZFS dataset or NFS mounted directory, as follows:
cd /mnt/data
git clone https://codeberg.org/Honeyguide/micropod-mastodon-suite.git
cd micropod-mastodon-suite/
Configure the .env file with your specific parameters #
Copy the sample.env file to .env and edit according to your needs.
You will need to set IPv4 and IPv6 addresses for the host, as well as IPv4 and IPv6 addresses of the upstream firewall/proxy host.
Set your domain name and CDN domain name.
Make sure to include the IPv4 address of your S3 host, along with the applicable ports. Include SMTP server and credentials as well.
# Host specific variables
# This is your VM's IP address
IP4_ADDRESS=""
# make sure to have square brackets around IPv6 address here
IP6_ADDRESS="[]"
# External proxy IP addresses, usually upstream Haproxy's IP addresses, used for real IP forwarding in nginx
# Do not set square brackets on v6 address here!
PROXY_IP_V4=""
PROXY_IP_V6=""
# Application specific variables
# database
DB_HOST="postgres"
DB_PORT="5432"
DB_NAME="mastodon_production"
DB_USER="mastodon"
DB_PASSWORD="mastodon"
# mastodon variables
RAILS_ENV="production"
DOMAIN="example.com"
SECRET_KEY=""
OTP_SECRET=""
VAPID_PRIVATE_KEY=""
VAPID_PUBLIC_KEY=""
ACTIVE_PRIMARY_KEY=""
ACTIVE_DETERMINISTIC_KEY=""
ACTIVE_KEY_DERIVATION_SALT=""
MY_MAIL_HOST=""
MY_MAIL_PORT=""
MY_MAIL_USERNAME=""
MY_MAIL_PASSWORD=""
MY_MAIL_FROM_ADDRESS=""
S3_BUCKET="mastodon"
S3_REGION="garage"
S3_USER=""
S3_PASSWORD=""
S3_CDN_HOSTNAME=""
# leave elastic user and password empty for now
ELASTIC_USERNAME=""
ELASTIC_PASSWORD=""
DEEPL_API_KEY=""
DEEPL_API_PLAN=""
# for haproxy container to S3 storage
# Add IP address of your S3 storage and set ports if different from the garage web port used here
S3_IP_ADDRESS_1=""
S3_READ_PORT_NUMBER="3902"
S3_WRITE_PORT_NUMBER="3900"
If you do not have existing values for secret key, OTP secret, vapid keys, active keys, leave these blank as they will be auto-generated and stored for future use on container restarts.
It is not necessary to set a username or password for elasticsearch (which is actually opensearch).
Building and running the containers #
We’ve adopted a build local approach to podman containers, as opposed to pulling prebuilt containers from a container registry. This is an opinionated choice.
For some podman compose setups we’ve also adopted custom shell scripts to start/stop containers in a defined build and run order, as opposed to using podman compose up -d or podman compose down directly.
For Micropod-mastodon-suite you can initiate building and running the containers with:
./start-mastodon-suite.sh
This step can take 30-45 minutes to complete, and the containers will be running automatically.
The script will build all containers and store images locally. Then it will start them in a staggered fashion, where some containers do specific things and exit, while others serve as templates for service-specific containers.
For example, the maintaindb will perform administrative functions on the postgresql server before any Mastodon containers are running.
The mastodon container will perform all the necessary steps to build a Mastodon-capable container image, but won’t run directly. The various sidekiq processes, and mastodon-streaming and mastodon-web, are duplicates of the source mastodon container with different entrypoints configured for each.
When done, check the output of the following command to see if all images are live:
podman ps -a
To stop the containers you can run:
./stop-all-containers.sh
Setting up a new instance #
There are many scripts in the scripts/ subdirectory that can perform different functions. On a freshly installed host we recommend the following:
Setup Admin User #
To setup an admin user, run the script scripts/run-create-admin-user.sh as follows:
./scripts/run-create-admin-user.sh --username name --email email@address
View the credentials in the file data/mastodon/private/mastodon.owner.credentials to login as first user.
Setup opensearch index #
To setup opensearch indexes run
./scripts/run-setup-es-index.sh
Enable query_stats in postgresql #
To enable pg_stat_statements in the pgdata tool for administrators, run
./scripts/run_enable_postgresql_query_stats.sh
External monitoring and restarting of sidekiq queues #
Podman on FreeBSD doesn’t have podman container healthcheck monitoring like it does on Linux, as this is a systemd service.
Instead you have to build your own monitoring solution via cron jobs.
If you want to do external monitoring of the sidekiq queues, and trigger automatic container restarts on wedged queues, create a cron job for the root user with crontab -e.
Paste the following in, making sure to configure the INSTANCE variable to match the IP address of the host running Micropod-mastodon-suite, and setting the MONITORHOST variable to be the IP address of an alertmanager server, if available.
MAILTO=""
INSTANCE="1.2.3.4"
MONITORHOST="1.2.3.5"
*/2 * * * * QUEUES="ingress:mastodon-sidekiq-ingress-1 ingress:mastodon-sidekiq-ingress-2 pull:mastodon-sidekiq-pull push:mastodon-sidekiq-push" /mnt/data/micropod-mastodon-suite/scripts/cron-sidekiq-healthcheck.sh
This will monitor the 2 sidekiq ingress queues, and sidekiq pull and push queues.
You can list all the sidekiq queues to be monitored if desired, see the source for scripts/cron-sidekiq-healthcheck.sh
Do not change the timing, it is expected to run every 2 minutes.
An
alertmanagerserver is available as part of micropod-monitoring
Happy tooting!