Jul 23, 2020

ZFS High Availability Filesystem With minio on FreeBSD

Introduction

minio is a well-known S3 compatible object storage platform that supports high availability and scalability features and is very easy to configure.

There is a separate post already describing how to set up minio on FreeBSD.

This post explains how you can use minio (or any other S3-compatible storage platform) to provide HA filesystems on FreeBSD.

We describe two ways to mount S3 buckets into the file system on FreeBSD:

  • s3fs (pkg install fusefs-s3fs)
  • s3backer (pkg install fusefs-s3backer) with any filesystem like ZFS or FFS on top

s3fs provides direct access to the bucket content but is relatively slow. It can be mounted on many nodes at the same time though.

s3backerprovides the S3 bucket as block device and you can use any file system like ZFS on top of it. With appropriately configured caching it can be very fast since transactions are asynchronous.

Since a zpool or other file systems can only be mounted on one host at the same time, it is suitable e.g. for HA storage or if a container (jail) can be run on different hosts but not in parallel (e.g. in a failover scenario).
Then, the filesystem in the bucket can be unmounted and re-mounted on another host via network quickly.

Of course, network drives can also be mounted e.g. via NFS, but minio as storage layer provides scale out and high availability features that are not easily replicated by NFS file servers or NAS systems.

The following steps use the same values as the previous post explaining how to set up minio.

fusefs & Credential Preparations

To ensure that fusefs-based filesystems can be run on FreeBSD, you need to add fuse to /boot/loader.conf:

$ echo fuse_load="YES" >> /boot/loader.conf

Either reboot or load the module the first time by hand:

$ kldload fuse

Also, both S3-tools need the access key. Create a file containing the login credentials from your minio setup:

$ echo "myaccesskey:mysecretkey" > /usr/local/etc/s3fs.accesskey

Mount Bucket as Filesystem

Install the file system driver:

$ pkg install fusefs-s3fs

Create a mount point, e.g. /mnt/s3bucket and the bucket you want to mount, in our case test-bucket (simply via the minio web interface).

Then you can mount the bucket already:

$ s3fs test-bucket /mnt/s3bucket -o passwd_file=/usr/local/etc/s3fs.accesskey,url=https://10.10.10.10:9000 -o use_path_request_style -o use_cache=/tmp/s3fs -o enable_noobj_cache -o no_check_certificate -o use_xattr -o complement_stat

You can get a detailed description of the options via man s3fs.

If you want to mount the bucket automatically on boot, add the following line to your /etc/fstab:

test-bucket		/mnt/s3bucket	    fuse	rw,_netdev,allow_other,mountprog=/usr/local/bin/s3fs,late,passwd_file=/usr/local/etc/s3fs.accesskey,url=https://10.10.10.10:9000,use_path_request_style,use_cache=/tmp/s3fs,enable_noobj_cache,no_check_certificate,use_xattr,complement_stat	0	0

Bucket As Raw Device With ZFS

To create a filesystem like ZFS inside the bucket, install s3backer:

$ pkg install fusefs-s3backer

We again use an empty bucket called test-bucket which we mount from the command line first (assuming that you have unmounted the s3fs-bucket above first):

$ s3backer --blockSize=128k --size=500g --listBlocks test-bucket --baseURL=https://10.10.10.10:9000/ --insecure --accessFile=/usr/local/etc/s3fs.accesskey --blockCacheFile=/tmp/s3backer.cache --blockCacheSize=1000000 /mnt/s3bucket

This command creates a 500GB raw device and also sets the cache size to 1,000,000 128k blocks.

Then let us create the zpool:

$ zpool create -O compression=lz4 -O atime=off s3pool /mnt/s3bucket/file

Note that there is file being appended to the path that you have used for mounting the bucket.

Àfter it has been created, it can be imported simply by running e.g.

$ zpool import -d /mnt/s3bucket s3pool

That’s it.

You can also mount the block device again via /etc/fstab:

s3pool               /mnt/s3bucket       fuse    rw,allow_other,mountprog=/usr/local/bin/s3backer,late,blockSize=128k,size=500g,listBlocks,baseURL=https://10.10.10.10:9000/,insecure,accessFile=/usr/local/etc/s3fs.accesskey,blockCacheFile=/tmp/s3backer.cache,blockCacheSize=1000000 0 0

After mounting the block device in /etc/fstab, you will want to mount the the zpool.

See Also