minio is a well-known S3 compatible object storage platform that supports high availability and scalability features and is very easy to configure.
There is a separate post already describing how to set up
minio on FreeBSD.
This post explains how you can use
minio (or any other S3-compatible storage platform) to provide HA filesystems on FreeBSD.
We describe two ways to mount S3 buckets into the file system on FreeBSD:
- s3fs (
pkg install fusefs-s3fs)
- s3backer (
pkg install fusefs-s3backer) with any filesystem like ZFS or FFS on top
s3fs provides direct access to the bucket content but is relatively slow. It can be mounted on many nodes at the same time though.
s3backerprovides the S3 bucket as block device and you can use any file system like ZFS on top of it. With appropriately configured caching it can be very fast since transactions are asynchronous.
zpool or other file systems can only be mounted on one host at the same time, it is suitable e.g. for HA storage or if a container (jail) can be run on different hosts but not in parallel (e.g. in a failover scenario).
Then, the filesystem in the bucket can be unmounted and re-mounted on another host via network quickly.
Of course, network drives can also be mounted e.g. via NFS, but
minio as storage layer provides scale out and high availability features that are not easily replicated by NFS file servers or NAS systems.
The following steps use the same values as the previous post explaining how to set up minio.
fusefs & Credential Preparations
To ensure that fusefs-based filesystems can be run on FreeBSD, you need to add
$ echo fuse_load="YES" >> /boot/loader.conf
Either reboot or load the module the first time by hand:
$ kldload fuse
Also, both S3-tools need the access key. Create a file containing the login credentials from your
$ echo "myaccesskey:mysecretkey" > /usr/local/etc/s3fs.accesskey
Mount Bucket as Filesystem
Install the file system driver:
$ pkg install fusefs-s3fs
Create a mount point, e.g.
/mnt/s3bucket and the bucket you want to mount, in our case
test-bucket (simply via the
minio web interface).
Then you can mount the bucket already:
$ s3fs test-bucket /mnt/s3bucket -o passwd_file=/usr/local/etc/s3fs.accesskey,url=https://10.10.10.10:9000 -o use_path_request_style -o use_cache=/tmp/s3fs -o enable_noobj_cache -o no_check_certificate -o use_xattr -o complement_stat
You can get a detailed description of the options via
If you want to mount the bucket automatically on boot, add the following line to your
test-bucket /mnt/s3bucket fuse rw,_netdev,allow_other,mountprog=/usr/local/bin/s3fs,late,passwd_file=/usr/local/etc/s3fs.accesskey,url=https://10.10.10.10:9000,use_path_request_style,use_cache=/tmp/s3fs,enable_noobj_cache,no_check_certificate,use_xattr,complement_stat 0 0
Bucket As Raw Device With ZFS
To create a filesystem like ZFS inside the bucket, install
$ pkg install fusefs-s3backer
We again use an empty bucket called
test-bucket which we mount from the command line first (assuming that you have unmounted the
s3fs-bucket above first):
$ s3backer --blockSize=128k --size=500g --listBlocks test-bucket --baseURL=https://10.10.10.10:9000/ --insecure --accessFile=/usr/local/etc/s3fs.accesskey --blockCacheFile=/tmp/s3backer.cache --blockCacheSize=1000000 /mnt/s3bucket
This command creates a 500GB raw device and also sets the cache size to 1,000,000 128k blocks.
Then let us create the
$ zpool create -O compression=lz4 -O atime=off s3pool /mnt/s3bucket/file
Note that there is file being appended to the path that you have used for mounting the bucket.
Àfter it has been created, it can be imported simply by running e.g.
$ zpool import -d /mnt/s3bucket s3pool
You can also mount the block device again via
s3pool /mnt/s3bucket fuse rw,allow_other,mountprog=/usr/local/bin/s3backer,late,blockSize=128k,size=500g,listBlocks,baseURL=https://10.10.10.10:9000/,insecure,accessFile=/usr/local/etc/s3fs.accesskey,blockCacheFile=/tmp/s3backer.cache,blockCacheSize=1000000 0 0
After mounting the block device in
/etc/fstab, you will want to mount the the