Introduction #
minio is a well-known S3 compatible object storage platform that supports high availability and scalability features and is very easy to configure.
There is a separate post already describing how to set up minio on FreeBSD.
This post explains how you can use minio (or any other S3-compatible storage platform) to provide HA filesystems on FreeBSD.
We describe two ways to mount S3 buckets into the file system on FreeBSD:
- s3fs (
pkg install fusefs-s3fs) - s3backer (
pkg install fusefs-s3backer) with any filesystem like ZFS or FFS on top
s3fs provides direct access to the bucket content but is relatively slow. It can be mounted on many nodes at the same time though.
s3backerprovides the S3 bucket as block device and you can use any file system like ZFS on top of it. With appropriately configured caching it can be very fast since transactions are asynchronous.
Since a zpool or other file systems can only be mounted on one host at the same time, it is suitable e.g. for HA storage or if a container (jail) can be run on different hosts but not in parallel (e.g. in a failover scenario).
Then, the filesystem in the bucket can be unmounted and re-mounted on another host via network quickly.
Of course, network drives can also be mounted e.g. via NFS, but minio as storage layer provides scale out and high availability features that are not easily replicated by NFS file servers or NAS systems.
The following steps use the same values as the previous post explaining how to set up minio.
fusefs & Credential Preparations #
To ensure that fusefs-based filesystems can be run on FreeBSD, you need to add fuse to /boot/loader.conf:
$ echo fuse_load="YES" >> /boot/loader.conf
Either reboot or load the module the first time by hand:
$ kldload fuse
Also, both S3-tools need the access key. Create a file containing the login credentials from your minio setup:
$ echo "myaccesskey:mysecretkey" > /usr/local/etc/s3fs.accesskey
Mount Bucket as Filesystem #
Install the file system driver:
$ pkg install fusefs-s3fs
Create a mount point, e.g. /mnt/s3bucket and the bucket you want to mount, in our case test-bucket (simply via the minio web interface).
Then you can mount the bucket already:
$ s3fs test-bucket /mnt/s3bucket -o passwd_file=/usr/local/etc/s3fs.accesskey,url=https://10.10.10.10:9000 -o use_path_request_style -o use_cache=/tmp/s3fs -o enable_noobj_cache -o no_check_certificate -o use_xattr -o complement_stat
You can get a detailed description of the options via man s3fs.
If you want to mount the bucket automatically on boot, add the following line to your /etc/fstab:
test-bucket /mnt/s3bucket fuse rw,_netdev,allow_other,mountprog=/usr/local/bin/s3fs,late,passwd_file=/usr/local/etc/s3fs.accesskey,url=https://10.10.10.10:9000,use_path_request_style,use_cache=/tmp/s3fs,enable_noobj_cache,no_check_certificate,use_xattr,complement_stat 0 0
Bucket As Raw Device With ZFS #
To create a filesystem like ZFS inside the bucket, install s3backer:
$ pkg install fusefs-s3backer
We again use an empty bucket called test-bucket which we mount from the command line first (assuming that you have unmounted the s3fs-bucket above first):
$ s3backer --blockSize=128k --size=500g --listBlocks test-bucket --baseURL=https://10.10.10.10:9000/ --insecure --accessFile=/usr/local/etc/s3fs.accesskey --blockCacheFile=/tmp/s3backer.cache --blockCacheSize=1000000 /mnt/s3bucket
This command creates a 500GB raw device and also sets the cache size to 1,000,000 128k blocks.
Then let us create the zpool:
$ zpool create -O compression=lz4 -O atime=off s3pool /mnt/s3bucket/file
Note that there is file being appended to the path that you have used for mounting the bucket.
Àfter it has been created, it can be imported simply by running e.g.
$ zpool import -d /mnt/s3bucket s3pool
That’s it.
You can also mount the block device again via /etc/fstab:
s3pool /mnt/s3bucket fuse rw,allow_other,mountprog=/usr/local/bin/s3backer,late,blockSize=128k,size=500g,listBlocks,baseURL=https://10.10.10.10:9000/,insecure,accessFile=/usr/local/etc/s3fs.accesskey,blockCacheFile=/tmp/s3backer.cache,blockCacheSize=1000000 0 0
After mounting the block device in /etc/fstab, you will want to mount the the zpool.