Jul 8, 2020

minio Distributed Mode on FreeBSD

Introduction

minio is a well-known S3 compatible object storage platform that supports high availability features.

For FreeBSD a port is available that has already been described in 2018 on the vermaden blog.

Nonetheless, for a distributed setup along the lines of the minio documentation with TLS encryption, even the official minio documentation unfortunately lacks some detail.

That means the certificate setup below might be interesting even if you plan to run minio on another platform and not FreeBSD.

In the offical TLS documentation linked to above, self signed certificates are described but not how a Root CA certificate is created which is needed when minio is to be run not only on a single host.

The setup in this document assumes two hosts with two drives each, the minimum setup for enabling Erasure Code. This setup can easily be scaled to more nodes and also managed by tools like ànsible.

Both hosts in this example use the IP addresses 10.10.10.10 and 10.10.10.11 and each mounts its two drives at /mnt/minio-drive1 and /mnt/minio-drive2.

Installing The Packages

Install the following packages on both hosts to run minio:

$ pkg install minio 
...
$ pkg install openssl
...

Certificate Set Up

On both hosts, create /usr/local/etc/ssl and /usr/local/etc/ssl/CAs directories.

On one of the hosts, run the following commands in the /usr/local/etc/ssl/CAs directory:

openssl genrsa -out rootca.key 8192
openssl req -sha256 -new -x509 -days 3650 -key rootca.key -out rootca.crt

Copy these root certificates into the /usr/local/etc/ssl/CAs directory of the other system.

On both systems, create /usr/local/etc/ssl/openssl.conf. Replace the example IP address 10.10.10.10 below with the IP address of the host on which you create this file:

basicConstraints = CA:FALSE
nsCertType = server
nsComment = "OpenSSL Generated Server Certificate"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer:always
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = 10.10.10.10

This alt name is important as minio will check the certificate. Please note that the document Use OpenSSL (with IP address) to Generate a Certificate also lists a similar configuration file which does not correctly add the alt names to the certificate though.

Afterwards, in the same directory, run

$ openssl genrsa -out private.key 4096
...
$ openssl req -new -key private.key -out public.csr
...
$ openssl x509 -req -in public.csr -CA CAs/rootca.crt -CAkey CAs/rootca.key -CAcreateserial -out public.crt  -days 3650 -sha256 -extfile openssl.conf

Feel free to use appropriate values for each of the certificate attributes when asked by openssl.

You can check if the correct alt name IP address entry is contained in the certificate with openssl x509 -noout -text -in public.crt - if you don’t see it in the output, check your configuration file above.

Configure minio Service

Add the following to the /etc/rc.conf file on both systems:

minio_enable=“YES“
minio_disks=“https://10.10.10.10:9000/mnt/minio-data1 https://10.10.10.10:9000/mnt/minio-data2 https://10.10.10.11:9000/mnt/minio-data1 https://10.10.10.11:9000/mnt/minio-data2“
minio_certs="/usr/local/etc/ssl"
minio_env="MINIO_ACCESS_KEY=myaccesskey MINIO_SECRET_KEY=mysecretkey"

Prepare Drives and Start minio

On both systems, run

$ chown minio /mnt/minio-data1 && chmod u+rxw /mnt/minio-data1
$ chown minio /mnt/minio-data2 && chmod u+rxw /mnt/minio-data2

To test your configuration, you can run minio from the shell so you see the output:

su -m minio -c 'env \\
MINIO_ACCESS_KEY=myaccesskey \\
MINIO_SECRET_KEY=mysecretkey \\
minio server \\
-S /usr/local/etc/ssl \\
https://10.10.10.10:9000/mnt/minio-data1 \\
https://10.10.10.10:9000/mnt/minio-data2 \\
https://10.10.10.11:9000/mnt/minio-data1 \\
https://10.10.10.11:9000/mnt/minio-data2'

Please note that the –config-dir parameter shown in old documentation is obsolete as minio stores the configuration within the data directories meanwhile.

When everything works, you should be able to connect by using the access key and the secret key from above as credentials to both systems with your browser on https://10.10.10.10:9000 and https://10.10.10.11:9000 and both hosts should replicate all the data that you load into each of them.

If everything works, stop both minio server processes with kill -9 and start them as service with /usr/local/etc/rc.d/minio start.

See Also