Veeam TrueNas and S3 bucket with immutability

Alain

Cadet
Joined
Nov 17, 2020
Messages
7
I am hoping to setup a truenas box and use the built in Minio server to create S3 buckets with immutability to take advantage of Veeam's guaranteed ransomware recovery.

I have been able to create an locked S3 bucket using SSH commands but veeam also requires versioning on the bucket. My understanding is that versioning , in turn, requires erasure coding to be enabled.

Unfortunately It seems to me that it is not possible to enable erasure coding on the built in minio server. Is this correct?
 

Alain

Cadet
Joined
Nov 17, 2020
Messages
7
Thanks Kris. I had successfully setup the object lock bucket but veeam also requires the versioning feature on the bucket. If I understood correctly, versioning requires erasure coding and if I understand you correctly then erasure coding requires a clustered setup.

I followed the directions provided by minio at Minio erasure coding. However when I connect to the bucket using the S3 browser it shows versioning to be disabled.
 

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
It is possible to do erasure coding with minio locally (without the cluster), you just have to feed it more disks in the configuration. That would probably be a good suggestion ticket on jira.ixsystems.com for us to evaluate if we could bring that support into the UI for a future version. Additionally, there is now a Minio Plugin for TrueNAS 12.0. If you're not afraid of using the CLI a bit, it would be possible to enable erasure coding inside that plugin using their standard documented processes.


You'd need to put the additional disks into the plugin/jails /etc/rc.conf using the "minio_disks=" variable
 

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
Joining in on this one as this is a client request. I'm generally familiar with the CLI, but in the current TrueNAS release, I can't find the associated rc.conf file to modify as it's not showing up under plugin/jails when I activate it using the built-in S3 option from the services section. Or is the only way to do this is by using the plugin instead of the S3 service?
 

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
OK - I've switched over to using the plugin, but by default there are no "minio_disks=" in the conf file.

Code:
cron_flags="$cron_flags -J 15"

# Disable Sendmail by default
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Run secure syslog
syslogd_flags="-c -ss"

# Enable IPv6
ipv6_activate_all_interfaces="YES"
rtsold_enable="YES"
ifconfig_epair0b="SYNCDHCP"
ifconfig_epair0b_ipv6="inet6 auto_linklocal accept_rtadv autoconf"
minio_enable="YES"
minio_env="MINIO_ACCESS_KEY=xxxxxxxxxxxx MINIO_SECRET_KEY=xxxxxxxxxxxx"


What format should that be in? A path to a zvol? Also I just noted that when I'm in the jail CLI "disk geom list" shows me the physical disks, so I'm a little confused about what it's using for storage at the moment and how it sees it as disks (iocage/jails/minio/root)
 

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
OK - after a fair bit of tinkering, the following configuration seems to be working:

Install the plugin
Map a dataset to the jail under /mnt
Create 4 folders (data1... data4) under /mnt
Ensure that they are owned by minio (chown -R minio /mnt from inside the jail)
Add the following line to /etc/rc.conf in the jail
minio_disks="/mnt/data1 /mnt/data2 /mnt/data3 /mnt/data4"
(the shortcut of /mnt/data{1...4} frequently used in the Minio documentation doesn't seem to work in the jail)

@Alain : If you're going to use this with Veeam, you'll also need to create your certificate and key. Use the Minio doc examples with openssl which is available in the jail and move the resulting private.key and public.crt to /usr/local/etc/minio/certs and also chown them to the minio account.

Restart the jail and you should be good to go.

Then you can use the mc client to create a bucket with the object-lock option enabled:
Code:
mc mb mintest/testlock2 --with-lock
Bucket created successfully `mintest/testlock2`.


For a production installation, I would strongly recommend splitting the dataX folders to individual mount points to separate datasets or ideally datasets from separate zpools to ensure a maximum of parallel IO.
 

dwoerz

Cadet
Joined
Mar 5, 2016
Messages
1
@infraerik
Thank you for the writeup, works well and simple to follow.

Quick question, since you are creating a "raid" of folders, do you know what kind of "raid" it is?

Real question being, if i store 10TB in minio, how much actual space will be used up? for example if it was "Raid 5" ~ 12.5TB of used space would be used? I am using this for Veeam. Veeam tells me that i am using 28.2 TB of object storage, but TrueNAS is telling me that i am using 90TB of disk space. Seems a bit excessive and a bit beyond "non optimal"
 

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
This is a bare bones really inefficient way of doing this mostly for a home lab proof of concept :smile:

The default configuration of the Minio erasure coding ends up being the equivalent of mirroring in terms of internal redundancy (on top of whatever your zpool is doing). If you are looking to build something a little more production ready or using any significant amounts of data, then I would either build out individual single disk zpools (or mirrored zpools if this is really important stuff) and map those to the individual "disks" as consumed by Minio. From there you can also adjust the erasure coding redundancy by adding

MINIO_STORAGE_CLASS_STANDARD=EC:2

to the end of the minio_env line with the keys. EC:2 is telling Minio to use 2 "disks" worth of parity as the default storage class (so roughly the equivalent of a RAID6) and is the minimum possible for Minio.

That 28:90 ratio seems *really* large and I'm wondering if there isn't some kind of block size mismatch between the various components that is adding to the inefficiency.

If you're building this off of an existing zpool, then it might be worth multiplying the number of "disks" presented and modifying the erasure code redundancy level down to EC:2 and see how that affects the data usage pattern. Intuitively that should work but I haven't done the math.
 

xness

Dabbler
Joined
Jun 3, 2022
Messages
30
Do you know if MINIO_STORAGE_CLASS_STANDARD can still be modified after creating the initial MinIO configuration?
Found your excellent article @infraerik but wasn't sure about the consequences / benefits the described multi-disk setup has in regards to storage efficiency with Veeam.

I mean after all ZFS is running an error-resilient pool and what would we need versioning or further mirroring for? On first sight this seems like unnecessary overhead, unless actually running MinIO in clustered mode over multiple storage appliances / disks for redundancy. Usually for Veeam backups you require an immutable retention period – which if I understand correctly can be achieved with object locking for which erasure coding is not required – and multiple VM backup restore points which in turn automatically takes care of storing multiple versions – while versioning seems to be the main benefit of erasure coding – no?

Hope you can clear that up– would be great!

edit: Also just saw this in their docs:–
Starting with RELEASE.2022-06-02T02-11-04Z, MinIO implements a zero-parity erasure coded backend for single-node single-drive deployments. This feature allows access to erasure coding dependent features without the requirement of multiple drives.

MinIO only starts in SNSD mode if the storage volume or path is empty or only contain files generated by a previous SNSD deployment. [...]
However the current version in TrueNAS' plugins unfortunately is 2021.12.27.07.23.18_2
 
Last edited:

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
It's complicated :smile:

There are a number of ways to accomplish the task, keeping in mind that Minio is designed to handle the resilience natively which will be redundant with the resilience provided by ZFS.

It's worth noting that Minio's immutability/object lock feature requires erasure coding which can be done on a single server (it doesn't require clustered mode across multiple servers), just multiple disks.

Scenario 1 : Using the standard Minio plugin on top of a ZFS filesystem. In this case, there is no object lock at the Minio layer so that if the server sending data to Minio is compromised, they can request the deletion of the S3 objects. In this case, we can use ZFS snapshots to provide the immutability protection, but there is the question about session integrity as a snapshot may be taken in the middle of a copy operation resulting in an incomplete or mismatched catalog/data situation.

Scenario 1a : As you noted according to the Minio documentation, with the version released on 2022-06-02, you should be able to enable object locking on a single "disk" configuration which would be ideal with an existing ZFS server. But I've been busy this month and haven't yet had a chance to test this out. If you're running TrueNAS Scale, you can use Docker to deploy the latest version, but if you're still on TrueNAS core, you'll have to wait or manually compile it yourself. I've done it once to fix some problems and it's not horribly complicated as it compiles down to a single binary which you can copy into the jail, but we're getting way outside of the standard configuration and operations processes if you're not already used to compiling Go binaries on BSD.

Scenario 2 : Customisation of the Minio plugin to use multiple mountpoints on a ZFS filesystem and with at least 4 mapped directories, erasure coding and object lock can be enabled. Not a good idea since Minio sees them as autonomous endpoints and will try to parallelise IO to the "disks" which are all on the same physical pool which results in massive contention (and bad performance). In addition we add significant parity overhead since Minio and ZFS will both manage parity. Noting that by default Minio will be functionally a RAID1 in terms of the amount of parity data created. In this case, the S3 protocol cannot be used to attack the backups, but a direct attack on the ZFS server could be used to destroy the underlying pool.

Scenario 3 : Customisation of the Minio plugin to use multiple mountpoints on multiple ZFS filesystems with at least 4 mapped directories, erasure coding and object lock can be enabled. Not a great idea as we still have the double parity overhead, but at least we've solved the parallelisation problem. Assuming multiple RAID1 pools, we can distribute across at least 6 pools which allows us to modify the default parity setting in Minio to something closer to RAID6.

Scenario 4 : Same as Scenario 3, but with single disk pools. Don't. ZFS failure modes on single disk pools are catastrophic.

Scenario 5 : Assuming you have the budget, buy a separate server with a bunch of disks and install Minio directly and don't use ZFS at all. In this case you'll want to really lock down the network access connections and possibly disable SSH and/or get MFA on SSH connections.

Like everything it's all a matter of trade-offs and risk profiling. Personally, I'm hoping that the new versions of Minio that enable object locking on single disk setups works as advertised as then I get the benefit of ZFS's proven data protection, auto-healing and so on. That said, if you have a lot of data to store on Minio, setting up dedicated servers would be the best solution from a performance standpoint as Minio is more efficient about spreading IO across available disks, and logically it should be equally robust *in this specific context* as ZFS

Oh, and yes, if you modify the _MINIO_STORAGE_CLASS_STANDARD_ to EC:2 or EC:3 after installation but before writing data it will be taken into account. I think that it should even work after data has been written, as the internal catalog will know which parity scheme was used and new writes will conform to the current settings (untested)

Hope this helps
 

xness

Dabbler
Joined
Jun 3, 2022
Messages
30
Thanks for the detailed feedback and insight – really valuable!

So basically to use MinIO with ZFS the only logical way is to use the 2022-06 release supporting erasure coding on single-node single-drive deployments.

Otherwise use MinIO with Erasure Coding the way it's meant to be over multiple spread out single or RAID0 disks and let it handle parity.

edit: I created a suggestion to upgrade the MinIO plugin in iXsystems' Jira
 
Last edited:

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
Just to clarify on the first point, single-node, single-drive where the single drive is a ZFS filesystem, yes.

Just to test I dug out the VM I used for compiling Minio the last time and recompiled the latest version, pushed it into a jail and it works. On a single disk path, it allows me to create a bucket with object lock and versioning enabled.

If you want to give it a test drive (please not production), here's a link to the binary for TrueNAS Core : https://www.dropbox.com/s/3ky1v9uu192snpj/minio?dl=0

You should be able to install a fresh MinIO Plugin and then copy this file to the following path (from the TrueNas environment):

/mnt/<plugin pool>/iocage/jails/<jail name>/root/usr/local/bin/

I don't think that the executable bit will survive Dropbox so you'll probably need to chmod a+x on this file before it will run. But you'll be able to get an advance preview for the day that the plugin binary gets updated.

Keep me posted on how it goes!
 

xness

Dabbler
Joined
Jun 3, 2022
Messages
30
The temptation is too big not to use it in prod aka to not compile an executable hehe – this feature just seems predestined to work with ZFS

How'd you do that? Can't seem to find any build scripts of MinIO for FreeBSD online;– not too familiar with Go's cross-platform dependencies. Just found the cross-compile.sh – let's see if I can somehow compile it successfully from FreeBSD
 
Last edited:

xness

Dabbler
Joined
Jun 3, 2022
Messages
30
@Kris Moore just replied to my Jira suggestion and he suggests switching from CORE to SCALE, as CORE may or may not see upgrades to MinIO in the future
 

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
Heh - corporate secret. Kidding :cool:

I installed a VM with FreeBSD 13, then installed Go with a "pkg install". From there, I just followed the Minio compilation instructions on Github and it spits out a binary. The only thing not accounted for in their instructions are that under BSD you can't feed the env variable the same way so you need to manually do:

Code:
set GO111MODULE=on


before running

Code:
go get github.com/minio/minio
go install github.com/minio/minio


And the resulting binary ends up in go/bin/

In any case I've pushed it into my lab and will start throwing some Veeam traffic at it and see how it behaves.
 

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
@Kris Moore just replied to my Jira suggestion and he suggests switching from CORE to SCALE, as CORE may or may not see upgrades to MinIO in the future

Just to clarify, yes if you are a heavy Minio user, I'm going to strongly encourage a migration to SCALE be something you plan for down the road. The plugin system on CORE is for all intents and purposes deprecated and in pretty rough shape right now,. That plus some other lingering issues would make it a good switch for any heavy Object Storage users on TrueNAS.
 

infraerik

Dabbler
Joined
Oct 12, 2017
Messages
24
Completely agree - even if it's marginally more work, you can easily use the native Minio Docker image on TrueNAS SCALE if you want faster updates than the plugin.
 
Top