S3 and ZFS and SLOG and tunables vs Veeam performance

[stefanoc]

Hi All,

I'm in the process of setting up a new system as a backup storage for Veeam.
Supermicro SSG-640P-E1CR36L, CPU Xeon Gold 6326 CPU 2.9GHz 16C, 256 GB RAM ECC, 4x 10GE NICs

OS disks 2x SSD (mirrored by TrueNAS setup)
DATA disks 22x HDD 20 TB (each vDev by 6 disks RAIDz2, plus hotspares)
CACHE 2x Intel SSD/P5520 3.84TB U.2

I'm looking for suggestions to setup the ZFS filesystem to reach the maximum performance writing backup data through S3 (Minio).
Veeam Backup and Replication V12 is set to write 1MB data chunks as default, but can be tuned.

an then...

1. It is safe to use a single P5520 as SLOG?
2. May I use a TrueNAS to manage the mirroring of the P5520s to provide redundancy, or better use VROC?
3. Do I need to change the tunable vfs.zfs.dirty_data_max_max ?
4. There are other tunables to consirer?
5. It's better to set the volume Sync policy to Disabled?
6. Do I need to change the Record Size to fit Veeam settings? And S3 is going to change the data block size?
7. Better to turn off LZ4?

Thank you.
Regards.
Stefano

 

[jgreco]

1. It is safe to use a single P5520 as SLOG?

Yes. Ask yourself, what's the worst that could happen? The P5520 dies and degrades to using the in-pool ZIL.

TrueNAS to manage the mirroring of the P5520s to provide redundancy, or better use VROC?

TrueNAS does not support VROC. Or really any other RAID either.

It's better to set the volume Sync policy to Disabled?

Then why do you need a SLOG? If you're going to disable sync (and I'm unclear on why this application would require sync), why not just disable sync and omit the SLOG?

Better to turn off LZ4?

Almost never. LZ4 is fast and cheap.

 

[blanchet]

For Veeam, the best setup is the hardened XFS repository.

Hardened Repository - User Guide for VMware vSphere

To protect your backup files from loss as a result of malware activity or unplanned actions, you can add to your backup infrastructure a hardened repository based on a Linux server. The hardened repository...

helpcenter.veeam.com

You can check this fact by browsing the Veeam forums.

So you should run a tiny bhyve VM with Debian 11 on your TrueNAS Core server, then configure it to host the Veeam Backup.

In my opinion, you should use S3 only when you host your backup on the cloud.

 

[stefanoc]

Hi Blanchet,

Thank you for your reply.
I'm setting up a new "cloud" system to host off-site scaleout/2nd backup copies. Our legacy systems are based on ReFS.
Anyway, all Veeam products are moving to S3 for primary storage, Veeam Backup and Replication, Veeam 365, Cloud Connect, etc.
I'm not talking only about docs and tech-notes, but Veeam representatives and tech support are pushing a lot S3 as backup destination.
This is the new way of storing data for the vendor...
I would like to rely on ZFS, avoiding Minio bare metal install and Erasure Coding management complexity.
I'm testing today SCALE and Minio (Official Charts), but certificate handling in not included in the app (container) setup wizard...
Regards.
Stefano

 

[NickF]

I get that VEEAM is going in this direction. But it's certainly not a requirement. Running object storage on top of ZFS in a container is going to yield reduced performance no matter how you shake it. ZFS through compression can already get alot of the benefits of erasure coded storage, and can also become immutable through read-only snapshots.

To me this is all just a different way to do the same things we're already doing in ZFS.

 

[stefanoc]

After some testing, we are reaching the limit of Veeam proxy CPUs before the barrier of TrueNAS SMB and S3 data ingest capacity.
Writing data at peak of 780 MByte/s.
Changing record size from 128 to 512 does not affect the write speed with this setup, guess due to proxy push limit.