Supermicro 36 HDD setup for Veeam and S3

[stefanoc]

Hi all,

we are in the process of ordering a new Supermicro device to host Veeam backup data.
Since we are not familiar with ZFS with this post post we are looking for an advise.
Today we user ReFS on RAID 60 on an HPE Apollo 4200 system that reached the capacity limit.

Our ideal setup is TrueNAS SCALE (for future scale-out needs) hosting Veeam backups written via S3.
Most of the backup jobs are set to run forever forward incremental, some of them with weekly synthetic full (thanks to FastClone support in ReFS).

The hardware configuration is the following:

Supermicro Storage SuperServer SSG-640P-E1CR36L
1x Intel Xeon Gold 6326 Processor 16 Cores (24M Cache, 2.90/3.50 GHz) 185W
8x 32GB 3200MHz DDR4 ECC Registered DIMM Module
2x 480GB Samsung SSD SM883 Mixed Use Data Center SERIES 2.5IN SATA3 (Operating system RAID1)
2x Intel P5520 3.84TB Drive - 2.5 NVMe U.2 PCIe-4.0 Drive (ZIL cache RAID1)
34x 18TB Enterprise Class SAS3 12Gb/s 7200RPM - 3.5" HDD
2x Intel X520-DA2 - 10GbE Dual-Port SFP+ Server Adaptor

We are not looking for the maximum performance, but for a balanced/safe ZFS layout.

1. Looking at our needs Is TrueNAS SCALE a good choice in terms of stability?
2. 4x 8HDD RAIDZ2 + 2HDD as global hotspare disks is really a balanced config?
3. ZIL cache disk built by 2x Intel P5520 3.84TB Drive NVMe drives in RAID1 with VROC will help us?
4. No L2ARC device is configured, the system will rely on ARC with 256GB of ECC RAM, are we going to have a good reading performance?
5. S3 (without erasure coding) works fine with this setup in terms of performance and reliability?
6. There are some tuning rules we should consider for S3 use?

Thank you for your help.
Regards.
Stefano

 

[sretalla]

1. Looking at our needs Is TrueNAS SCALE a good choice in terms of stability?

You're running a storage project, so, no. CORE is where stability is for storage at the moment.

2. 4x 8HDD RAIDZ2 + 2HDD as global hotspare disks is really a balanced config?

That should be OK... hot-spares may be more than required safety, but not harmful.

3. ZIL cache disk built by 2x Intel P5520 3.84TB Drive NVMe drives in RAID1 with VROC will help us?

I don't think your workload is going to be sync writes (VEEAM does SMB doesn't it?), so not useful at all.

4. No L2ARC device is configured, the system will rely on ARC with 256GB of ECC RAM, are we going to have a good reading performance?

You may find a small L2ARC with metadata only settings would help if the incremental backups require reads of metadata before writes begin.

5. S3 (without erasure coding) works fine with this setup in terms of performance and reliability?

6. There are some tuning rules we should consider for S3 use?

I can't comment on these, but I understand it's well used by others who may comment.

 

[majerus]

Sounds like a business purchase, I would do one of a few things. Reach out to IX about a supported system they are very reasonable on the cost. I would then also avoid scale atm , run core if its business IMO

 

[stefanoc]

Thank you for the reply.

S3 is the new Veeam trend, the data is divided in chunks.
This will change how TrueNAS manages backup data streams, but I don't know how and what is the performance impact.

In the meantime I'll try to contact IX....

 

[majerus]

Good luck, the IX folks are easy to work with and get a hold of. Let us know how it goes

 

[danb35]

I've heard Veeam suggested quite a bit around here as backup software for client PCs, but the 10 GB downloadable ISO--with no other way to download it--kind of turns me off. Should probably look into it a bit more, though.

 

[Ericloewe]

10 GB downloadable ISO

Do they ship pirate ISOs of Microsoft Office 2003, 2007, 2010 and Windows 7 and Windows 95 ESR2 with the Windows 95 Plus Pack alongside their software? Because I can't imagine what they're doing with that huge ISO.

 

[danb35]

Because I can't imagine what they're doing with that huge ISO.

I just downloaded v12, and my mistake--it's only 9.3 GB. I guess the .iso format makes sense for VMs (gee, VM, Veeam...), but I don't need their software for that--that's what Proxmox Backup Server is for. But I have a couple of physical Windows machines that I'd like something more robust than Windows Backup for--and for that application, I'm not sure what to do with that big of a .iso. It's too big for a DVD-R, even a double-layer one. Mounting the .iso and copying its contents onto a small USB SSD resulted in the installed copy of Veeam wanting to back up to that SSD. I haven't yet tried to image it onto a USB stick; maybe that's the way to go. Though a little bit of Googling indicates that Windows can now mount the .iso natively, which I hadn't realized--older versions of Windows didn't have that capability.

But to your question:

Code:

 dan@Dan-MBP-2019  /Volumes/VEEAM BACKUP  du -sh *
7.4M    AIR
611M    Agents
1.4G    Backup
124M    Catalog
 18M    Cloud Portal
 60K    EULA
229M    EnterpriseManager
 85M    Explorers
3.0G    Packages
3.0G    Plugins
513M    Redistr
302M    Setup
392K    Setup.exe
123M    Updates
2.0K    autorun.inf

 

[Ericloewe]

Though a little bit of Googling indicates that Windows can now mount the .iso natively, which I hadn't realized--older versions of Windows didn't have that capability.

Yeah, I think it was introduced in Windows 8.

3.0G Packages

3.0G Plugins

Yeah, two different devs accidentally packaged the same directory and were too scared to get rid of the other one just in case.
Probably not, but let's not let reality ruin a fine story.

 

[Lix]

That iso is for the server install++, you just need to download the agent for a simple Windows install.

 

[blanchet]

Introduction
I am managing of fleet of TrueNAS servers for 5 years to host Veeam backup.
I have tried different setup over the years, so I can give you my advice based on my experience.

Concerning the software
TrueNAS Core is really more mature than TrueNAS Scale for storage application.
If you do not need GlusterFS, stay with TrueNAS CORE.

In my opinion, S3 for Veeam is interesting only when you consume storage on the cloud.
You have to know that TrueNAS Core use Minio in a single disk configuration to implement S3
It is very easy to setup, but it has limitations

• it does not scale-out
• it does not support erasure coding
• it supports neither bucket quota nor user quota. Only a single whole quota for the S3 service.

On the other hand the files are stored as regular files on the ZFS filesystem so you can also manage them with the standard SSH console instead of using awscli.

For Veeam, the best setup is configuring a XFS hardened repository that runs on a Linux VM directly on the TrueNAS server.
This setup has all the benefit of ReFS plus the immutable backups.
Procecure

• You create a bhyve VM with a serial console
  

  SOLVED - Quick guide to install Debian 10 in bhyve with the serial console

  Abstract To have a lighter Linux virtual machine, you can enable the serial console and then remove the VNC device after the operating system installation with VNC. But do you know that you can directly run the Debian 10 installer on the serial console without VNC ? It is faster because you will...

  www.truenas.com
• You create a very big zvol that you add as a big vdisk to the VM
• you format it with XFS
• Then you connect to the Veeam server.
  
  

  Hardened Repository - User Guide for VMware vSphere

  To protect your backup files from loss as a result of malware activity or unplanned actions, you can add to your backup infrastructure a hardened repository based on a Linux server. The hardened repository...

  helpcenter.veeam.com

Concerning the pool layout
4x 8HDD RAIDZ2 + 2HDD is a good layout for a backup server


Concerning the hardware

• Having a ZIL is useless for Veeam: ZIL is used only for synchronous access typically for VMware.
  
  

  What is the ZFS ZIL SLOG and what makes a good one

  We go into what the ZFS ZIL SLOG is. We also show a few common choices as ZFS ZIL SLOG devices, how they work and what makes a good one

  www.servethehome.com
• I think that you do not need L2ARC because you have already 256 GB of RAM
• Which HBA do you plan to use ?

The Supermicro 36-bays server can host a very large amount of data but the server is very heavy.
It means that the maintenance is really boring:

• If you want open the server for a maintenance then you have to pull it from the rack after removing all the disks and all the power supplies to reduce the mass to prevent the rack from tipping over.
• The worse case is when you have to replace the motherboard: you may have to pull/push several time the server because you have forgot to plug something.

It is the reason why, now I prefer having a 2U server and a JBOD

• The 2U server is lighter so you can pull it while keeping the disks and the power supplies inside
• The JBOD has only a SAS expander so there is no reason to open it: you will not have to install new network interfaces, change memory, etc. The SAS expander is more reliable than a server motherboard because it has only few chips.
• a JBOD can host 44 disks in 4U with direct disk front/rear access
• I have no experience yet with top loading JBOD with 60-disks in 4U.

Conclusion
Do not underestimate the required expertise and time to support yourself a large backup server.
For example if you do not run the good firmware version on your HBA, you can have very bad surprise when your server is under a very high load.

So make your life and business easier and call iXsystems to get a quote for example for TrueNAS R20 + an ES24 expansion shelf.

 

[stefanoc]

Introduction
I am managing of fleet of TrueNAS servers for 5 years to host Veeam backup.
I have tried different setup over the years, so I can give you my advice based on my experience.

Concerning the software
TrueNAS Core is really more mature than TrueNAS Scale for storage application.
If you do not need GlusterFS, stay with TrueNAS CORE.

In my opinion, S3 for Veeam is interesting only when you consume storage on the cloud.
You have to know that TrueNAS Core use Minio in a single disk configuration to implement S3
It is very easy to setup, but it has limitations

• it does not scale-out
• it does not support erasure coding
• it supports neither bucket quota nor user quota. Only a single whole quota for the S3 service.

On the other hand the files are stored as regular files on the ZFS filesystem so you can also manage them with the standard SSH console instead of using awscli.

For Veeam, the best setup is configuring a XFS hardened repository that runs on a Linux VM directly on the TrueNAS server.
This setup has all the benefit of ReFS plus the immutable backups.
Procecure

• You create a bhyve VM with a serial console
  

  SOLVED - Quick guide to install Debian 10 in bhyve with the serial console

  Abstract To have a lighter Linux virtual machine, you can enable the serial console and then remove the VNC device after the operating system installation with VNC. But do you know that you can directly run the Debian 10 installer on the serial console without VNC ? It is faster because you will...

  www.truenas.com
• You create a very big zvol that you add as a big vdisk to the VM
• you format it with XFS
• Then you connect to the Veeam server.
  
  

  Hardened Repository - User Guide for VMware vSphere

  To protect your backup files from loss as a result of malware activity or unplanned actions, you can add to your backup infrastructure a hardened repository based on a Linux server. The hardened repository...

  helpcenter.veeam.com

Concerning the pool layout
4x 8HDD RAIDZ2 + 2HDD is a good layout for a backup server


Concerning the hardware

• Having a ZIL is useless for Veeam: ZIL is used only for synchronous access typically for VMware.
  
  

  What is the ZFS ZIL SLOG and what makes a good one

  We go into what the ZFS ZIL SLOG is. We also show a few common choices as ZFS ZIL SLOG devices, how they work and what makes a good one

  www.servethehome.com
• I think that you do not need L2ARC because you have already 256 GB of RAM
• Which HBA do you plan to use ?

The Supermicro 36-bays server can host a very large amount of data but the server is very heavy.
It means that the maintenance is really boring:

• If you want open the server for a maintenance then you have to pull it from the rack after removing all the disks and all the power supplies to reduce the mass to prevent the rack from tipping over.
• The worse case is when you have to replace the motherboard: you may have to pull/push several time the server because you have forgot to plug something.

It is the reason why, now I prefer having a 2U server and a JBOD

• The 2U server is lighter so you can pull it while keeping the disks and the power supplies inside
• The JBOD has only a SAS expander so there is no reason to open it: you will not have to install new network interfaces, change memory, etc. The SAS expander is more reliable than a server motherboard because it has only few chips.
• a JBOD can host 44 disks in 4U with direct disk front/rear access
• I have no experience yet with top loading JBOD with 60-disks in 4U.

Conclusion
Do not underestimate the required expertise and time to support yourself a large backup server.
For example if you do not run the good firmware version on your HBA, you can have very bad surprise when your server is under a very high load.

So make your life and business easier and call iXsystems to get a quote for example for TrueNAS R20 + an ES24 expansion shelf.

Hi blanchet,

we decided to go for XFS using an Ubuntu 22.04 LTS VM on TrueNAS SCALE.
We will attach a big vDisk to it.

What about vDisk expansion on XFS, did you ever encountered issues?

Thank you.
Best regards.
Stefano

 

[blanchet]

Hi blanchet,

we decided to go for XFS using an Ubuntu 22.04 LTS VM on TrueNAS SCALE.
We will attach a big vDisk to it.

What about vDisk expansion on XFS, did you ever encountered issues?

Thank you.
Best regards.
Stefano

I do not use TrueNAS Scale so I cannot tell for this operating system, but on TrueNAS Core 13.0u4, vDisk expansion works well. I did it 3 times without issues.

Procedure

• Increase the zvol size
• Poweroff / poweron the VM to redetect the new vdisk size
• stop veeam and run gparted

Code:

systemctl stop veeam.mount                                                                                                                          
apt install gparted                                                                                                                                 
gparted  

With gparted in graphical mode

• Fix GPT location (click Fix it) after increasing the disk size
• Extend the XFS partition /dev/vdb1
• Reboot the computer at the end

 

[stefanoc]

Thank you Blanchet.

 

[stefanoc]

Hi blanchet,

out test system is up and running.
Have you ever considered to use raw device instead of a vDisk as backup data volume?

Regards.
Stefano

 

[blanchet]

Hi blanchet,

out test system is up and running.
Have you ever considered to use raw device instead of a vDisk as backup data volume?

Regards.
Stefano

On TrueNAS Core, a vm disk can be only a zvol or a file.

I use only zvol. It works well.

Anyway using a raw device is a poor idea, because you will not be protected anymore by ZFS.

 

[stefanoc]

Sorry, I got confused by raw file definition. I thought could use raw volume (ZFS dataset), like VMware does with raw device mapping...
Please discard my last question.