Planning new FreeNAS server

[NorCalTechSupport]

Hi all, I am planning my next big FreeNAS. I have an almost brand new SuperMicro Chassis with dual CPUs & 32GB RAM. The Chassis hos 36 6TB HDDs & 2 1TB SSDs for boot.

This new server will have just one job: Storage vault for Acronis Backup 12.5.

Some of what I backup are Exchange databases. Currently my two exchange databases add up to just under 2TB total size. I also backup a slew of production servers with total size per server being less than 500GB. In the next month or two I will be deploying a new Exchange Server and with luck I will be able to read all of the data into one database of apx 2TB.

When my SuperMicro arrived there was a gotcha as I could not create RAIDs so I called SuperMicro and they arraigned for a new RAID card and I was good to go. I believe the first RAID card a SUPER AOC-S3008L-L8E may be what I need for FreeNAS (comments are welcome). On the original RAID card all I could see were 36HDDs with no way to RAID them.

Currently I have:
32GB RAM
36 X 6TB HDDs
2 X 1TB SSDs
2 empty NVME slots

I am wondering if I would kill the machine or make it run like the wind if I added Four 2TB enterprise level SSDS in two RAID1 volumes and used one volume for read & the other for write cache? The reasoning behind my GIANT R/W cache is trying to handle GIANT files easier?

I am planning on using the two SSDs I currently have as a place for my logs.

I am planning on booting from a pair of NVME drives in RAID1 configuration.

And YES this time I will write down all the serial numbers on the drives and what slot they are in the day I build the beast instead of waiting 3.5 years as I did 4 years ago.

Thanks to everyone,

 

[Newfoundland.Republic]

How were you trying to RAID your disks with the original HBA? Was the original HBA flashed to IT mode?

 

[joeinaz]

"On the original RAID card all I could see were 36HDDs with no way to RAID them."

In FreeNAS this is exactly what you are looking for.

In FreeNAS, you need to present disks without any form of "RAID". Once you are able to see all 38 of your disks (including boot disks) at boot and in FreeNAS use the FreeNAS guides to configure your system to provide the "RAID like" storage redundancy you are looking for.

 

[LVLouisCyphre]

I'm new to FreeNAS, but not RAID and Unix. I still cannot believe people who don't RTFM or probably should now be RTFW (read the fabulous wiki). You don't use hardware RAID adapters with FreeNAS which strictly uses ZFS for storage; it's generally counter productive. ZFS does the RAID functionality. ZFS performs best when it has raw access to the drives.

 

[Constantin]

....This new server will have just one job: Storage vault for Acronis Backup 12.5... Some of what I backup are Exchange databases. Currently my two exchange databases add up to just under 2TB total size. I also backup a slew of production servers with total size per server being less than 500GB. In the next month or two I will be deploying a new Exchange Server and with luck I will be able to read all of the data into one database of apx 2TB.

Sounds like you have enough space with 36 drives. The question is how to best split them up to get a balance of speed and backup protection. I'd test actual use cases... As a starting point, a single pool with six Z2 VDEVs consisting of 6 drives each might be worth considering. That should allow for some fairly impressive I/O.

As others have mentioned, do not RAID these drives. Configure the HBA to be in IT mode and address each drive individually. I'd use the SSDs as a boot pool (both drives). IF possible, I'd consider partitioning them for boot vs. log use. You don't need much as a boot pool, I use 64GB per SATADOM.

I'm not sure if a SLOG helps in your case, but if you go that route, be sure to get a well-suited unit, or you may slow your system down rather than speed it up and/or put your data at risk. Among SSDs there is a wide variety of capabilities and good drives to use as a cache should be researched carefully, Serve the Home has a good summary (and many related articles) and the SLOG discussion here should be considered also. For a business-centric system like yours I'd use a mirrored pool of SLOGs.

I would not waste the NVME slots for boot pools. Use them for fast SLOGs or like devices that really take advantage of the PCIe speeds. For example, my P4801x runs at PCIe 3.0x4 speed, much faster than my single-VDEV pool.

Whether or not L2ARC makes any difference in your use case is a good question. I found L2ARC set to metadata only mode sped up my rsync sessions spectacularly.

 

[NorCalTechSupport]

How were you trying to RAID your disks with the original HBA? Was the original HBA flashed to IT mode?

No the original HBA came that way and at the time I was trying to get Four RAID6 volumes that HBA would not work so SuperMicro quickly sent me a HBA that allowed for RAID6. The good news is I have the original HBA safely on a shelf.

I love the SuperMicro server and it's speed (450MB/s to 600MB/s between volumes) but large files and Acronis 12.5 is chewing the storage volumes up and fragmenting them so bad they become unusable in just a week or two.

 

[Newfoundland.Republic]

Okay. Do NOT use the the hardware RAID. ZFS handles all the "raid-ing" and needs direct access to the disks (e.g. SMART, etc.). There are many resources on the best way to carve up your disks (e.g., volume groups into RAIDZ). Others have better experience in this area than I do.

There are likely better ways of using the 1TB SSDs (SLOG, etc.). You can pick up a couple of Kingston A400 120GB SSDs and mirror them for boot (about $25/each).

If you are using this for backups, this is not a place to cut corners or go against recommended practices.

 

[joeinaz]

"The good news is I have the original HBA safely on a shelf."

Can you reinstall the original HBA and run FreeNAS?