Proposed Build - First time FreeNAS

[nitrobass24]

Hi Guys - Long time lurker, first time poster.

Anyways looking to upgrade my NAS setup and figured I would go with ZFS and FreeNAS. Been using a Synology 1812+ for the past 5 years. Hardware RAID5/6 on Areca Controller and CentOS before that. This time I want to go with ZFS for its increased protection and performance. Been doing lots of reading but I would like to get some input on my build and have a few questions about how to organize my storage.

Proposed Setup:
Case: DS380B
Mobo: ASRock Rack Mini ITX DDR3 1333 Motherboards (C2550D4I) (I know there is a bug, will make sure I have fixed revision)
RAM: Crucial 16GB Kit DDR3L-160 0MT/s (PC3-12800) DR x8 ECC UDIMM Server Memory CT2KIT102472BD 160B
PSU: TBD - Probably Seasonic Modular
zPool1 - Media (TV, Music, Movies, Pictures) - RaidZ2 of 6x
Seagate 8TB IronWolf NAS SATA 6Gb/s NCQ 256MB Cache 3.5-Inch Internal Hard Drive (ST8000VN0022)

Now this is where I get a bit uncertain. I also need an NFS Datastore for ESX and Docker. I already own 3x
SAMSUNG PM853T Data Center Series MZ7GE960HMHP-0 0003 2.5" 960GB SATA III MLC Enterprise Solid State Drive

Best I can tell I have three options:

1. Setup a RAIDz1 or Stripe vDev and separate zPool for the ESX Datastore using all 3 of the SSDs. Replicate -> Media zPool for an online backup.
2. Setup a Mirror vDev and zPool for the ESX Datastore using 2 of the SSDs. Use the remaining SSD as L2ARC for the Media zPool.
3. Use all 3 SSDs as L2ARC for the media zPool and put my ESXi Datastore on the media zPool.

Lastly, I am likely going to benefit from the use of a ZIL due to the NFS requirement. Have read some differing opinions regarding size and if I need to mirror them. Only consistent requirement is PLP and Low Latency. Would appreciate a recommendation.

Thanks!

 

[Ericloewe]

Several things to consider:

• You're going for a lot of storage and a somewhat complex workload. You might want to start with 32GB and you definitely want to take a look at Crucial's 16GB DDR3 UDIMMs. With those, you can use 64GB of RAM on that board.
• L2ARC is generally best avoided under 64GB of RAM (though this may change with compressed ARC)
• Option 3 is a crazy option that has no chance in hell of doing you any good.
• As for SLOG, size itself is mostly irrelevant, you'd only need a few GB. However, such small SSDs are universally slow. Default choice for SLOG is either an Intel 750 or Intel P3700. Also consider not using sync writes. If the VMs aren't critical, this is definitely something to consider.

 

[nitrobass24]

Hi Eric - Thanks for the reply.

• Completely agreed on the RAM, but the 16GB UDIMMs are still prohibitively expensive. Hopefully they will come down in price at some point and I can swap out for more.
• Option 3 - Makes sense and agree. Do you have a strong opinion of a 3x drive RaidZ1 vs 2 Drive Mirror?
• L2ARC - Sounds like I need to do some real-world testing with my workload once I get the system up to see what kind of impacts I would see.
• ZIL - PCIe SSDs w/ PLP are still pretty pricey, might try going back to iSCSI as a compromise first. That way I get ASync writes to ISCSI but still get Sync writes on the ZFS filesystem.

 

[Ericloewe]

Completely agreed on the RAM, but the 16GB UDIMMs are still prohibitively expensive. Hopefully they will come down in price at some point and I can swap out for more.

They were ~220 bucks a day or so ago. It's a bit of a premium over DDR4 UDIMMs, but it's not crazy.

Option 3 - Makes sense and agree. Do you have a strong opinion of a 3x drive RaidZ1 vs 2 Drive Mirror?

It's hard to say. On a faster CPU, I'd expect equal performance, but the C2550 is a bit marginal.

That way I get ASync writes to ISCSI but still get Sync writes on the ZFS filesystem.

Huh? They're one and the same.

 

[nitrobass24]

Fair point Eric. I guess I should have said incremental cost. I picked up 32GB for $175 total. Spending 500% premium doesn't seem worth it to me right now.

Been doing a lot of reading on the internet and I am now considering going to Mirrored vDevs.

Single RAIDz2 6x 8TB = 32GB RAW
3x Mirrors of 2x 8TB = 30GB RAW

Most of this data is archive / cold storage of Media, so write performance on this zPool is not critical. If my understanding is correct resilvering is done at the vDev level and not the zPool level. Resilvering 10TB will be about 1/3 the time if done this way. Seems like a no brainer to sacrifice 2TB of RAW capacity.

Am I missing something?

 

[Ericloewe]

Most of this data is archive / cold storage of Media, so write performance on this zPool is not critical. If my understanding is correct resilvering is done at the vDev level and not the zPool level. Resilvering 10TB will be about 1/3 the time if done this way. Seems like a no brainer to sacrifice 2TB of RAW capacity.

Resilvering times are a somewhat secondary concern (it's going to be plenty fast). More important is the ability to lose any second disk before data loss. With mirrors, the wrong disk fails and you're screwed.

 

[ChriZ]

3 mirrors of 8TB disks is 24TB raw not 30..
Just saying...

 

[Ericloewe]

3 mirrors of 8TB disks is 24TB raw not 30..
Just saying...

Another important point.