My FreeNas Production Build - Sanity Check

[wreedps]

Hello,

I have been reading here for months and I am finally collecting hardware to build my permanent production server for the house. I have everything except all the memory and the 10gbe NICs. I am posting this for everyone to check my thoughts and provide feedback.

This server will be used as a Iscsi target for my 3 ESXi 6 hosts.

Chassis: SuperMicro 24-bay with SAS2 backplane and redundant PSUs
Mobo: X8DTN (i believe)
CPU: 2x Xeon X5650
Memory: 192GB Crucial 1333 ECC Reg
NIC: 2x Chelsio T420
HBA: LSI 9207-8i
SLOG: 2x Intel DC S3510 80GB
L2ARC: 2x Sandisk Extreme Pro 256GB
Pool: 10x 900GB Seagate Enterprise 10K SAS 6G Disks ( Need input on how to configure RaidZ pool)

Networking will all be connected to my homes core Nexus 5020 with LC fiber patch cables.

 

[Fuganater]

Holy RAM Batman!

 

[HoneyBadger]

S3510s are not ideal SLOG devices as they're only rated at 0.3 DWPD write endurance. The S3610 is significantly more durable (3 DWPD) but will cost you more as they only start at the 200GB size. S3700s (10 DWPD) I imagine might be too much money.

Regarding the pool, the general rule is "don't use RAIDZ for VMFS/virtualization" due to the poor performance of parity RAID systems in random write. But for a homelab scenario you may be able to get acceptable performance, especially given that you're using 10K SAS drives.

Mirrors gets you 5 vdevs and 4.5TB, following the "50% for iSCSI" thumbrule that's ~2.75TB before performance starts to drop due to pool fragmentation.

Single vdev RAIDZ2 (8+2) yields 7.2TB, ~3.6TB with "50% for iSCSI" and performance will probably hit a harder wall as fragmentation becomes an issue, because random writes on a RAIDZ pool suck.

If you're building anew though, you can always do an A:B test.

 

[wreedps]

I keep hearing more RAM is better! My ESX hosts have like 200GB. I can get memory really easy, so why not?

 

[wreedps]

S3510s are not ideal SLOG devices as they're only rated at 0.3 DWPD write endurance. The S3610 is significantly more durable (3 DWPD) but will cost you more as they only start at the 200GB size. S3700s (10 DWPD) I imagine might be too much money.

Regarding the pool, the general rule is "don't use RAIDZ for VMFS/virtualization" due to the poor performance of parity RAID systems in random write. But for a homelab scenario you may be able to get acceptable performance, especially given that you're using 10K SAS drives.

Mirrors gets you 5 vdevs and 4.5TB, following the "50% for iSCSI" thumbrule that's ~2.75TB before performance starts to drop due to pool fragmentation.

Single vdev RAIDZ2 (8+2) yields 7.2TB, ~3.6TB with "50% for iSCSI" and performance will probably hit a harder wall as fragmentation becomes an issue, because random writes on a RAIDZ pool suck.

If you're building anew though, you can always do an A:B test.

I will look into the S3610s. The s3700s are too expensive at the moment.

Any other thoughts/ideas keep em coming.

 

[HoneyBadger]

Couple more thoughts;

Make sure you enable sync=always for the iSCSI datastores or the SLOGs won't ever be used.

Probably not an issue as you're specifying iSCSI for VMFS with sync writes; but you have dual 10Gbps networking and a ton of RAM in play, so watch out for oversized transaction groups causing write stalls if you do heavy async I/O (eg: throwing some CIFS/OS-level NFS on it)

 

[wreedps]

Couple more thoughts;

Make sure you enable sync=always for the iSCSI datastores or the SLOGs won't ever be used.

Probably not an issue as you're specifying iSCSI for VMFS with sync writes; but you have dual 10Gbps networking and a ton of RAM in play, so watch out for oversized transaction groups causing write stalls if you do heavy async I/O (eg: throwing some CIFS/OS-level NFS on it)

To the first line: I do not have that setting enabled now or my test Freenas, could I enable it to try? Where exactly do you set this?

I will not be doing any CIFS or NFS. Does that help?

 

[HoneyBadger]

To the first line: I do not have that setting enabled now or my test Freenas, could I enable it to try? Where exactly do you set this?

From a shell prompt you need to enter

Code:

zfs set sync=always NAME

where NAME is your pool/dataset/zvol that you want to set sync on. WARNING: Performance will nosedive hard if you do not have an SLOG device in place to support the workload.

I will not be doing any CIFS or NFS. Does that help?

If you won't be doing any async I/O against this system then your SLOG will bottleneck writes down to a level where your drives can manage it, so it's a non-issue.

 

[wreedps]

Ok thanks. I guess everything else looks ok?

 

[wreedps]

Are there any alternatives to the Intel DC drives I should look at?

 

[wreedps]

Can you explain to me Async I/O in laymans terms?

 

[HoneyBadger]

Ok thanks. I guess everything else looks ok?

Other than your power bill. ;)

Are there any alternatives to the Intel DC drives I should look at?

Intel is the one I and others have the most confidence in, but anything with power-loss protection is a potential drive. I know that TrueNAS is using some enterprise Sandisk gear, and Samsung makes some PM-series enterprise drives. NVMe is also an option but if you think the DC S3700's are expensive, these are probably out of the question.

Can you explain to me Async I/O in laymans terms?

Async I/O means that ZFS will return a "write completed" back to the client as soon as the data hits RAM. Sync on the other hand won't send that until the data has landed on "stable storage" - be that the pool itself, or the SLOG device. Async is obviously faster, but runs the huge risk of losing data on power loss. As a side effect, certain combinations of network speed, RAM size, and pool write speed can cause very "bursty" performance. I think @jgreco already wrote a post on this, but maybe I can make a pretty one with graphs.

 

[wreedps]

Thank you Honeybadger. I have a good relationship with Sandisk. We buy hundreds of thousands of dollars a year of product. I will check with them on their DC drives.

Thanks again for answering my questions. I will post pictures of the setup when done. I am waiting for more MM fiber SFPs for the Nexus now.

 

[HoneyBadger]

I'm not sure if the Lightning Gen2 still has an SLC option, but that would be probably your best bet for a SAS SSD.

Until FreeBSD gets a driver for the "ULLtraDIMM" that is.

I believe the FreeNAS Mini's SLOG device is a Sandisk X110, which is definitely more "prosumer" but that might work as well. Haven't used one of them so I can't speak to their numbers, but that sounds like something I should buy to test out.

 

[wreedps]

Wow those X110 are 49.00 for a 64GB. Would 64GB be enough for my SLOG?

 

[HoneyBadger]

Wow those X110 are 49.00 for a 64GB. Would 64GB be enough for my SLOG?

Way more than enough. You'd have to ask someone with a FreeNAS Mini how well they perform, but general benchmark numbers seem to indicate they should perform fairly well. The X110 is the OEM name for the "Ultra Plus" and all three sizes got benchmarked here:

http://www.tomshardware.com/reviews/ultra-plus-ssd-nand,3502-5.html

Note the "4KB random write" graph at the bottom, look at the far left under "Queue depth 1" - the drive is still turning in over 20K IOPS.

Edit: I'm going to have to get my hands on some of these. They look rather promising as a budget SLOG.

 

[wreedps]

Well I already have a Sandisk Ultra Plus 128gb. I am going to get another one and I should be all set for my SLOG then!

 

[wreedps]

Nevermind it is a SSD Plus

 

[wreedps]

Do the Sandisk X110/UltraPlus offer the same type of data protection as the Intel DCs?

 

[HoneyBadger]

Do the Sandisk X110/UltraPlus offer the same type of data protection as the Intel DCs?

They advertise that they use a "non-volatile write cache" which they brand as "nCache" but it seems like you may have to use an ATA command to disable any potential for it to cache data in its onboard DRAM. Page 13 of this PDF mentions it as well as the commands:

http://www.sandisk.com/Assets/docs/Unexpected_Power_Loss_Protection_Final.pdf

It looks like the Sandisk U-series disks do have capacitors onboard. Not sure how the performance of the X-series would be affected by the volatile cache getting disabled but it's probably significant.

Again, "further testing required."