SOLVED FreeNAS NFS VM Datastore and Media Store

[coast1010]

Hey all,

I haven't purchased anything yet. My purchasing window will probably not be til June or July

So my home lab has grown a lot recently and my storage layer is an utter nightmare. I'm finally in a financial position to purchase something and I've been looking into a custom FreeNAS solution.

Use Cases/Requirements:

• I have ~20 VMs running from proxmox as the compute resource (~200 GB)
  • These are currently living as local storage. Yeah, bad idea, I know.
• Various lab tests: ELK Stack, etc would be fun to try out
• I have 1 PostgreSQL DB (10 GB)
• I'm currently serving Plex, but the hardware can't keep up with 2 1080p streams (~450 GB)
• Permanent/Safe storage for family photos/videos (400 GB)
• Time Machine backups for 2 Macbooks (1.5 TB total)
• rsnapshot backups for 4 Linux machines (500 GB total)

Limitations:

• 1 GBe
• Space: I don't have a rack and I want a "Tower" style chassis

Hardware:

Motherboard: Supermicro X11SSH-F: Amazon
CPU: Intel Xeon E3-1230 v6: Newegg
RAM: Crucial 32GB DDR4-2666 ECC (x2 16GB DIMMs): Newegg
OS Drives: SanDisk SSD PLUS 120GB Internal SSD (x2): Amazon
Chassis: Fractal Design Define R6: Amazon
PSU: Corsair TXM750M (750 Watt): Amazon
CPU Cooler: Noctua NH-U9S: Amazon

Which leaves us with the data drives. If I wasn't running VMs and a DB with a lot of sync writes, I think this would suffice and we'd be done:
Data Drives: Western Digital Red 4TB NAS HDD (x6): Amazon


Here's where the theories and options start. My background is in System Administration of RHEL with some DevOps experience. My knowledge of ZFS and FreeBSD is very limited, as well as just general Storage Performance and Data Integrity, but I'd love to learn more. Please correct me if I'm wrong.

If I understand correctly, the ZIL is specifically for sync writes. The ZIL is there so that in the event of a sudden power outage, ZFS can decide whether to commit or backout of a sync write that was in progress. With my current configuration, it's my understanding that my VM and DB performance would suffer because the ZIL would be on the same spinning rust as the data.


There's probably multiple ways to skin this cat.

(Option 1): The seemingly obvious choice for the FreeNAS Community for better IOPS and sync writes is a dedicated SLOG Device:
SLOG Device: Intel DC P3700 Series 400GB SSD: Ebay

• Is there a cheaper or more appropriate SLOG device? The options here appear very limited for Power Loss Protection

(Option 2): I could definitely be wrong here, but would it be possible to just have 2 ZPools?
(ZPool: Capacity & Stability): WD Red 4TB NAS HDD (x4 or x6) (Amazon link is above)
(ZPool: IOPS & Performance): Samsung 860 EVO 500 GB SSD (x4): Amazon

• I have a hard time imagining that 2 striped SSD mirrors would have slow performance for sync writes, but I don't fully understand or know ZFS so I could be wrong here
• I would leave the larger ZPool for backups, Plex, family photos, etc, and the smaller one would be only for VMs or things that need sync write performance

Is there an Option 3 or 4 that I haven't considered? Is Option 2 just a terrible idea? The reason why I'm hesistant on Option 1 is because I don't like the idea of buying a used part off of Ebay for what's supposed to be a super stable storage platform.



Other questions I have:

• Have I gone overboard or undershot the motherboard, CPU, or RAM?
• Is there any concerns with the overall project?

Thanks for reading

 

[PhiloEpisteme]

Thanks for the super detailed requirements list. :)

Motherboard: Supermicro X11SSH-F

Do you plan to use the M.2 support on that board? Perhaps for a ZIL? If not, perhaps opt for a board without M.2 to get those PCIe lanes back.

CPU: Intel Xeon E3-1230 v6

That'll likely handle your plex transcoding. Depending on the load of your other jails etc you may want something a little beefier. That is up to you though.

RAM: Crucial 32GB DDR4-2666 ECC (x2 16GB DIMMs)

Definitely a good idea to get 16GB DIMMs so you can hit 64GB later if you need to. Be sure to double-check Crucial's compatibility list for your board.

Chassis: Fractal Design Define R6

I own it and like it a lot. If you run a lot of drives in it and want to be sure it stays cool make sure to load up on fans. I've got 3 on top, 3 in front, 2 on bottom, and 1 out the back.

PSU: Corsair TXM750M (750 Watt)

Definitely enough. More than enough.

Western Digital Red 4TB NAS HDD (x6)

A solid pretty standard choice. With that Define R6, a few Fractal Design HDD trays, and 1 HDD/SSD bracket you can put 12 HDD, and 4 SSDs. If you use SATA DOMs as boot drives and a PCIe drive for your ZIL that'd give you 16 drives of storage space. Silly? Maybe. Cool that the case will handle it just fine? Definitely.

If I understand correctly, the ZIL is specifically for sync writes. The ZIL is there so that in the event of a sudden power outage, ZFS can decide whether to commit or backout of a sync write that was in progress. With my current configuration, it's my understanding that my VM and DB performance would suffer because the ZIL would be on the same spinning rust as the data.

I believe this is basically correct. There is an excellent link some place about the benefits of a ZIL on sync vs async writes. You COULD run without a ZIL to start and add a ZIL later if performance is an issue. As you correctly note, you'll want one with Power Loss Protection.

I have a hard time imagining that 2 striped SSD mirrors would have slow performance for sync writes, but I don't fully understand or know ZFS so I could be wrong here

You would expect better performance with an array made of SSDs. Though, bang for your buck, if performance is an issue, you could stick with spinning disks with a dedicated ZIL device.

• Have I gone overboard or undershot the motherboard, CPU, or RAM?
• Is there any concerns with the overall project?

It looks like you've got the basics down pretty good. I would suggest you consider whether you want that M.2 slot or not and if not perhaps opt for a board like the X11SSM-F which gives you those PCIe lanes back in the form of a PCIe slot. I don't think you are way under or over on the board etc. I _think_ if I were you I'd go with 6 drives in 1 RAIDZ2 vdev and opt to not use a dedicated ZIL device to start. And then if you find you're having issues you can consider how you want to address that such as adding a dedicated ZIL device or adding an SSD pool. It depends on your budget, of course.

 

[coast1010]

That'll likely handle your plex transcoding. Depending on the load of your other jails etc you may want something a little beefier. That is up to you though.

Any suggestions for a beefier CPU? Another option I've considered was the Intel Xeon E3 1275 v6

I own it and like it a lot. If you run a lot of drives in it and want to be sure it stays cool make sure to load up on fans. I've got 3 on top, 3 in front, 2 on bottom, and 1 out the back.

Luckily, last time I upgraded my gaming PC, I ordered 16 case fans instead of 2 because I didn't realize they came in boxes of 8, so I'm set here.

A solid pretty standard choice. With that Define R6, a few Fractal Design HDD trays, and 1 HDD/SSD bracket you can put 12 HDD, and 4 SSDs. If you use SATA DOMs as boot drives and a PCIe drive for your ZIL that'd give you 16 drives of storage space. Silly? Maybe. Cool that the case will handle it just fine? Definitely.

2 Things:

1. I didn't realize SATA DOMs were a thing. I'll definitely consider these.
2. I didn't think about the trays. Supposedly the PETG filament I have for my 3d Printer has a deflection temperature of 64-70 Celsius, but I'm not sure I'd want to risk running plastic in this environment... something about plastic potentially melting onto a live motherboard just sounds like a bad time.

You would expect better performance with an array made of SSDs. Though, bang for your buck, if performance is an issue, you could stick with spinning disks with a dedicated ZIL device.

Yeah, I think I'm leaning towards spinning disks myself, at least to start.

It looks like you've got the basics down pretty good. I would suggest you consider whether you want that M.2 slot or not and if not perhaps opt for a board like the X11SSM-F which gives you those PCIe lanes back in the form of a PCIe slot. I don't think you are way under or over on the board etc. I _think_ if I were you I'd go with 6 drives in 1 RAIDZ2 vdev and opt to not use a dedicated ZIL device to start. And then if you find you're having issues you can consider how you want to address that such as adding a dedicated ZIL device or adding an SSD pool. It depends on your budget, of course.

Okay. I think I'll more or less order the top section of my rant above, but maybe with a beefier CPU and SATA DOMs. I'll try a ZIL device if/when I need better disk performance from the VMs/DB.

Last question that I forgot to include! The Intel C236 Chip on that MoBo... do I need to disable RAID? Is that even a possibility? I'm having flashbacks to when a customer at work was buying some entry-level Dell for their big data stack and forgot to include that "HBA Passthrough" was a requirement and a bunch of RAID 0's do not qualify for JBOD (and I imagine it's a bad idea in ZFS as well).

Cheers

 

[Chris Moore]

There is an excellent link some place about the benefits of a ZIL on sync vs async writes. You COULD run without a ZIL to start and add a ZIL later if performance is an issue.

Is this the one you are talking about?

The ZFS ZIL and SLOG Demystified
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

also

Some insights into SLOG/ZIL with ZFS on FreeNAS
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

You would expect better performance with an array made of SSDs. Though, bang for your buck, if performance is an issue, you could stick with spinning disks with a dedicated ZIL device.

There has been some testing done to find definitive answers:

Testing the benefits of SLOG using a RAM disk!
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

SLOG benchmarking and finding the best SLOG
https://forums.freenas.org/index.php?threads/slog-benchmarking-and-finding-the-best-slog.63521/

The Intel C236 Chip on that MoBo... do I need to disable RAID?

We don't use hardware RAID with ZFS. It can create problems that result in data loss.

a bunch of RAID 0's do not qualify for JBOD (and I imagine it's a bad idea in ZFS as well).

Just like that. If it is like others I have worked with, it should have three modes, something like 'legacy', AHCI and RAID. If you set it in AHCI mode, it should work fine with FreeNAS.

 

[PhiloEpisteme]

@Chris Moore Yup, those are exactly the links I was referring to. Wow, I really gotta organize FreeNAS bookmarks!

Intel Xeon E3 1275 v6

That CPU has integrated graphics which FreeNAS doesn't need. I did a quick look at the new price for that one vs the 1270 and 1280 of the same generation. Looks like the 1275 is ~360, ~350 for the 1270, and the 1280 is ~380 on various sites. I imagine those CPUs will be more than beefy enough.

didn't realize SATA DOMs were a thing. I'll definitely consider these.

They are pretty cool and nice and small but also more expensive. A small SSD would be more than sufficient if the SATA DOM breaks the bank and if you're not worried about space and cabling for an SSD.

 

[Chris Moore]

Wow, I really gotta organize FreeNAS bookmarks!

These are some I have collected. If you have some good ones to suggest, just let me know:

https://www.ixsystems.com/community/resources/links-to-useful-threads.108/

 

[coast1010]

Thanks for the replies and the handy resource page. After reviewing the various links, I'll definitely be on the lookout for an Intel P3700 for the VMs performance. It seems to strike a good balance of performance, price (second-hand), etc

Cheers