BUILD: ZFS, ZLOG, ZCACHE, NVMe

[KpuCko]

Hello,
I want to build new FreeNAS storage. The box have to be multipropose box. It will provide SAMBA, NFS, iSCSI as main services, and KVM virtualization to separate additional services from the main box (I know what, FreeNAS provide KVM support). The KVM services will be related to DC, DNS, DHCP, NTP and so on.

So I want to use some benefits like KVM over IP, (I'm fan of Intel AMT, Supermicro), Wake on LAN.. all about remote management.
Also the NVMe disks will be used to store the main operating systems and act as a cache of the RAIDZ volume.

I haven't deep understanding in hardware related topics, but for this type of setup I think to use some vPRO cpu by Intel, because of AMT, much RAM - because of ZFS cache, and good cooling - because of less noise ;>

For now I only choose the case, here is it: Fractal Design - Node 804. At the beginning I will start with small 4x4TB HDD on 7.200RPMs and two NVMe disks for OS.
I think 64 GB is enough for the moment, but please keep in mind to be able to extend it if it's needed. I know that ZFS is memory hungry, so please suggest me mainboard with enough slots and much supported RAM limit.

Thanks a lot!

 

[jro]

So will you have FreeNAS virtualized in KVM? You can make that work, you'll want to set up passthrough for your HBAs so ZFS can interact with them directly.

When you say "cache of the RAIDZ volume", do you mean L2ARC or SLOG? If you're going to use one of the NVMe disks as a SLOG, make sure it's power safe, otherwise it defeats the purpose of adding a SLOG.

64GB should be plenty for ZFS to do its thing, especially with only 16TB raw. You can probably get away with 32GB RAM if you want.

 

[Chris Moore]

You appear to have a massive misunderstanding of how FreeNAS works. Please spend time learning about FreeNAS because you may need to use something other than FreeNAS to be able to do all the things you want.

 

[KpuCko]

@jro yeah, I mean SLOG. Everything is fine, I already have UPS. My UPS is supported by NUT, so I will connect the UPS to the FreeNAS and the UPS will command the NAS if something goes wrong with the power source.

@chris, thanks for the useful links, but is is so generic, please point me to some direction... I'm really aware of the things which FreeNAS can do for me (I mean the services which can provide) so I think I can do all of it with simple installation of FreeNAS.

I'm just asking you about hardware related advice.
Thanks in advance. Appreciated.

 

[HoneyBadger]

I would recommend you start with the excellent blogpost series from iXsystems here:

A Complete Guide to FreeNAS Hardware Design, Part I: Purpose and Best Practices
by Josh Paetzel; iXsystems Director of IT; Feb 3, 2015
http://www.freenas.org/blog/a-compl...are-design-part-i-purpose-and-best-practices/

A Complete Guide to FreeNAS Hardware Design, Part II: Hardware Specifics
by Josh Paetzel; iXsystems Director of IT; Feb 5, 2015
http://www.freenas.org/blog/a-complete-guide-to-freenas-hardware-design-part-ii-hardware-specifics/

A Complete Guide to FreeNAS Hardware Design, Part III: Pools, Performance, and Cache
by Josh Paetzel; iXsystems Director of IT; Feb 10, 2015
http://www.freenas.org/blog/a-compl...-design-part-iii-pools-performance-and-cache/

A Complete Guide to FreeNAS Hardware Design, Part IV: Network Notes & Conclusion
by Josh Paetzel; iXsystems Director of IT; Feb 12, 2015
http://www.freenas.org/blog/a-compl...ware-design-part-iv-network-notes-conclusion/

There is also the hardware recommendations guide; while a bit dated it still has a lot of useful information.

https://www.ixsystems.com/community/resources/hardware-recommendations-guide.12/

As well as the resource regarding iSCSI/block storage requirements:

https://www.ixsystems.com/community...quires-more-resources-for-the-same-result.41/

Regarding your intended plans; FreeNAS does not have KVM - it uses a different hypervisor called bhyve. I would recommend against using FreeNAS as your hypervisor host though; my experience has shown that it works at its best when it is left to the job it does well - being a SAN/NAS appliance.

Also the NVMe disks will be used to store the main operating systems and act as a cache of the RAIDZ volume.

FreeNAS cannot do this - once a disk is claimed for the boot volume, it is unavailable for other uses.

As pointed out by @jro when choosing devices for an SLOG, you should also validate that the devices have power loss protection - this is a requirement over and above a UPS.

 

[jro]

@jro yeah, I mean SLOG. Everything is fine, I already have UPS. My UPS is supported by NUT, so I will connect the UPS to the FreeNAS and the UPS will command the NAS if something goes wrong with the power source.

It's good that you have a UPS, but the point of the ZIL (and by extension, the SLOG) is to store in-flight sync writes on a power-stable medium until they're flushed from RAM into their permanent home on the pool. In the event of a crash (or if there's a hang during shutdown and your UPS dies), the system will be able to recover the sync writes from the SLOG after it reboots. If they SLOG isn't power stable, it totally defeats the purpose of using a SLOG in the first place; the writes will be lost and the SLOG will be empty when the system boots. You might as well set sync=disabled on the pool.

 

[IQless]

Everything is fine, I already have UPS. My UPS is supported by NUT, so I will connect the UPS to the FreeNAS and the UPS will command the NAS if something goes wrong with the power source.

Like @jro states, it's good that you have a UPS. Something to consider though, what if your power supply suddenly dies, or have a hiccup? Then the UPS won't help much. This is a much less likely scenario than a sudden power failure in the grid, but in any case, worth remembering :)