FreeNAS hardware configuration and storage questions

[celine]

Hi,

This post covers different topics so excuse me if I shouldn't post here.

I want to build a proxmox infrastructure with the FreeNAS storage unit.
I am a beginner with these FreeNAS and ZFS / RAIDZn subjects.
I need performance and storage for around 40 virtual machines.

For the FreeNAS server, I choice for moment this configuration:

Dell R740 with :

• Dual Intel Xeon Silver 4208 2.1G, 8C/16T, 9.6GT/s, 11M Cache, Turbo, HT (85W) DDR4-2400
• Dual 32GB RDIMM, 2933MT/s => 64G (I have read that the more ZFS has available RAM, the more ZFS performs.)
• BOSS controller card + with 2 M.2 Sticks 240G (RAID 1),FH for the FreeNAS OS installation
• Dual 960GB SSD SATA Mix Use 6Gbps 512 2.5in Hot-plug AG
• 6 units of 1.8TB 10K RPM SAS 12Gbps 512e 2.5in Hot-plug Hard Drive
• Dual Broadcom 5720 Quad Port 1GbE BASE-T, rNDC

On this configuration, I have differents questions:

• For the CPU part, is it useful to have a double Intel Xeon or only one is enough?
• Are 64 GB of RAM enough ?
• Does anyone know if the BOSS + controller card with 2 M.2 240G keys (RAID 1) is recognized by FreeNAS OS?

For the storage part : I read differents topics on the SSD caching and RAIDZ2 / RAIDZ3, but differents opinions...
What's solution is the best ? Cost/Performance
How does SSD caching performance affect the configuration of different drives? And conversely? (SSD caching with 7.2KRPM / 10KRPM)

Indeed I hesitate with (performance for virtual machines):

• Dual 960GB SSD mix use + 6 units of 1.8TB 10K RPM SAS 12Gbps 512e 2.5in
• Dual 960GB SSD mix use + 6 units of 4TB 7.2K RPM SATA 6Gbps 512n 3.5in

What's your opinions / recommendations ?

Thanks !

 

[sretalla]

Does anyone know if the BOSS + controller card with 2 M.2 240G keys (RAID 1) is recognized by FreeNAS OS?

Maybe, but it should not be used even if it is... there is no good reason to use a hardware RAID solution with ZFS. A single (or mirrored with ZFS, not hardware if you are super-concerned with uptime) simple boot SSD is the right approach.

Are 64 GB of RAM enough ?

For 40 VMs, maybe not... what RAM and IOPS requirement do you have per VM? (or in total?)

For the CPU part, is it useful to have a double Intel Xeon or only one is enough?

Depends if your VMs are running applications that need more threads... only you can say.

For the storage part : I read differents topics on the SSD caching and RAIDZ2 / RAIDZ3, but differents opinions...
What's solution is the best ? Cost/Performance
How does SSD caching performance affect the configuration of different drives? And conversely? (SSD caching with 7.2KRPM / 10KRPM)

Again, depends on your needs... for high IOPS and block storage (iSCSI/NFS), you will want a SLOG (probably an Intel PCIe card) for write performance smoothing.

You will also need to run Mirrored VDEVs to ensure when your (approx. 30GB of) SLOG is consumed, the pool can catch up quickly enough. More mirrors (VDEVs in general) in a pool is better for IOPS. With only 6 spinning disks, you're limited to 3. Clearly SSDs have more IOPS per VDEV, so you might want to consider that.

With enough RAM, L2ARC (sometimes referred to as cache) may not be needed, but you can look at it if your RAM is maxed already and you see a lot of ARC misses. In some scenarios, it can be useful to run L2ARC in metadata only mode, but I don't think that will apply for your case as your storage volume is not large enough in total.

 

[celine]

Hi @sretalla

For the compute (CPU and RAM) virtual machines, I've 3 others proxmox hosts.

This hardware configuration is ONLY for the network storage attach to this 3 compute nodes.

Thanks

 

[sretalla]

For the compute (CPU and RAM) virtual machines, I've 3 others proxmox hosts.

This hardware configuration is ONLY for the network storage attach to this 3 compute nodes.

In that case, a single CPU is enough.

64GB of RAM is also appropriate for the amount of storage you propose.

 

[celine]

In that case, a single CPU is enough.

64GB of RAM is also appropriate for the amount of storage you propose.

Ok, thanks

For this point :

Maybe, but it should not be used even if it is... there is no good reason to use a hardware RAID solution with ZFS. A single (or mirrored with ZFS, not hardware if you are super-concerned with uptime) simple boot SSD is the right approach.

It's to install the FreeNAS OS.
In the various tutorials I have read, I have not seen an installation of FreeNAS in which we select two disks to make a RAID 1 with ZFS.

Therefore, I thought:
- Hardware RAID 1 for OS installation
- ZFS RAIDZn software for "storage" disks

 

[sretalla]

I have not seen an installation of FreeNAS in which we select two disks to make a RAID 1 with ZFS.

But you can (and should if you want a mirrored boot pool). Just pick the 2 disks instead of 1 in the installer screen.

 

[danb35]

ZFS RAIDZn software for "storage" disks

You really shouldn't be using parity RAID for block storage applications. For a lot more information, see:

Some differences between RAIDZ and mirrors, and why we use mirrors for block storage

ZFS is a complicated, powerful system. Unfortunately, it isn't actually magic, and there's a lot of opportunity for disappointment if you don't understand what's going on. RAIDZ (including Z2, Z3) is good for storing large sequential files...

www.ixsystems.com

 

[celine]

Thanks for your feedbacks.

@danb35 : So if I use 4 vdev x (2*4To 7.2Krpm), is it possible to increase his performances with SSD caching / ARC / Others ?

Or with this solution, I do only use 15Krpm or SSD disks for VM performances ?

An applications examples lists :
- MongoDB
- Nginx / Apache
- Elasticsearch
- MySQL / Postgre
- HAProxy
- SFTP servers

Thanks