ZFS ZIL L2ARC 16GB RAM 10GbE

[bumann]

Hi,

i am new to FreeNAS, I have some design questions.
I want to use it for shared NFS Storage for two Proxmox VE Systems.

My Hardware is:
Intel Xeon X3430 CPU, 16GB ECC RAM (MAX)
3x 32GB USB Stick for FreeNAS System
2x 4TB 7200 NAS HDD Mirror
1x 2 Port Infiniband 10GbE NIC
2x 160gb Intel 3510 SSD (not used yet)

Do you think I can get better performance with the SSD ZIL and L2ARC?
I read in the forum it isn't good to use ZIL/ARC with only 16GB RAM.

 

[Chris Moore]

My Hardware is:
Intel Xeon X3430 CPU, 16GB ECC RAM (MAX)

Why do you say that is the 'MAX' memory? The CPU should support 32GB, and honestly I would think you will want more.

3x 32GB USB Stick for Freenas System

I don't see a reason to use three. Better to buy a single small capacity SSD.

2x 4TB 7200 NAS HDD Mirror

To host virtualization, IOPS would be important and more drives will give more IOPS. To that end, if I were building this, I would rather go with 8 x 1TB drives and create 4 mirrored vdevs in my pool. I am no pro at the virtualization thing, but I would expect better performance from that. A single mirror, the way you suggest, would only give the performance of a single drive, which can be quite poor if you are trying to read and write data from multiple sources simultaneously.

1x 2 Port Infiniband 10GbE NIC

Brand matters. Some cards work well with FreeNAS and others do not work as well and yet others may not work at all. This network thing brings me back to the drives though, if you want to actually be able to utilize the speed of 10GB networking, you need a bare minimum of 10 drives. Sorry.

2x 160gb Intel 3510 SSD (not used yet)

What would you use these for?

Do You think I can get better performance with the ssd ZIL and L2ARC

Absolutely, under the correct circumstances, but you need to use the right kind of hardware and you should maximize other avenues such as RAM. What system board are you thinking to use?
It would be grand if you could give us more details. Is this hardware you have and intend to re-purpose or is this something you want to purchase?

I read in the forum it isn´t good to use ZIL/ARC with only 16GB RAM

You really need more memory. Lookup tables are kept in RAM and trying to makeup for not having RAM by adding SSDs is not going to make it work better.

You might find these resources useful:

FreeNAS® Quick Hardware Guide
https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

 

[Nick2253]

To add a couple more things to @Chris Moore's excellent response:

Intel Xeon X3430 CPU

The X3430 is an extremely old and outdated processor. It's power hungry, and you could pay for a semi-modern system in a year or less with the power savings alone. At approximately 30 euro cents per kWh (Q4 average in Germany), an X3430-based system will cost you about €400 per year to operate. A Pentium G4xxx based system will use less than a quarter as much power, and provide just as much performance. I'm not sure what the used electronic market is like for you in Germany, but even going back as old as a Sandy Bridge would be a huge step up from what you have now.

The other major benefit to moving up to a newer platform is you'd be able to grow to 64 GB of memory. For your use case, adding more memory would lead to major performance gains.

ZIL

All ZFS systems have a ZIL. When you're really talking about here is a SLOG (separate intent log).

2x 160gb Intel 3510 SSD (not used yet)

I'm assuming these are a DC S3510. This is a good choice for a SLOG, but not a great choice for an L2ARC. If you can't get to 64GB of memory, I would not recommend using an L2ARC.

 

[bumann]

Thanks for your reply, I think I must change some hardware parts :)
The server I want to use for the System is to old, max 16gb ram and too power hungry, thanks for the tip.

My new Plan is to buy a DELL PowerEdge T30 Server Xeon E3-1225 V5 8GB RAM 1TB for 350€.
I want to add
3x32GB usb3 SanDisk Ultra Fit 32GB
2x16GB RAM /buy
1x 10gbe Mellanox MHQH29-XTC
1x DC S3510 160GB for SLOG
1x M2 / NVME SSD (Samsung 960 EVO 250 GB ?) for L2ARC /buy
1x Bigger Case to add more HDD /buy
8x1TB 7200 HDD
1xPCIe RAID Controller to connect more drives, i think i have some controller must lock for type

(buy = i must buy it)

 

[wblock]

I strongly recommend replacing the USB boot drives with a 128G or 64G SSD.

 

[Chris Moore]

I strongly recommend replacing the USB boot drives with a 128G or 64G SSD.

I already told him that but he didn't listen.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[nightshade00013]

Mellanox MHQH29-XTC

I have my doubts on this as well. I know the Mellanox Ethernet cards work ok but the infiniband are a long shot for networking. If you already have it I would suggest spinning up a FreeBSD or FreeNAS install and see if it will work before you make plans with it.

 

[bumann]

ok, many thanks for the tips. I will change the boot drive to ssd and test the nic. I think about to buy another system because of the ram cost.
I think about a used server but i need one is not so loud.

 

[bumann]

Here some updates from my system.
I tested the mellanox card with no luck and change to Intel x520 dual card.
I buyed the DELL PowerEdge T30 Server Xeon E3-1225 V5 and fit it into new case where i can install up to 15 3.5 Drives.
I crossflashed my Fujitsu Sas LSI SAS2008 based controller.

So my test System is now:
1x DELL PowerEdge T30 Server Xeon E3-1225 V5 Mainboard
3x32GB usb3 SanDisk Ultra Fit 32GB
2x16GB ECC RAM in Dual chanel mode
1x 10gbe Intel X520 Dual Port NIC
2x DC S3510 160GB
1x M2 / NVME SSD Samsung 960 EVO 250 GB
8x1TB 7200 HDD
1xPCIe RAID Controller / LSI SAS2008 based / 8 Port

The main goal is to build a NFS Storage Server with many IOPs for my two Proxmox Servers.
I want to build a 4 mirrored vdevs in my pool as suggested.

I have some Questions:
1.What is the reason to change the 3xusb3 sticks to one 128gb consumer SSD? Is it a performance issue?
2.Do you think 32gb RAM is ok to use the NVME SSD as L2ARC Disk. What is the best size in this situation.
3. Should i use both DC S3510 SSD with mirror for SLOG or only one disk is ok? Size for SLOG?
4. Should i install a additional 8gb ECC Ram (I have one), and lost dual channel for the 2x16gb ram instead?

 

[IQless]

1.What is the reason to change the 3xusb3 sticks to one 128gb consumer SSD? Is it a performance issue?

Someone please correct me if I am wrong.

The main reason would be that USB sticks have a tendency to die with little to no notice. SSD is much more reliable and it will inform you of issues due to SMART info. It will also be much faster when it comes to upgrades.

 

[Bidule0hm]

Yep, exactly, USB sticks are very unreliable, a SSD will be better than even a mirror of USB sticks. And it'll be faster during updates of course. No need for 128 GB unless cost is not a problem or you already have it ;)

Whatever you chose remember to keep a backup of the FreeNAS config somewhere in case you need to reinstall the system.

 

[bumann]

thanks for info, i will change boot drive to small ssd.

What do you think about the other questions:

2.Do you think 32gb RAM is ok to use the NVME SSD as L2ARC Disk. What is the best size in this situation.
3. Should i use both DC S3510 SSD with mirror for SLOG or only one disk is ok? Size for SLOG?
4. Should i install a additional 8gb ECC Ram (I have one), and lost dual channel for the 2x16gb ram instead?

 

[Bidule0hm]

2. AFAIK the rule is to max out the RAM before even thinking about a L2ARC. But wait for someone else to confirm.

3. Don't know.

4. More RAM is almost always better than keeping dual channel.

 

[wblock]

Size for SLOG?

From the User Guide:

ZFS currently uses 16 GB of space for SLOG. Larger SSDs can be installed, but the extra space will not be used. SLOG devices cannot be shared between pools. Each pool requires a separate SLOG device. Bandwidth and throughput limitations require that a SLOG device must only be used for this single purpose. Do not attempt to add other caching functions on the same SSD, or performance will suffer.

 

[nightshade00013]

https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/

Probably also good information.

SLOG should be mirrored however since a failing device can mean data loss.

Before doing L2ARC you will probably want a lot more ram, using the SLOG will free some up but not enough to make the addition worthwhile on a low ram system so basically if you can't buy the 64GB of ram you should wait for L2ARC till you can as the additional ram will end up being of more benefit than the L2ARC which will be much slower.

 

[bumann]

ok, i have ordered additional 32gb ram. so i will go with 64 gb ecc ram.
i will use both DC S3510 SSD with mirror for SLOG.
I will add two 4tb hdd drives in mirror in a separate pool.
Do you think it make sense to add 1/2 DC S3510 ssd drive (in mirror) to slog and 1/2 size of mve ssd l2arc to each pool?

 

[nightshade00013]

I personally wouldn't. Better to underprovision the SLOG and not risk killing your pool/data than to worry about trying to use them double duty. The SLOG will see a lot of writes to it and having it give up the ghost will really suck if you are not prepared for it.

L2ARC however can be a single drive or a stripped array since it's loss just drops it and causes the system to just use RAM as ARC. For that matter with 64GB of RAM it may not even be needed.

 

[wblock]

Do you think it make sense to add 1/2 DC S3510 ssd drive (in mirror) to slog and 1/2 size of mve ssd l2arc to each pool?

From the User Guide quote above:

Do not attempt to add other caching functions on the same SSD, or performance will suffer.