Looking to improve performance

[rmccullough]

I am looking at making some upgrades over the next year or 2, but I am trying to figure out where I might attain some modest gains with a little investment now.

Is it generally a good idea to use an SSD for Jail/Bhyve storage vs putting it on your pool? Does it usually offer better performance and then use snapshots for data protection?

When should one consider adding a SLOG?

 

[Heracles]

We can not tell you what you should upgrade in your system if you do not tell us what your system is.

Your system will be as slow as your slowest component. So tell us about everything and only then will we be able to tell you where you can get more speed and how.

 

[rmccullough]

I apologize. It’s in my signature, but for historical reasons I should probably put it here.


Supermicro 2U (CSE-826A-R1200LPB) 12x 3.5" Drive Bays
2x Supermicro (PWS-920P-1R) 920Watt Power Supply - 80 Plus Platinum
Supermicro X9DRi-LN4F+

• 1x Intel Xeon E5-2620 v1 HEx (6) Core @ 2.0GHz
• 1x Intel Xeon E5-2630 v1 HEx (6) Core @ 2.2GHz
• 4x Intel® i350 GbE controller
• 32GB Hynix ECC Registered PC3-10600 (8 x 4GB)

32GB Supermicro SATADOM (boot drive)
2x LSI 9210-8i (8 port HBA flashed to ITMODE for JBOD)
9 x 2TB SAS 6GB/S Hitachi GST ULTRASTAR (1 x 2TB Seagate IronWolf as spare)

I have 1 bhyve VM running Ubuntu 20.04 and about 10-12 jails running various things like Plex, OpenVPN, LAMP stack, Sonarr, Radarr, Lidarr, Plex, etc.

 

[rmccullough]

Fwiw, I am watching a pair of processors on eBay to look and upgrade that, but wanted to know if I should be looking at SLOG or ssd for jails/VM as well.

I will also probably upgrade drives in this machine or to a whole new chassis and motherboard in the future to reduce noise.

 

[adrianwi]

Throwing lots more RAM looks like an easy recommendation, along with your own suggestion of an SSD pool to run the jails and any databases on.

 

[rmccullough]

I’m wondering if I should try to find 4x8gb on eBay and add an SSD to one of the trays for now. Thanks.

 

[jgreco]

I apologize. It’s in my signature, but for historical reasons I should probably put it here.


Supermicro 2U (CSE-826A-R1200LPB) 12x 3.5" Drive Bays
2x Supermicro (PWS-920P-1R) 920Watt Power Supply - 80 Plus Platinum
Supermicro X9DRi-LN4F+

• 1x Intel Xeon E5-2620 v1 HEx (6) Core @ 2.0GHz
• 1x Intel Xeon E5-2630 v1 HEx (6) Core @ 2.2GHz
• 4x Intel® i350 GbE controller
• 32GB Hynix ECC Registered PC3-10600 (8 x 4GB)

32GB Supermicro SATADOM (boot drive)
2x LSI 9210-8i (8 port HBA flashed to ITMODE for JBOD)
9 x 2TB SAS 6GB/S Hitachi GST ULTRASTAR (1 x 2TB Seagate IronWolf as spare)

I have 1 bhyve VM running Ubuntu 20.04 and about 10-12 jails running various things like Plex, OpenVPN, LAMP stack, Sonarr, Radarr, Lidarr, Plex, etc.

Uh, technically it isn't supported to have mismatched CPU's.

If you feel like your system is slow CPU-wise, the E5-2643v2 CPU's are generally cheap on eBay, and are 6 cores at 3.5GHz (21GHz), which makes singlethreaded things fly. If you shop carefully on eBay, you can pick up a pair for $120 sometimes.

The other CPU option I think is sensible are the E5-2697v2's, 12 cores at 2.7GHz (32GHz) which makes for fast work of parallel tasks. You can pick up a pair for $250 sometimes.

Neither is likely to make FreeNAS itself all that much faster, but if you have lots of jails, it can be a win.

Memory is also cheap.

 

[Heracles]

For that many jails and a VM on top of that, RAM is where you need to go. As @jgreco mentioned, to have mismatched CPU is asking for trouble.

Also, you did not specified the layout you used for building your vdevs and pool. For running many things in parallel, you should go for mirrors.

 

[rmccullough]

@jgreco I agree and would like matched CPUs. That is why I am looking at upgrading to a matched pair.

I am watching for some E5-2667v2 and E5-2643 V2. I would personally rather have fewer cores at a higher clock as I would like better transcoding performance in Plex.

@Heracles fair point. Thus far I haven't seen the system use all of the 32GB of ram, but I know that more ram is usually a good thing for Free/TrueNAS so I may try to pick up another 32GB of ram as well.

I created 1 pool and 1 vdev for the 9x2TB drives.

 

[ChrisRJ]

I would recommend to read up on performance tuning in general, since the process is always the same. If the system (any system) does not yield the desired performance, you need to look for the bottleneck, "fix" it, and then start all over again. Repeat until you are satisfied.

Something that needs to be done largely by you, is to define your performance requirements and measure the actual values against them. You will likely see that this approach alone will make things much clearer. Without this information, no reasonably targeted help is possible.

 

[sretalla]

I created 1 pool and 1 vdev for the 9x2TB drives.

With that setup, SLOG will never be useful. At least that much should be clear. A pool of Mirrored VDEVs (as many as possible) is what you should have first before considering SLOG to flatten out IO peaks.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

 

[ChrisRJ]

With that setup, SLOG will never be useful. At least that much should be clear. A pool of Mirrored VDEVs (as many as possible) is what you should have first before considering SLOG to flatten out IO peaks.

I must admit that I don't understand the logic in this (or my understanding about SLOG devices is wrong). I agree that it usually makes sense to first look into multiple mirror VDEVs, and whether they provide enough of a performance improvement over RAIDZx. But if I want to skip that, for whatever reason, I don't understand why simply adding a properly sized SLOG (or pair thereof) will not give me the performance boost I need. My understanding is that a sync write request is sent in parallel to the SLOG and the regular pool VDEVs. And as soon as either of those (SLOG or pool VDEVs) confirm that the write is completed, the request is confirmed to the client. And as I see it the SLOG speed will not depend on the speed (IOPS) of the pool VDEVs, unless the SLOG capacity is fully exhausted.

Looking forward to feedback.

 

[Heracles]

or my understanding about SLOG devices is wrong

That would be pretty common... Do you think that SLOG is a write cache ? If you do, then indeed you got it wrong.

Here is an article about the basics written by @jgreco

Extracts :

The solution to this is to move the writes to something faster. This is called creating a Separate intent LOG, or SLOG. This is often incorrectly referred to as "ZIL", especially here on the FreeNAS forums.

The ZFS write cache. (No, it's not the ZIL.)

 

[ChrisRJ]

That would be pretty common... Do you think that SLOG is a write cache ? If you do, then indeed you got it wrong.

Here is an article about the basics written by @jgreco

Did you actually read my post in its entirety? Because I think that I describe there as my understanding is actually (at least in essence), what is written in the linked article.

 

[mstang1988]

I must admit that I don't understand the logic in this (or my understanding about SLOG devices is wrong). I agree that it usually makes sense to first look into multiple mirror VDEVs, and whether they provide enough of a performance improvement over RAIDZx. But if I want to skip that, for whatever reason, I don't understand why simply adding a properly sized SLOG (or pair thereof) will not give me the performance boost I need. My understanding is that a sync write request is sent in parallel to the SLOG and the regular pool VDEVs. And as soon as either of those (SLOG or pool VDEVs) confirm that the write is completed, the request is confirmed to the client. And as I see it the SLOG speed will not depend on the speed (IOPS) of the pool VDEVs, unless the SLOG capacity is fully exhausted.

Looking forward to feedback.

People like to get angry on here about slogs as you can already see.

SLOGS are only going to be useful for sync writes and don't have to be too large (but low latency is good). ASYNC writes still go to your vdev pool after they are aggregated in a transaction group and written directly to your pool.

 

[Patrick M. Hausen]

@ChrisRJ Are you using NFS or iSCSI with sync=always for VMs?

Because if you don't you simply won't have synchronous writes and that is the reason why an SLOG device won't buy you anything.

 

[mstang1988]

I am looking at making some upgrades over the next year or 2, but I am trying to figure out where I might attain some modest gains with a little investment now.

Is it generally a good idea to use an SSD for Jail/Bhyve storage vs putting it on your pool? Does it usually offer better performance and then use snapshots for data protection?

When should one consider adding a SLOG?

Where is the slow? Is it in network transfers? RAW disk performance? Responsiveness? What's your I/O size and mix? I'd say describe more about your perceived slowness and folks can help out.

Generally Truenas/freenas likes lots of RAM and sync I/O likes low latency responses.

 

[ChrisRJ]

@ChrisRJ Are you using NFS or iSCSI with sync=always for VMs?

Because if you don't you simply won't have synchronous writes and that is the reason why an SLOG device won't buy you anything.

Thanks for asking! But I am not using FreeNAS for running VMs off of it at all . I simply took part in the discussion out of interest.