Next Best Performance Upgrades?

[isopropyl]

Looking to see if I can increase performance in my system.
I heard adding a caching SSD may not be beneficial for zfs depending on the setup.

What are the best options for my setup curiously?
Specs in my signature.

I was thinking maxing out my ram. Would this give me massive gains, or only marginal?

I am on a 1gbps link, but I don't think I can even saturate that. I will likely upgrade to 10gbps for the hell of it in the future. But beyond these two things, what are my next options, besides a full SSD pool.

 

[WI_Hedgehog]

Are you simply looking to do the IT version of a drunken bender? (in which case you're in the right place--though most members here like saving others' money -->I<-- have no problem spending it for you.)

Being serious for a moment, if you're not saturating a 1Gbps link you're probably not bottlenecking your server at all, and if it is the case the server is outrunning the client your money is best kept in your pocket. Two years from now prices on current equipment will have fallen drastically and storage size will have increased. The only thing that depreciates as fast as computer equipment is a 34-year-old's looks I mean brand new car.

 

[isopropyl]

I

Are you simply looking to do the IT version of a drunken bender? (in which case you're in the right place--though most members here like saving others' money -->I<-- have no problem spending it for you.)

Being serious for a moment, if you're not saturating a 1Gbps link you're probably not bottlenecking your server at all, and if it is the case the server is outrunning the client your money is best kept in your pocket. Two years from now prices on current equipment will have fallen drastically and storage size will have increased. The only thing that depreciates as fast as computer equipment is a 34-year-old's looks I mean brand new car.

I'm not sure if I am saturating the 1gbps. Would CrystalDiskMark be a reliable way to determine if I am?
Also if I am not, wouldn't that mean thr bottleneck is actually the server? Because if it isn't saturating it, then I'm not using the full link. If I was saturating it, it would be at 100% meaning the bottleneck would be the link.

And yeah I know stuff deprecates over time, that's just technology.

 

[WI_Hedgehog]

With your system I would *guess* the bottleneck is *probably* the 1Gbps link, and it's *possibly* saturating at 600Mbps or so?

If you plug a spare drive into the server and do a cp to the array you might be surprised at how fast it is. Of course if you're running 3+ users with a lot of random reads/writes (video editing) that would be different.

 

[isopropyl]

With your system I would *guess* the bottleneck is *probably* the 1Gbps link, and it's *possibly* saturating at 600Mbps or so?

Ok maybe I misunderstand something here.

If it is saturating at 600Mbps, wouldn't that mean it is not thr 1gbps link that is the bottleneck? Because 600mbps is less than 1gbps.

Also is my 64gb of ram not actually that restrictive?
Edit: Btw I know I mentioned iSCSI. After a little research, I notice that there is a "Multi-Path" for iSCSI. But I'm a little confused on how it works. Seems like you need another NIC active in the machine. In which case, it seems like it'd make more sense just to setup LACP?

 

[WI_Hedgehog]

If it is saturating at 600Mbps, wouldn't that mean it is not thr 1gbps link that is the bottleneck? Because 600mbps is less than 1gbps.

Also is my 64gb of ram not actually that restrictive?

There is packet overhead, so 1Gbps tends to saturate somewhere between 600 & 700Mbps (of data) unless Jumbo Frames is enabled (more data on a frame with fewer frames needed) (from my experience, and I'm not an authority on the subject). Your system is likely to far outpace the network connection.

If you're running TrueNAS on bare metal with no add-ons and 1 user you're probably okay with RAM. There are always fringe cases to show differently, but by definition those aren't common.

 

[isopropyl]

unless Jumbo Frames is enabled (more data on a frame with fewer frames needed) (from my experience, and I'm not an authority on the subject).

I had tried enabling Jumbo Frames throughout my devices in the past, but noticed some serious instability. It seemed to just slow everything down or cause things to freeze up. Seems like a PITA because it needs to be enabled on every single device too (from what I read). So I don't really plan on using it again unless I am wrong. Also it doesn't seem that commonly used anyways. Maybe for a reason lol.

There is packet overhead, so 1Gbps tends to saturate somewhere between 600 & 700Mbps (of data)

Also ah, didn't realize there was that much packet overhead, wow. But ok good to know.

Crystal Disk Mark Testing:

The only major thing the Windows machine and NAS is doing currently is running a first scan of large libraries into Jellyfin (fresh install). Whether or not that will massively affect this. Not sure.
But running CrystalDiskMark on my Windows 10 Enterprise LTSC machine, these are what I get for results..

First on iSCSI (Z drive). I have CDM Profile set to "Real World Performance" and "Read+Write[+Mix]":

Second on the Samba network share (Y:). Again, I have CDM Profile set to "Real World Performance" and "Read+Write[+Mix]":

Not sure if the settings for CrystalDiskMark were the optimal way to test this, so please correct me if I did them wrong. But that is what they show.



I notice that my CPU usage according to TrueNAS on the main dashboard (first pic) is only hitting about 2-4% max and it drops back to 0. So it goes like 0% to 1% to 0% to 2% to 0% to 1% to 0% to 4%. Not sure if that is weird?

and here is more detailed CPU and Network graphs from TrueNAS reporting if it is of interest.

Btw igb0 is not even an active link, it's down.. so not sure why it even shows anything there on the graph. I see it's in b/s though so idk not concerned.
But just wanted to make that clear. igb1 is the only active link.

 

[isopropyl]

Guess I didn't pose a question.
Are the results from CrystalDisk, a bit mediocre?

Only 6 MB/s random..?
I guess the sequential are pretty close to what you suggested though. 100MB/s comes out around 800mbps right. And the network graph shows a max of around 780mbps for RX and TX.. So I guess I am bottlenecked at my networking.

But the random speeds don't make sense to me? That is super low.

Also is that low CPU usage normal? I know the CPU is a bit overkill, but not much more than 4%? lol
Right now it's resilvering and I'm playing a movie from it (Jellyfin on my desktop, but the media file on the NAS) and it is showing "Highest Usage: 1% (3 threads at 1%)". Like that is insanely low it seems. Not complaining, but just wanting to make sure it's not constricted or something.

 

[WI_Hedgehog]

Reading your Are 2 of My Drives Failed? thread it seems you are looking to not spend money unnecessarily, which is completely reasonable and what the majority of members here strive for. With that in mind, benchmarks aren't "real world" use; if your system runs well and you're happy with it don't worry about benchmarks. If your system is not meeting your needs then isolate the bottleneck and "make things faster."

A potential alternative is this is your hobby and you're looking to squeeze more out of your system, and that can result in knowledge useful in real-world applications (and increased value to either your company, or another company). To that end look up top on the menu bar under Resources, and I also have a resource link in my signature that covers topics in logical order of progression; the more you read the more this all makes sense. Perhaps acquire used equipment on the cheap and set up a test system dedicated to TrueNAS and expanding your knowledge. This helps with testing parts, theories, etc. without taking your main system down or doing something unintentionally catastrophic. It also helps with system design and testing because you can slowly build up your next server (on the cheap) and compare it against your production (running) server. As @joeschmuck said, parts should be considered consumable (kind of like cars, they don't last forever though we often wish they would, and the ones that tend to last forever are the ones we wish would die ).