Truenas Core 13 L2ARC Drive to Help with Slower Transfers?

[Jazz30-06]

Hello all,

I finally got my machine up and running and... the data transfer rates are kind of slow. I'm on gigabit LAN and I was only seeing like 20Mb transfer speeds at best to my NAS. Is this something that a L2ARC drive would help with? I've been looking at getting an optane drive and running with that, but I'm not sure if it will solve the problem. For reference:

Main PC:
7600X
32GB RAM
2.5Gb Killer LAN port
Crucial P5 drive that tests fine

NAS:
Xeon E5-1650 v4
32GB RAM
1Gb LAN port (Intel?)
256G Samsung 850 Pro
3x 4T WD Red+ in RaidZ

Router:
Netgear RAX35

I don't think there is anything in the network that could bog it down that much and I am using the NAS as a backup for everything I am doing on my main machine as well as long-term storage for CFD results when I get them. So, I'd really like to make things more efficient if possible.

Any help would be appreciated.

Thanks

 

[ChrisRJ]

RealTek NIC in your main PC?

If you mention the router because you are using the built-in switch, that could very well be the reason.

You provided zero information how you did your performance test, so that may also a reason. Please also check the forum rules (linked at the top of the screen in red) how to best ask questions.

 

[Jazz30-06]

RealTek NIC in your main PC?

If you mention the router because you are using the built-in switch, that could very well be the reason.

You provided zero information how you did your performance test, so that may also a reason. Please also check the forum rules (linked at the top of the screen in red) how to best ask questions.

Apologies, but how does my post not conform to the rules? I just missed the type of LAN port, which is now updated. When I posted this, I was referencing the windows transfer speed measurement, but I've now run a test in CrystalDisk and it looks like I am primarily just saturating the network:

I'm very new to the storage game, but the exception to saturating the network is the RND4k Q1T1, which googling says is small file random read/write. Which... might be the case when I am doing large folder syncs with a variety of data?

 

[c77dk]

Those disks - check if they're CMR or SMR. If SMR, then change them asap

 

[Jazz30-06]

Those disks - check if they're CMR or SMR. If SMR, then change them asap

Regular Reds are SMR. Red+ are CMR, but thanks for noting that.

 

[Davvo]

L2ARC will not help you. Nor will a SLOG (if it does that's the issue).
What is your NAS motherboard? Such performance usually happens when you have a realtek NIC on it.
How are you sharing those files? By SMB? Which kind of files? How large? What are the datasets settings?
Did you perform any test on the NAS side?

 

[ChrisRJ]

Apologies, but how does my post not conform to the rules?

I did not mean to convey that, but wanted to point out that in addition to specifying rules, there is also good guidance on how to best ask questions. No need to worry

 

[Jazz30-06]

L2ARC will not help you. Nor will a SLOG (if it does that's the issue).
What is your NAS motherboard? Such performance usually happens when you have a realtek NIC on it.
How are you sharing those files? By SMB? Which kind of files? How large? What are the datasets settings?
Did you perform any test on the NAS side?

It's an HP Z440 workstation that has been converted. The NIC is Intel (Intel I218LM GbE platform LAN connect networking controller)
Files are being shared via SMB and Nextcloud. When I was seeing 20Mb/s, it was over SMB. Not sure how to measure for Nextcloud
File types are mixed from Gigs to Kilos. I left the dataset settings as default.
I'm honestly not sure how to test on the NAS side. But I thought using CrystalDisck over the network would capture the performance of the entire system. Am I missing something?

 

[Davvo]

You don't have syncwrites always on, do you?
An iperf test from the NAS would be useful.

 

[Jazz30-06]

You don't have syncwrites always on, do you?
An iperf test from the NAS would be useful.

Are you talking about this?

Also, took a minute to figure it out, but here is the iperf test results:

 

[Davvo]

Are you talking about this?
View attachment 61834

No, but I guess you used the standard values for everything?

Also, took a minute to figure it out, but here is the iperf test results:

View attachment 61835

We can rule out anything network related I think.

 

[Jazz30-06]

No, but I guess you used the standard values for everything?


We can rule out anything network related I think.

Yes, I don't really know what I am doing, so I had no reason to deviate from standard settings.

That's good that we can rule out networking because the "network" is just the single switch in the router.

 

[Jazz30-06]

No, but I guess you used the standard values for everything?


We can rule out anything network related I think.

And, for some reason, I can't find the syncwrites setting you are referring to if what I posted is not it.

 

[winnielinnie]

You get 20 MB/s when copying files to the NAS? What about copying files from the NAS to your client?

Can you test this out with a single large 1 GB file of random data? From Windows to TrueNAS, this will be a large sequential write.

(The ARC/L2ARC won't help when writing new data to the NAS.)

And, for some reason, I can't find the syncwrites setting you are referring to if what I posted is not it.

SMB uses async writes by default, and your settings use the default, as seen in your screenshot. Sync is set to "standard", which lets the application decide.

 

[Davvo]

And, for some reason, I can't find the syncwrites setting you are referring to if what I posted is not it.

You did, it's the Sync parameter in the screenshot you posted.
I just missed it before.

 

[Jazz30-06]

You get 20 MB/s when copying files to the NAS? What about copying files from the NAS to your client?

Can you test this out with a single large 1 GB file of random data? From Windows to TrueNAS, this will be a large sequential write.

(The ARC/L2ARC won't help when writing new data to the NAS.)


SMB uses async writes by default, and your settings use the default, as seen in your screenshot. Sync is set to "standard", which lets the application decide.

A single random 1GB file from an online generator moves in both directions at ~100MB/s

Trying to generate 1000 files of 1MB to see if that's it.

When I noticed things being slow, it was a mix of both larger files and small ones.

 

[winnielinnie]

A single random 1GB file from an online generator moves in both directions at ~100MB/s

But did you test this over SMB from Windows Explorer? This way all your tests use the same method/software.

 

[Davvo]

When I noticed things being slow, it was a mix of both larger files and small ones.

This is expected in a RAIDZ pool, but not to the extent of 20MB/s I think.
Anyway, I suggest you reading the following resource.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

 

[Jazz30-06]

But did you test this over SMB from Windows Explorer? This way all your tests use the same method/software.

Yes, I am checking things with file transfers through explorer.

A folder with 1000 1MB files transfers at 30-40 MB/s.

FROM the NAS is 13-18 MB/s

 

[Jazz30-06]

This is expected in a RAIDZ pool, but not to the extent of 20MB/s I think.
Anyway, I suggest you reading the following resource.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

So, when I configured the pool, there was ONLY the option of "ZFS" (or "raidZ", can't exactly remember) or "mirror", not z1, z2, or z3. I'm not really sure how else I could've configured it.