slow 10G throughput

jcizzo

Explorer
Joined
Jan 20, 2023
Messages
79
I'm sure this has been beaten to death. others have asked it but they seem to be old threads.
I'm trying to ascertain if the network performance of my nas is reasonable given its hardware or if something is 'off'.

my nas (truenas core latest stable) consists of the following hardware:

supermicro x11ssh-f motherboard (flashed with latest firmware).
i3-7100T cpu.
32GB ram (ecc)
onboard nics are intels (210 i believe).
first slot (closest to cpu) is an intel x710-da2)
2nd slot - LSI 9200-based HBA in IT mode.

5 spinners attached to the HBA card (these store video/music) -set as raid-Z (so, the raid 5 equivalent).
2 SSD's mirrored for file storage (legal docs, resumes, generally the most important files).
2 SSD's for the OS.
1 NVME set as a zil for the 5 spinners.

the truenas box is bare metal and ONLY a nas.. there are NO other apps/services (no plex, or anything of that nature).
i have another download station that runs win10 that handles plex/downloads/video processing etc..) it too has a x710-da2 nic (bought at the same time as a pair. the nas and win10 box are connected by dac.

when transferring movies from the win10 box to the truenas core box, all starts off at 10Gb/s, but after a few seconds (maybe 5 or so), it the transfer:

spinners - drops down to 2-300Mb/s.. (400 if i disable hyperthreading).
SSDs - drops down to 450Mbs.

i guess i can delete the nvme and reformat it as a regular shared dataset and see if the problem still exists.
I guess i just can't imagine, even with that little cpu, 10G throughput would be a struggle.

lastly, i've made a bunch of hardware changes and blah blah.. maybe it just needs a fresh re-install of the os, however i'd like the opinions of the pros (you folks) to see if that's really 'it'. with those nics, ram, and generally the hardware that makes up the nas, i would've thought (on my humble little network at least) sustained 10G would be easy from one machine to the other.

Thanks in advance!
 

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
What makes you think you need a SLOG (what you are calling a ZIL)
Why is it there - what do you think its doing?

On your datasets - is sync set to default, enabled or disabled?
You imply a windows client. Are there amy other clients?
 

jcizzo

Explorer
Joined
Jan 20, 2023
Messages
79
What makes you think you need a SLOG (what you are calling a ZIL)
Why is it there - what do you think its doing?

On your datasets - is sync set to default, enabled or disabled?
You imply a windows client. Are there amy other clients?
Does having a slog/zil matter? from all i've read it seems that it might improve write performance as the ARC fills, plus in the event there's a powerloss or disconnect, or something, it would aid in mitigating file loss.

i'm very new to this so go easy on me..

during this testing where i'm copying files to the nas, there aren't any other devices connected. i'm still learning how to best setup accounts and folders, and all the other nuances of zfs and truenas core.

the way it's set up is as follows:

all devices are connected to my switch via their onboard 1Gb ethernet ports, which is for basic connectivity as would be in any other situation (file copy and management (so i can remote in to them), rokus, laptops, etc...)

the TNC nas and mediastation (win10 with plex, handbrake, and all the things i need for media download and processing/rendering) are connected directly to one another on the back-end, via a separate subnet, via dac using their own intel x710-da2. from there a simple smb share is created and a drive/folder mapped so the mediastation thinks the "movie" folder is local.

there isn't anything between the TNC nas and win10 mediastation.

the mediastation is built expressly for media downloads, processing/rendering media, and as a plex server. plex reads from the mapped drive on the 10G connection to TNC and then shoots it out over the 1G interface to the rest of the network.

i apologize for the long-winded explanation. just trying to provide as much info as possible.
i would like to utilize iperf, however A) i dunno how to use it properly, and B), i don't think this is a network issue.. my gut is telling me that the problem is occurring the write to the drives.. buffers are filling faster than the files can be written to the disks..

oh btw, i watch the cpu and ram utilization and as you can imagine, zfs cache gets completely utilized, and cpu resources go from idle, to 30%, to as high as 70+% (it even gives me a warning).. once the file copy is done, all drops back to 0-1% idle as it should.
 
Last edited:

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,945
A SLOG only has an effect when using Sync writes. Sync writes occur usually in the following circumstances:
1. NFS Transfers / Shares
2. iSCSi from VMWare
3. You have set sync=enabled on a dataset
4. Transfer to / from a MAC (maybe)

Othwise it does nothing. It is not a write cache, it is not a read cache. Lastly, I don't know what your NVMe drive is - but it almost certainly isn't any use as a SLOG anyway as a SLOG has very specific hardware requirements that almost all consumer drives don't do.

When it comes to speed of writes. Async > Sync (with Slog) > Sync (Without Slog). What a SLOG does is make Sync writes safer, IF you use sync writes which most home users won't, don't and shouldn't

Having said all that, if it does nothing then it has no effect on what is going on.

What are the HDD's - you have just said they are Spinners, but no other details?

As a general comment - you have done nothing wrong with the hardware unless the disks are SMR
 

jcizzo

Explorer
Joined
Jan 20, 2023
Messages
79
thanks for the reply.
my point about the slog in response to your query was that, as you pointed out, it's not affecting write speed.

the spinners are old seagates, and yes they're sata2 (3Gb/s). however there are 5 of them and if you do the math, not including 1 drive for parity (which i know zfs handles it differently, but as a general rule), even at ~4 drives x ~3Gb/s, it should be able to sustain at least 600MBytes/s (at least, i would think).

the HBA is an LSI 92xx in IT mode (of course). it's pcie2.0 based, but thats still more than enough bandwidth to saturate the spinners (i would think).

plus, as i said earlier, i also have 2 SSDs (connected directly to the motherboard, samsung, newish, although the model number escapes me at the moment), and those are set as raidz mirrors. when writing to those, it still sustains about 500Mbytes/s.. that is with the same video files (roughly 25~Gigs in size). i can almost see (almost) how the SSD's might be limited, since there are only 2 of them and they're mirrored.. but even then, if each is connected at 6Gb/s, there shouldn't be any problem with throughput.

of course, performing the exam same file copy across the 1G interface runs as smooth as glass.. (112Mbytes/s the entire copy).

maybe its just the cpu isn't up to snuff? maybe TNC just needs to be re-installed from scratch?

yes, i'm wordy.. sorry.. i'm typing out as i think..
 
Top