SOLVED: Slow read speed on on LSI2008 RAIDZ2 - needs x8 slot

[Alister]

System

CPU: i5-6400
RAM 2x16GB 2x8GB 48GB Total
Motherboard H170A PC Mate
NIC Chelsio Communications Inc S320-LP-CR 10GbE MTU9000

Pool1
Drives 6xWD40EFRX-68W RAIDZ2
Controller LSI SAS2008 [Falcon] rev03

Pool2
Drives 2x 120GB SSDs (Cheap Kingstons)
Controller M/B SATA ports

Switch MikroTik CSS326-24G-2S+RM - TrueNAS & Client in SPF+ ports

PC (Client)
CPU AMD 5950x
Memory 32GB
NIC Chelsio Communications Inc S320-LP-CR 10GbE MTU9014
NVMe used for tests Read 3400MB/s Write 3300 MB/s

Iperf3 gets 8 - 9 Gb/s

Copy 18GB test file from pool 1 to NVMe SSD 80-90 MB/s
Copy18GB test file from NVMe SSD to pool 2 200 then pauses then 200 - as expected with these SSDs
Copy 18GB test file from pool 2 to NVMe SSD ~350 MB/s

then

Copy 18GB test file from pool 2 to pool1 on client ~330 MB/s (this is weird?)

Zpool -status is clear for both pools

lspci -vv for SAS2008 shows

LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)

is this a problem?


diskinfo -ctv /dev/[Disk] attached

 

[jgreco]

Mmm, would say it seems a little slow. However, the older Chelsio cards are somewhat limited; your system is probably working really hard to get that 8-9Gbps iperf3.

Have you found the optimization guide I posted recently in Resources?

Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks

Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...

www.truenas.com

 

[Alister]

You're missing the point

Pool 1 - the SAS2008 gets 80MB/s
Pool 2 - the M/B SATA gets 300+ MB/s

I included all the testing to show that

the 10GB NIC is not the issue
the 10GB Switch is not the issue
the client drive is not the issue
etc

The single variable is SAS2008 RAIDZ2 pool

 

[c77dk]

if you look with gstat while doing a transfer - are the disks busy?
How is the pool setup? Dedup? sync ?

 

[Alister]

No DeDup or sync just a straight RAIDZ2 array

SAS2008 2 ports , 4 SATA cables per port, with 3 drives per port

Gstat show busy rarely excess 20% with odd spike to 40% most of the time <6% kBps is ~20000 - 25000

 

[WI_Hedgehog]

Am I understanding this correctly?

The spinners that can handle about 150 MB/s max. (sequential without seeks) are doing Copy-On-Write (meaning there are seeks) at 80 MB/s.

The SSDs that can handle about 350 MB/s max. are striped? and handling 300 MB/s, or 150 MB/s each?

Copying an 18 GB existing test file (not incoming) from mirrored? SSD to sequential-write HDD gets cached in 48 GB RAM and the transfer seems faster, though the write may not have actually completed?

 

[Alister]

I'm coping 18 or 60GB file off the 6 spinners at a total 80MB/s onto the network

I just copied with pool1 on the server (Putty)

kBps ~30000 busy 60-70%

I was wondering if

NIC in x16 slot
SAS2008 in x4 slot

is the issue and turns out it is

Just swapped the cards and am now able to copy from pool1 to NVMe at 300 MB/s - drives are now showing>30% and spiking to 80% - I'm limited by the Chelsio Communications Inc S320-LP-CR which is showing as LnkSta: Speed 2.5GT/s (ok), Width x4 (downgraded)

(Now need to find a CHEAP PCIe 2/3 x4 SPF card)


Solution LSI SAS2008 / 9100-8i cards need a x8 PCIe slot - LnkSta: Speed 5GT/s (ok), Width x8 (ok)

 

[WI_Hedgehog]

NIC in x16 slot
SAS2008 in x4 slot

Just swapped the cards and am now able to copy from pool1 to NVMe at 300 MB/s - drives are now showing>30% and spiking to 80% - I'm limited by the Chelsio Communications Inc S320-LP-CR which is showing as LnkSta: Speed 2.5GT/s (ok), Width x4 (downgraded)

Solution LSI SAS2008 / 9100-8i cards need a x8 PCIe slot - LnkSta: Speed 5GT/s (ok), Width x8 (ok)

Agreed, the LSI SAS2008 supports 8 PCIe lanes and complies with the PCI Express 2.0 specification that extends the PCIe data rates up to 5.0 GT/s per lane per Broadcom (specs linked).

 

[jgreco]

You're missing the point

I'm really not. The point I'm asking you about is whether or not any additional tuning had been done. It seems clear that something is a bottleneck, but sometimes with the older cards it is just that there's a lot of overhead in the handling of all the activity; back in the mid 2000's it was absolutely necessary to set big buffers and stuff so that the kernel could just plow through large tracts of bytes without encountering a context switch from kernel back to userland to get more data queued up. My head still goes there when I see an old NIC, an old HBA, and a low thread count CPU. It seems in the meantime you've found some other bottlenecks in the system, so that's fine, I expected SOMETHING was a bottleneck but you gotta start somewhere. Tuning is cheap and easy if it turns out to be the issue. Because you do have some slower parts in your system, even if you can fix the PCIe lane issues, you will find it advantageous to trade off some RAM for network buffers. You've definitely got the right ideas about testing this though.