SOLVED: Slow read speed on on LSI2008 RAIDZ2 - needs x8 slot

[Alister]

<div class="bbWrapper">System<br /> <br /> CPU: i5-6400<br /> RAM 2x16GB 2x8GB 48GB Total<br /> Motherboard H170A PC Mate<br /> NIC Chelsio Communications Inc S320-LP-CR 10GbE MTU9000<br /> <br /> Pool1<br /> Drives 6xWD40EFRX-68W RAIDZ2<br /> Controller LSI SAS2008 [Falcon] rev03<br /> <br /> Pool2<br /> Drives 2x 120GB SSDs (Cheap Kingstons)<br /> Controller M/B SATA ports<br /> <br /> Switch MikroTik CSS326-24G-2S+RM - TrueNAS &amp; Client in SPF+ ports<br /> <br /> PC (Client)<br /> CPU AMD 5950x<br /> Memory 32GB<br /> NIC Chelsio Communications Inc S320-LP-CR 10GbE MTU9014<br /> NVMe used for tests Read 3400MB/s Write 3300 MB/s <br /> <br /> Iperf3 gets 8 - 9 Gb/s<br /> <br /> Copy 18GB test file from pool 1 to NVMe SSD 80-90 MB/s<br /> <span style="color: rgb(20, 20, 20)">Copy18GB test file from NVMe SSD to pool 2 200 then pauses then 200 - as expected with these SSDs</span><br /> Copy 18GB test file from pool 2 to NVMe SSD ~350 MB/s<br /> <br /> then<br /> <br /> Copy 18GB test file from pool 2 to pool1 on client ~330 MB/s (this is weird?)<br /> <br /> Zpool -status is clear for both pools<br /> <br /> lspci -vv for SAS2008 shows<br /> <br /> LnkSta: Speed 5GT/s (ok), Width x4 (downgraded)<br /> <br /> is this a problem?<br /> <br /> <br /> diskinfo -ctv /dev/[Disk] attached</div>

 

[jgreco]

<div class="bbWrapper">Mmm, would say it seems a little slow. However, the older Chelsio cards are somewhat limited; your system is probably working really hard to get that 8-9Gbps iperf3.<br /> <br /> Have you found the optimization guide I posted recently in Resources?<br /> <br /> <div class="bbCodeBlock bbCodeBlock--unfurl js-unfurl fauxBlockLink" data-unfurl="true" data-result-id="32332" data-url="https://www.truenas.com/community/resources/high-speed-networking-tuning-to-maximize-your-10g-25g-40g-networks.207/" data-host="www.truenas.com" data-pending="false"> <div class="contentRow"> <div class="contentRow-main"> <h3 class="contentRow-header js-unfurl-title"> <a href="https://www.truenas.com/community/resources/high-speed-networking-tuning-to-maximize-your-10g-25g-40g-networks.207/" class="link link--internal fauxBlockLink-blockLink" target="" rel="" data-proxy-href=""> Resource - High Speed Networking Tuning to maximize your 10G, 25G, 40G networks </a> </h3> <div class="contentRow-snippet js-unfurl-desc">Both FreeBSD and Linux come by default highly optimized for classic 1Gbps ethernet. This is by far the most commonly deployed networking for both clients and servers, and a lot of research has been done to tune performance especially for local...</div> <div class="contentRow-minor contentRow-minor--hideLinks"> <span class="js-unfurl-favicon"> <img src="https://www.truenas.com/community/data/assets/logo/FORUMSFAVICON.png" alt="www.truenas.com" class="bbCodeBlockUnfurl-icon" data-onerror="hide-parent"/> </span> www.truenas.com </div> </div> </div> </div></div>

 

[Alister]

<div class="bbWrapper">You're missing the point<br /> <br /> Pool 1 - the SAS2008 gets 80MB/s<br /> Pool 2 - the M/B SATA gets 300+ MB/s<br /> <br /> I included all the testing to show that <br /> <br /> the 10GB NIC is not the issue<br /> the 10GB Switch is not the issue<br /> the client drive is not the issue<br /> etc<br /> <br /> The single variable is SAS2008 RAIDZ2 pool</div>

 

[c77dk]

<div class="bbWrapper">if you look with gstat while doing a transfer - are the disks busy?<br /> How is the pool setup? Dedup? sync ?</div>

 

[Alister]

<div class="bbWrapper">No DeDup or sync just a straight RAIDZ2 array <br /> <br /> SAS2008 2 ports , 4 SATA cables per port, with 3 drives per port<br /> <br /> Gstat show busy rarely excess 20% with odd spike to 40% most of the time &lt;6% kBps is ~20000 - 25000</div>

 

[WI_Hedgehog]

<div class="bbWrapper">Am I understanding this correctly?<br /> <br /> The spinners that can handle about 150 MB/s max. (sequential without seeks) are doing Copy-On-Write (meaning there are seeks) at 80 MB/s.<br /> <br /> The SSDs that can handle about 350 MB/s max. are striped? and handling 300 MB/s, or 150 MB/s each?<br /> <br /> Copying an 18 GB existing test file (not incoming) from mirrored? SSD to sequential-write HDD gets cached in 48 GB RAM and the transfer seems faster, though the write may not have actually completed?</div>

 

[Alister]

<div class="bbWrapper">I'm coping 18 or 60GB file off the 6 spinners at a total 80MB/s onto the network<br /> <br /> I just copied with pool1 on the server (Putty)<br /> <br /> kBps ~30000 busy 60-70%<br /> <br /> I was wondering if<br /> <br /> NIC in x16 slot<br /> SAS2008 in x4 slot<br /> <br /> is the issue and turns out it is<br /> <br /> Just swapped the cards and am now able to copy from pool1 to NVMe at 300 MB/s - drives are now showing&gt;30% and spiking to 80% - I'm limited by the Chelsio Communications Inc S320-LP-CR which is showing as LnkSta: Speed 2.5GT/s (ok), Width x4 (downgraded)<br /> <br /> (Now need to find a CHEAP PCIe 2/3 x4 SPF card)<br /> <br /> <br /> <b>Solution </b>LSI SAS2008 / 9100-8i cards need a x8 PCIe slot - LnkSta: Speed 5GT/s (ok), Width x8 (ok)</div>

 

[WI_Hedgehog]

<div class="bbWrapper"><blockquote data-attributes="member: 6012" data-quote="alister" data-source="post: 745919" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=745919" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-745919">alister said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> NIC in x16 slot<br /> <b>SAS2008 in x4 slot</b><br /> <br /> Just swapped the cards and am now able to copy from pool1 to NVMe at 300 MB/s - drives are now showing&gt;30% and spiking to 80% - I'm limited by the Chelsio Communications Inc S320-LP-CR which is showing as LnkSta: Speed 2.5GT/s (ok), Width x4 (downgraded)<br /> <br /> <b>Solution </b>LSI SAS2008 / 9100-8i cards need a x8 PCIe slot - LnkSta: Speed 5GT/s (ok), Width x8 (ok) </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Agreed, the <a href="https://docs.broadcom.com/doc/12352283" target="_blank" class="link link--external" rel="nofollow ugc noopener">LSI SAS2008</a> supports 8 PCIe lanes and complies with the PCI Express 2.0 specification that extends the PCIe data rates up to 5.0 GT/s per lane per Broadcom <span style="font-size: 10px">(specs linked)</span>.</div>

 

[jgreco]

<div class="bbWrapper"><blockquote data-attributes="member: 6012" data-quote="alister" data-source="post: 745881" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=745881" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-745881">alister said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> You're missing the point </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><br /> I'm really not. The point I'm asking you about is whether or not any additional tuning had been done. It seems clear that something is a bottleneck, but sometimes with the older cards it is just that there's a lot of overhead in the handling of all the activity; back in the mid 2000's it was absolutely necessary to set big buffers and stuff so that the kernel could just plow through large tracts of bytes without encountering a context switch from kernel back to userland to get more data queued up. My head still goes there when I see an old NIC, an old HBA, and a low thread count CPU. It seems in the meantime you've found some other bottlenecks in the system, so that's fine, I expected SOMETHING was a bottleneck but you gotta start somewhere. Tuning is cheap and easy if it turns out to be the issue. Because you do have some slower parts in your system, even if you can fix the PCIe lane issues, you will find it advantageous to trade off some RAM for network buffers. You've definitely got the right ideas about testing this though.</div>