Terrible read performance - how to troubleshoot

[paulinventome]

<div class="bbWrapper">So new build, got two zpools, one is a ZRAID2 with 8x16TB drives and the other is a mirror with 2x4TB SSDs, server is running over 10gbe.<br /> <br /> Tested from both a Mac and a Windows machine and with both pools and seeing the same, I get around 400MB-500MB/s writes and reads are around 1MB/s, yes 1. So something is up and I need some pointers to track down.<br /> <br /> I am just running SMB service. CPU and the system barely registers the load. I am looking at the Reports side, if I look at the SSD for example I see the write maxing at 260MB/s (still think this is poor and I assume the external 400/500 I am seeing is because of RAM. But the report shows that the disk reads being tiny. Same for the ZRAID2 pool.<br /> <br /> So I'm fairly new to this, where can I start troubleshooting? Are there any tools within TrueNAS UI that would let me check the disc performance locally as a first step? I am fairly happy with command lines too.<br /> <br /> Does a 1MB/s read performance either off RAID or SSD ring any bells with anyone?<br /> <br /> This is replacing an ageing Synology 3612xs which has been running 8 years, I still get 600MB/s to and from that. Same network, same clients, same testing software (Blackmagic Speed Test)<br /> <br /> Any pointers would be really appreciated!<br /> <br /> thanks<br /> Paul</div>

 

[sretalla]

<div class="bbWrapper"><blockquote data-attributes="member: 49916" data-quote="paulinventome" data-source="post: 702744" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=702744" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-702744">paulinventome said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> So I'm fairly new to this, where can I start troubleshooting? Are there any tools within TrueNAS UI that would let me check the disc performance locally as a first step? I am fairly happy with command lines too. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote><code class="bbCodeInline">fio</code><br /> <br /> <code class="bbCodeInline">zpool iostat -v</code><br /> <br /> <code class="bbCodeInline">iostat</code></div>

 

[paulinventome]

<div class="bbWrapper">So I found the dd command<br /> <br /> SSD is (READ/WRITE)<br /> 107374182400 bytes transferred in 21.386735 secs (5020597342 bytes/sec)<br /> 107374182400 bytes transferred in 12.710596 secs (8447612230 bytes/sec)<br /> <br /> ZRAID2 is<br /> 107374182400 bytes transferred in 21.654265 secs (4958569686 bytes/sec)<br /> 107374182400 bytes transferred in 12.709673 secs (8448225497 bytes/sec)<br /> <br /> from the command line in the UI. Now the Reports section of truenas doesn't report these does it? I found the fact that the SSD and the ZRAID2 results were remarkably similar kind of odd so I wanted to see the reports to make sure the discs being used did make sense but the reports really don't feature the above.<br /> <br /> I believe my next port of call is iperf? Reading some other threads point to network card but what is odd in my circumstances is that early on in the build I had a single SSD running that I benchmarked remotely running happily at 500MB/s both directions. So I have seen this working..<br /> <br /> cheers<br /> Paul</div>

 

[paulinventome]

<div class="bbWrapper"><blockquote data-attributes="member: 55604" data-quote="sretalla" data-source="post: 702763" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=702763" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-702763">sretalla said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> <code class="bbCodeInline">fio</code><br /> <br /> <code class="bbCodeInline">zpool iostat -v</code><br /> <br /> <code class="bbCodeInline">iostat</code> </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>So zpool istat gives<br /> <br /> capacity operations bandwidth<br /> pool alloc free read write read write<br /> ---------- ----- ----- ----- ----- ----- -----<br /> OmeMedia 21.7M 3.62T 0 18 367 1.83M<br /> OmeVault 209G 116T 0 7 42 1.36M<br /> boot-pool 1.29G 215G 0 0 9.12K 161<br /> ---------- ----- ----- ----- ----- ----- -----<br /> <br /> Which doesn't follow the dd results shown above so either the dd is not working or those results aren't recorded as part of truenas (most likely)<br /> <br /> Stupid question but what are the units for bandwidth?<br /> <br /> And fio, need to look into this too.<br /> <br /> I think my dd results are rubbish, I just read something about compression (and it's all zero's right?)<br /> <br /> thanks<br /> Paul</div>

 

[paulinventome]

<div class="bbWrapper">Just another addition, with compression off I am getting more realistic results<br /> <br /> 107374182400 bytes transferred in 373.466036 secs (287507222 bytes/sec)<br /> 107374182400 bytes transferred in 134.937192 secs (795734525 bytes/sec)<br /> <br /> 107374182400 bytes transferred in 304.602843 secs (352505517 bytes/sec)<br /> 107374182400 bytes transferred in 112.694355 secs (952791136 bytes/sec)<br /> <br /> This is closer to what I expect, although the SSD reading at 795MB/s is beyond the SSD specs but with a mirror setup does TrueNAS pull from both for speed?<br /> <br /> Guess this is a NIC issue, maybe or setup</div>

 

[sretalla]

<div class="bbWrapper">using dd to test write or read performance is a minefield of things that can cause results that are due to things you're not looking at.<br /> <br /> where are you reading from and writing to (or just show the exact command you're using) and maybe it will be clear...<br /> <br /> Main pitfall... using /dev/zero or /dev/random. those are both going to have misleading outcomes due to compression in the first case and CPU limitations in the second.<br /> <br /> Bytes is the unit for those commands.<br /> <br /> Look around for examples of fio, it's probably the best tool for whole pool testing.</div>

 

[paulinventome]

<div class="bbWrapper"><blockquote data-attributes="member: 55604" data-quote="sretalla" data-source="post: 702781" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=702781" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-702781">sretalla said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> using dd to test write or read performance is a minefield of things that can cause results that are due to things you're not looking at.<br /> <br /> where are you reading from and writing to (or just show the exact command you're using) and maybe it will be clear...<br /> <br /> Main pitfall... using /dev/zero or /dev/random. those are both going to have misleading outcomes due to compression in the first case and CPU limitations in the second.<br /> <br /> Bytes is the unit for those commands.<br /> <br /> Look around for examples of fio, it's probably the best tool for whole pool testing. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Thanks. I think I have verified that the pools are performing locally closer to what I expect. My task is now why over a network and SMB share on both Mac and windows I am seeing sub 1MB/s read speeds <br /> <br /> Thanks<br /> Paul</div>