Very slow scrub speed

Jorsher

Explorer
Joined
Jul 8, 2018
Messages
88
I'm using TrueNAS 13 Core. Scrubs used to be completed within a few days, now they take over two weeks. Hash-checking torrents also seems to be much slower than in the past, it's taken almost a week to hash-check 20TB, whereas in the past this also was much faster. I understand that hash-checking or scrubbing while other activities are going on will slow them down. They are slow even when it's the only activity.

Pool layout:
3 x vdevs in the pool
Each vdev is a 8-disk raidz2
I've used about 210TB out of ~240TB

Drive layout:
Onboard LSI-3008 SAS3 with 2 x SAS cables for 8 x 14TB drives
LSI 9208-8e with a single SAS cable to each (2) HP 468406-B21
Each HP 468406-B21 SAS2 with 2 x SAS cables for 8 x 16TB Exos drives

I understand that running 8 drives over a single SAS2 connection between the HP and LSI 9028 may be a bottleneck, but it should still (I think) be a bit faster than it is.

Is the iozone tool a good way to test performance of the NAS? What's a suggested way to run it to get an idea of "real world" performance? Anything else I can look into?
 

Samuel Tai

Never underestimate your own stupidity
Moderator
Joined
Apr 24, 2020
Messages
5,398
Your pool is >80%, which is the knee when performance drops precipitously. You need to expand your pool to regain performance.
 

Jorsher

Explorer
Joined
Jul 8, 2018
Messages
88
Your pool is >80%, which is the knee when performance drops precipitously. You need to expand your pool to regain performance.
Thanks for the response. I've heard this. TrueNAS warns you about it. Yet I've approached 99% of storage capacity and don't recall scrub speeds ever being as low as 100MB/s. I've assumed the "80%" is just a rule of thumb (like 2gb ram per tb) and wouldn't necessarily apply in my case where I have 30TB free across 24 drives.

What is the best method for testing read/write locally? (not across the network). Last time I tried iozone with 99% space used, writing/reading 128GB files, I received ~1.9GB/s write/rewrite and 4.9GB/s read/reread. I understand cache will skew those numbers but I wasn't sure how to disable it.

Perhaps I could benefit from a rebalance script like I've seen in another thread. I've always added storage as I ran out, so I likely have two vdevs almost fully used while one vdev is about half empty. Will check...
 
Last edited:

Jorsher

Explorer
Joined
Jul 8, 2018
Messages
88
Guess this could be the problem :) Will look into adding more storage and rebalancing.

sugUHgR.png
 

mav@

iXsystems
iXsystems
Joined
Sep 29, 2011
Messages
1,428
Will look into adding more storage and rebalancing.
Once your pool got high fragmentation after being filled to the limit, it may never reduce it again unless you delete that data. Torrents also tend to cause data fragmentation due to their random download order. When I used using transmission for torrents, I had separate pool of single SSD to temporary store incoming files, that were then automatically sequentially copied to the main pool on completion. It allowed to keep good sequential read speeds for them later.
 

ver151set

Dabbler
Joined
Jan 28, 2015
Messages
37
I know this is older, but there is a inplace zfs rebalancing script that works pretty good. It also defrags your pool after a couple passes. My 10 year old 8 vdev pool is perfectly balanced with less than 5% fragmentation

 

Jorsher

Explorer
Joined
Jul 8, 2018
Messages
88
I know this is older, but there is a inplace zfs rebalancing script that works pretty good. It also defrags your pool after a couple passes. My 10 year old 8 vdev pool is perfectly balanced with less than 5% fragmentation

Thanks. I'll try that. I believe the biggest issue is the drives are always pretty active, but part of it is likely how imbalanced the vdevs are. Am building a new pool and moving some items to a small flash pool to hopefully relieve some of the busyness on the spinning disks.
 
Top