poor read speed, great write on raid10 6 disks

[sretalla]

Note that the filename has no path, so that's going in the PWD... probably /root (so not testing the pool)

 

[tngri]

Note that the filename has no path, so that's going in the PWD... probably /root (so not testing the pool)

well I was wondering about that but

Code:

ant@mastodon:~ % pwd
/mnt/tank/Users/ant
ant@mastodon:~ % ll
total 10493562
....
....
-rw-r--r--  1 ant  ant        47031 Aug  4  2019 iozone.xls
-rw-r--r--  1 ant  ant  10737418240 Sep  9 13:01 randreadwrite.fio      <=======================
-rw-r--r--  1 ant  ant  26843545600 Aug 17 16:41 tmp.dat
ant@mastodon:~ % 

 

[hescominsoon]

with your data not spread evenly the first thing i would do is export your data to another pool or mahcine..destroy the pool..rebuild it..then pull the data back. Once that's done try your read testing again.

 

[tngri]

with your data not spread evenly the first thing i would do is export your data to another pool or mahcine..destroy the pool..rebuild it..then pull the data back. Once that's done try your read testing again.

I think that's the only thing I can do, waste of time :'(
I noticed some smart error (pending sectors) on a drives when coming back to the WebUI ...

 

[hescominsoon]

if you are getting smart errors i would replace the drive AFTER you get the data off the pool

 

[tngri]

I get them since I moved the chassis for installing the Chelsio, I pretty sure it's a cable issue ... wait and see ?

 

[awasb]

Speaking of s.m.a.r.t ...

_before_ moving data to and fro (I'm sure you have a backup already) I'd check the following:

1.) Are "tunables" enabled? If so, disable them. (Just to be sure you are at "default system level" without those fancy pants automatic adjustments. ;-) )

2.) Do you have any PCIe-cards/-modules installed? What's the config in the BIOS? Is any NVMe-storage attached?

3.) Try this one:

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

 

[tngri]

Speaking of s.m.a.r.t ...

_before_ moving data to and fro (I'm sure you have a backup already) I'd check the following:

1.) Are "tunables" enabled? If so, disable them. (Just to be sure you are at "default system level" without those fancy pants automatic adjustments. ;-) )

2.) Do you have any PCIe-cards/-modules installed? What's the config in the BIOS? Is any NVMe-storage attached?

3.) Try this one:

solnet-array-test

Back in the late '90's, I was managing a bunch of large whitebox storage servers. For the largest of these, I had the pleasure of building and deploying a massive storage server, 8 shelves of 9 drives each, Seagate ST173404LW 73GB drives, a...

www.truenas.com

1) You mean unchek the "auto-tunable" (system options) only or also remove all the "tunables" entries ? (set by auto tunable ? dunno .... some are still activated after disable auto-tunable)

2) I have only 8x PCIE port occupied by the Chelsion 320 (Eth10 SFP+ dual) I'm testing, so impossible do add nvme my board only have SATA (ASRock Rack C2750D4I)

3) Thanks having a look

 

[ChrisRJ]

160MB/s read out of cache ? that seems low even out of ARC, 3xstripes shoud five me 3xSingle Sata drive no ? (~300MB/s)

You mixed up what I wrote about read and write. With SMB (async write by default) write speeds higher than read ones are not that uncommon.

 

[awasb]

1) You mean unchek the "auto-tunable" (system options) only or also remove all the "tunables" entries ? (set by auto tunable ? dunno .... some are still activated after disable auto-tunable)

Before running any benchmarks I (for my part) disable/erase everything with the description „generated by autotuning script“ (not sure about the exact wording, but you‘ll know, when you read it).

If you want them back, you could reenable the generator script anytime.

Try to run the benchmark without any tunables. I have the exact same board. (Just half the RAM and onboard 1Gb-NIC.) If you won‘t get an „average“ of roundabout 80-115MB/s reading (single drive performance varies on a „hot“ production pool, don‘t be surprised) then I’d guess, something‘s wrong with your BIOS, controller(s), or settings.

If the benchmark meets the expectations, I‘d try to re-create the pool. But not before. (And not before fixing cables … ;-) )

 

[tngri]

thanks a lot Awasb for helping, I tried to run the script sadly I got

Code:

root@mastodon:/mnt/tank/home # ./solnet-array-test-v2.sh
Illegal option -

regarding the BIOS I have reviewed it, checked all options about SATA, everything seems normal ...

 

[awasb]

Over here it runs actually without any hickups. Just copied to the base-dir of the pool,

Code:

chmod 700

executed script. (TrueNAS Core 12-U5.1)

 

[tngri]

Test running, my bad, I had CRLF endline, the file being downloaded from a windows machine :D

Still on the run but I have first benchmark, amazing tool

Code:

!!ERROR!! dd: /dev/ada3: Input/output error

Array's average speed is 123.238 MB/sec per disk

Disk    Disk Size  MB/sec %ofAvg
------- ---------- ------ ------
ada0     2861588MB    140    113 ++FAST++
ada1     2861588MB    150    122 ++FAST++
ada2     2861588MB    140    114 ++FAST++
ada3     2861588MB      0      0 --SLOW--
ada5     2861588MB    166    135 ++FAST++
ada6     2861588MB    143    116 ++FAST++

"Awaiting completion: initial parallel array read" ..... gstat show me that ada3 is doing nothing when all others drives are crazy at work !

Do I need to recreate the entire pool as @hescominsoon suggested, or maybe just wip/replace/resilvet that damn ada3 ?

 

[awasb]

I'd check cables first. When everything is fine (with "physical" connection, that is), have a look at

Code:

zpool status -v

 

[tngri]

Code:

pool: tank
state: ONLINE
  scan: scrub repaired 0B in 05:21:56 with 0 errors on Sun Mar  8 05:21:57 2020
config:

        NAME                                            STATE     READ WRITE CKSUM
        tank                                            ONLINE       0     0     0
          mirror-0                                      ONLINE       0     0     0
            gptid/cdc1a35e-a93a-11e9-95a1-d05099c188d1  ONLINE       0     0     0
            gptid/cf82bc53-a93a-11e9-95a1-d05099c188d1  ONLINE       0     0     0
          mirror-1                                      ONLINE       0     0     0
            gptid/dd0fbcf3-a93a-11e9-95a1-d05099c188d1  ONLINE       0     0     0
            gptid/df012241-a93a-11e9-95a1-d05099c188d1  ONLINE       0     0     0
          mirror-2                                      ONLINE       0     0     0
            gptid/ec9e4ce9-a93a-11e9-95a1-d05099c188d1  ONLINE       0     0     0
            gptid/1996590b-42c8-11ea-87b4-d05099d3036a  ONLINE       0     0     0

cables seems fine

 

[awasb]

Cables fixed/swapped? (BTW: Wow. Last scrub 8th of March. That's a long time. I do this twice a month.)

 

[tngri]

Well to me not a cable issue at the end : I switch ada3 and ada4 cables and stil got SMART and sol-test error, same situation
So now I'm doing some backup (the non essentials mainly because I want to keep some movies etc...)

and then what do you think ? wipe/replace/resilver ada3 (hope that's enough) or completly redo the pool ?

note : And sure I fix the Cron for Srubbing after thing back to normal ... :P

 

[awasb]

I‘d start with a new disk as replacement for „ada3“ and resilvering. Not for any special reason. Just out of curiosity, if that fixes the problem.

(You could setup a new pool as well. But you wouldn‘t know for sure, what the exact cause for degraded performance was. Though probability points to a hardware problem.)

 

[tngri]

oops, I made a detach instead of "offline" the bad drive.... I guess not cancelable ? so I think I will need to redo the entire pool at the end ....

 

[awasb]

No matter what ... please report the results for reading performance ;-)